-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add caching file system to hive connector #13904
Conversation
jainxrohit
commented
Dec 30, 2019
•
edited by highker
Loading
edited by highker
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some comments
presto-hive/src/main/java/com/facebook/presto/hive/HiveClientModule.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/HiveClientModule.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/HiveClientModule.java
Outdated
Show resolved
Hide resolved
...ptor/src/main/java/com/facebook/presto/raptor/filesystem/RaptorCachingHdfsConfiguration.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/HdfsModule.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you remove the period in the commit title? https://chris.beams.io/posts/git-commit/ is a good commit message guideline.
Nice article, fixed the commit message. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM % nits
presto-hive/src/main/java/com/facebook/presto/hive/CachingFileOpener.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/HiveCachingHdfsConfiguration.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/HdfsModule.java
Show resolved
Hide resolved
The test failure is due to permission/auth setting. Try overriding the following function in @Override
public void setPermission(Path path, FsPermission permission)
throws IOException
{
dataTier.setPermission(path, permission);
} But in general, I would suggest overriding all default functions from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
coding style comments
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/HiveCachingHdfsConfiguration.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some comments
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-cache/src/main/java/com/facebook/presto/cache/CachingFileSystem.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/CachingFileOpener.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. @shixuan-fan, could you give it a final review and merge it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, ideally we would want to have three commits:
- Fixing caching file system
- Raptor side change
- Hive side change
But since it is already reviewed, I won't bother breaking it down. I'll merge it once we've completed the internal repo pull request that adapt to this one.