Adding support for rubix scheme and populating filesystem counters #77
Conversation
abhishekdas99
requested review from
vrajat,
pvam,
shubhamtagra and
ankitdixit
rubix-core/pom.xml
Outdated
| @@ -15,19 +15,15 @@ | |||
|
|
|||
| <properties> | |||
| <main.basedir>${project.parent.basedir}</main.basedir> | |||
| <dep.hadoop2.version>2.6.0</dep.hadoop2.version> | |||
There was a problem hiding this comment.
this is not required, it is provided by root pom
| } | ||
|
|
||
| @VisibleForTesting | ||
| public CachedReadRequestChain(String fileToRead) | ||
| throws IOException | ||
| { | ||
| this(fileToRead, ByteBuffer.allocate(1024)); | ||
| this(fileToRead, ByteBuffer.allocate(1024), null); |
There was a problem hiding this comment.
simpler to create new Statistics object here to keep rest of the code simple
| @@ -102,13 +107,17 @@ public void initialize(URI uri, Configuration conf) throws IOException | |||
| throw new IOException("Cluster Manager not set"); | |||
| } | |||
| super.initialize(uri, conf); | |||
| fs.initialize(uri, conf); | |||
| this.uri = URI.create(uri.getScheme() + "://" + uri.getAuthority()); | |||
| this.workingDir = new Path("/user", System.getProperty("user.name")).makeQualified(this); | |||
There was a problem hiding this comment.
is it necessary to set workingDir?
There was a problem hiding this comment.
For most of the filesystem implementations, the initialize method sets the working directory and uri for that class.Working directory is required to create the absolute path. Just to be consistent for all the filesystem implementations, i did this change
| @@ -100,6 +103,9 @@ public Integer call() | |||
| log.info(String.format("Read %d bytes from cached file", read)); | |||
| fileChannel.close(); | |||
| raf.close(); | |||
| if (statistics != null) { | |||
| statistics.incrementBytesRead(read); | |||
There was a problem hiding this comment.
Is there a way that we can provide our own Statistics implementation? Basically, we would want to differentiate amount of data read locally (i.e. in CachenRRC) and amount of data read from other nodes (i.e. in NonLocalRRC).
There was a problem hiding this comment.
Yes right now we are only populating bytes read counter for local read. Current implementation of NonLocalRead stats gives you both remote read and non local read combined. But once we have the asynchronous cache warm up feature, it will be easy to differentiate between the data read from the non local cache and data we are downloading in the remote node as part of this request.
There was a problem hiding this comment.
Do you know how are these statistics shown by Hive jobs? i.e. does it call FileSystem.printStatistics method? Based on that we can think of workarounds
There was a problem hiding this comment.
FileSystem has the statistics object. This is part of hadoop file system level counter. Hive just gets a report from AM and prints it
| @@ -206,7 +219,27 @@ public boolean mkdirs(Path path, FsPermission fsPermission) | |||
| public FileStatus getFileStatus(Path path) | |||
| throws IOException | |||
| { | |||
| return fs.getFileStatus(path); | |||
| FileStatus originalStatus = fs.getFileStatus(path); | |||
There was a problem hiding this comment.
So changing listing's path to rubix scheme was the main change to support rubix scheme!!
There was a problem hiding this comment.
Yes this one is main along with initializing the remotefilesystem with appropriate scheme (for CachingNativeS3FileSystem, its s3n) so that when we are reading from the remote filesystem, it gets reflected properly
No description provided.