Downgrade Guava to version 15.0 to work with Hadoop 2.6.0 #280
Conversation
Due to the issue reported in HADOOP-10961, Hadoop 2.6.0 does not work with Guava version 16.0 or after because the deprecation of a constructor of Stopwatch. To make sure Gobblin supports Hadoop 2.6.0, we need to downgrade Guava to version 15.0. Signed-off-by: Yinan Li <liyinan926@gmail.com>
|
@chavdar can you review? |
| @@ -27,7 +27,7 @@ | |||
|
|
|||
| public class AvroFsHelperTest { | |||
|
|
|||
| @Test(expectedExceptions = IllegalArgumentException.class) | |||
There was a problem hiding this comment.
For some reason, Hadoop 2.6.0 seems not recognizing the s3 scheme.
There was a problem hiding this comment.
What exception does it throw? Is it possible to add this to expectedExceptions instead of disabling the test?
There was a problem hiding this comment.
Caused by: java.io.IOException: No FileSystem for scheme: s3
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at gobblin.source.extractor.hadoop.AvroFsHelper.createFileSystem(AvroFsHelper.java:77)
at gobblin.source.extractor.hadoop.AvroFsHelper.connect(AvroFsHelper.java:85)
There was a problem hiding this comment.
Turns out the reason for this is: "For some reason, the jar hadoop-aws.jar which contains the implementation to NativeS3FileSystem is not present in the classpath of hadoop by default in the version 2.6". Fixed this by adding a dependency on hadoop-aws.
…tials Signed-off-by: Yinan Li <liyinan926@gmail.com>
|
Updated. |
|
LGTM |
Downgrade Guava to version 15.0 to work with Hadoop 2.6.0
|
Just a quick comment. Not sure if it is related though it would seem that gobblin/bin/gobblin-mapreduce.sh still requires guava-18.0.jar. # Jars Gobblin runtime depends on
LIBJARS=$USER_JARS$separator$FWDIR_LIB/gobblin-metastore.jar,$FWDIR_LIB/gobblin-metrics.jar,\
$FWDIR_LIB/gobblin-core.jar,$FWDIR_LIB/gobblin-api.jar,$FWDIR_LIB/gobblin-utility.jar,\
$FWDIR_LIB/guava-18.0.jar,$FWDIR_LIB/avro-1.7.7.jar,$FWDIR_LIB/metrics-core-3.1.0.jar,\
$FWDIR_LIB/gson-2.3.1.jar,$FWDIR_LIB/joda-time-2.8.1.jar,$FWDIR_LIB/data-1.15.9.jar |
|
@liyinan926 passed on another box, the original box ,which running ambari master who's port is 8080. Thanks every ! |
|
@mark1900 that does indeed look like a bug. @liyinan926 is it possible to build |
Due to the issue reported in HADOOP-10961, Hadoop 2.6.0 does not work with Guava version 16.0 or after because the deprecation of a constructor of Stopwatch. To make sure Gobblin supports Hadoop 2.6.0, we need to downgrade Guava to version 15.0.
Signed-off-by: Yinan Li liyinan926@gmail.com