org.apache.hadoop.hdfs.HftpFileSystem could not be instantiated

[wwshuan]

The workload of Kmeans always failed in the prepare phase with the following errors:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hibench-2.2.1/common/mahout-distribution-0.7-cdh4/examples/target/mahout-examples-0.7-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hibench-2.2.1/common/mahout-distribution-0.7-cdh4/examples/target/dependency/slf4j-jcl-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hibench-2.2.1/common/mahout-distribution-0.7-cdh4/examples/target/dependency/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.HftpFileSystem could not be instantiated: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.DelegationTokenRenewer.(Ljava/lang/Class;)V from class org.apache.hadoop.hdfs.HftpFileSystem
at java.util.ServiceLoader.fail(ServiceLoader.java:224)
at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2275)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2286)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2303)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:322)
at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:383)
at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:281)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:422)
at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:168)
at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:151)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
at org.apache.mahout.clustering.kmeans.GenKMeansDataset.main(GenKMeansDataset.java:604)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.DelegationTokenRenewer.(Ljava/lang/Class;)V from class org.apache.hadoop.hdfs.HftpFileSystem
at org.apache.hadoop.hdfs.HftpFileSystem.(HftpFileSystem.java:84)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
... 21 more

exec "$HADOOP_EXECUTABLE" --config $HADOOP_CONF_DIR jar ${DATATOOLS} org.apache.mahout.clustering.kmeans.GenKMeansDataset -libjars $MAHOUT_HOME/examples/target/mahout-examples-0.7-job.jar ${COMPRESS_OPT} ${OPTION}

CDH 4.3 installed with Cloudera Manager 4.6
HiBench 2.2.1
RHEL6.3 64-bit

Thanks.
Vincent

[AllanY]

Till now we only support CDH4.1 and have not yet begun the work of porting it to CDH4.3.

[wwshuan]

Thanks Allan. Got it.

[wwshuan]

Hi Allan,
I'm not sure, but just double confirm with you the version of CDH4.1. Is CDH4.1.4 supported by HiBench 2.2.1?

[AllanY]

Vincent, basically we have tested the running of HiBench 2.2.1 on cdh 4.1.2. So currently we cannot promise HiBench can work well on higher versions of cdh.