repository-hdfs does not work with elasticsearch 2.0.0-beta1

[dadoonet]

When starting elasticsearch 2.0.0-beta1 with repository-hdfs 2.2.0-m1, I get:

[2015-09-07 15:13:34,386][INFO ][org.elasticsearch.node   ] [Gaea] version[2.0.0-beta1], pid[12997], build[bfa3e47/2015-08-24T08:41:25Z]
[2015-09-07 15:13:34,386][INFO ][org.elasticsearch.node   ] [Gaea] initializing ...
Exception in thread "main" java.lang.IllegalStateException: failed to load bundle [...SKIPPED...]
Likely root cause: java.lang.IllegalStateException: jar hell!
class: org.apache.commons.collections.ArrayStack
jar1: /Users/dpilato/Documents/Elasticsearch/apps/elasticsearch/elasticsearch-2.0.0-beta1/plugins/repository-hdfs/commons-beanutils-core-1.8.0.jar
jar2: /Users/dpilato/Documents/Elasticsearch/apps/elasticsearch/elasticsearch-2.0.0-beta1/plugins/repository-hdfs/commons-collections-3.2.1.jar

[costin]

There seems to be a difference between the ES behaviour in ZIP vs JAR mode. This is what I get when running the test suite (from the project):

[2015-09-07 18:31:42,864][WARN ][org.elasticsearch.bootstrap] JNA not found. native methods will be disabled.
java.lang.ClassNotFoundException: com.sun.jna.Native
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:195)
    at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:43)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:86)
    at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:55)
    at org.elasticsearch.test.ESTestCase.<clinit>(ESTestCase.java:106)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:278)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:562)
[2015-09-07 18:31:42,867][WARN ][org.elasticsearch.bootstrap] cannot check if running as root because JNA is not available
[2015-09-07 18:31:42,868][WARN ][org.elasticsearch.bootstrap] cannot mlockall because JNA is not available
[2015-09-07 18:31:42,869][WARN ][org.elasticsearch.bootstrap] cannot register console handler because JNA is not available
[2015-09-07 18:31:43,417][WARN ][org.elasticsearch.bootstrap] Your ide or custom test runner has jar hell issues, you might want to look into that
java.lang.IllegalStateException: jar hell!
class: org.apache.jasper.compiler.Localizer
jar1: C:\Users\costin\.gradle\caches\modules-2\files-2.1\tomcat\jasper-runtime\5.5.12\f3a50a55414655b9843f5a089923ea83d49dc55e\jasper-runtime-5.5.12.jar
jar2: C:\Users\costin\.gradle\caches\modules-2\files-2.1\tomcat\jasper-compiler\5.5.12\c594866c64565344c0e7bdc9bf4fee70290c4dd5\jasper-compiler-5.5.12.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:210)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:128)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86)
    at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:62)
    at org.elasticsearch.test.ESTestCase.<clinit>(ESTestCase.java:106)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:278)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:562)
[2015-09-07 18:31:43,757][INFO ][org.elasticsearch.test   ] Setup InternalTestCluster [TEST-CHILD_VM=[0]-CLUSTER_SEED=[562052492325177341]-HASH=[2BB0F17F441F0]-cluster] with seed [7CCCFC38E2A47FD] using [2] data nodes and [0] client nodes
[2015-09-07 18:31:43,934][INFO ][org.elasticsearch.node   ] [node_t0] version[2.0.0-beta1], pid[13840], build[bfa3e47/2015-08-26T13:38:19Z]
[2015-09-07 18:31:43,934][INFO ][org.elasticsearch.node   ] [node_t0] initializing ...
[2015-09-07 18:31:43,940][INFO ][org.elasticsearch.plugins] [node_t0] loaded [repository-hdfs], sites []
[2015-09-07 18:31:43,976][INFO ][org.elasticsearch.env    ] [node_t0] using [1] data paths, mounts [[OS (C:)]], net usable_space [61.7gb], net total_space [120.7gb], spins? [unknown], types [NTFS]
[2015-09-07 18:31:45,940][INFO ][org.elasticsearch.node   ] [node_t0] initialized
[2015-09-07 18:31:45,941][INFO ][org.elasticsearch.node   ] [node_t0] starting ...
[2015-09-07 18:31:45,943][INFO ][org.elasticsearch.transport] [node_t0] bound_address {local[1]}, publish_address {local[1]}
[2015-09-07 18:31:45,951][INFO ][org.elasticsearch.discovery] [node_t0] TEST-CHILD_VM=[0]-CLUSTER_SEED=[562052492325177341]-HASH=[2BB0F17F441F0]-cluster/53-y5LG9SQOwEFHbZvQOKg
[2015-09-07 18:31:45,952][INFO ][org.elasticsearch.cluster.service] [node_t0] new_master {node_t0}{53-y5LG9SQOwEFHbZvQOKg}{local}{local[1]}{mode=local}, reason: local-disco-initial_connect(master)
[2015-09-07 18:31:45,966][INFO ][org.elasticsearch.node   ] [node_t0] started
[2015-09-07 18:31:45,966][INFO ][org.elasticsearch.test   ] Start Shared Node [node_t0] not shared
[2015-09-07 18:31:45,987][INFO ][org.elasticsearch.node   ] [node_t1] version[2.0.0-beta1], pid[13840], build[bfa3e47/2015-08-26T13:38:19Z]
[2015-09-07 18:31:45,987][INFO ][org.elasticsearch.node   ] [node_t1] initializing ...
[2015-09-07 18:31:45,988][INFO ][org.elasticsearch.plugins] [node_t1] loaded [repository-hdfs], sites []
[2015-09-07 18:31:46,003][INFO ][org.elasticsearch.env    ] [node_t1] using [1] data paths, mounts [[OS (C:)]], net usable_space [61.7gb], net total_space [120.7gb], spins? [unknown], types [NTFS]
[2015-09-07 18:31:46,056][INFO ][org.elasticsearch.gateway] [node_t0] recovered [0] indices into cluster_state
[2015-09-07 18:31:46,248][INFO ][org.elasticsearch.node   ] [node_t1] initialized
[2015-09-07 18:31:46,248][INFO ][org.elasticsearch.node   ] [node_t1] starting ...
[2015-09-07 18:31:46,249][INFO ][org.elasticsearch.transport] [node_t1] bound_address {local[2]}, publish_address {local[2]}
[2015-09-07 18:31:46,252][INFO ][org.elasticsearch.discovery] [node_t1] TEST-CHILD_VM=[0]-CLUSTER_SEED=[562052492325177341]-HASH=[2BB0F17F441F0]-cluster/v4MNoSfaRb-yXiQ7tqZjWQ
[2015-09-07 18:31:46,253][INFO ][org.elasticsearch.cluster.service] [node_t0] added {{node_t1}{v4MNoSfaRb-yXiQ7tqZjWQ}{local}{local[2]}{mode=local},}, reason: local-disco-receive(from node[{node_t1}{v4MNoSfaRb-yXiQ7tqZjWQ}{local}{local[2]}{mode=local}])
[2015-09-07 18:31:46,261][INFO ][org.elasticsearch.cluster.service] [node_t1] detected_master {node_t0}{53-y5LG9SQOwEFHbZvQOKg}{local}{local[1]}{mode=local}, added {{node_t0}{53-y5LG9SQOwEFHbZvQOKg}{local}{local[1]}{mode=local},}, reason: local-disco-receive(from master)
[2015-09-07 18:31:46,272][INFO ][org.elasticsearch.node   ] [node_t1] started
[2015-09-07 18:31:46,272][INFO ][org.elasticsearch.test   ] Start Shared Node [node_t1] not shared
[2015-09-07 18:31:46,319][INFO ][org.elasticsearch.repositories.hdfs] test using _default_ mappings: [{"_default_":{"dynamic_templates":[{"template-strings":{"match_mapping_type":"string","mapping":{"fielddata":{"loading":"eager"}}}},{"template-longs":{"match_mapping_type":"long","mapping":{"doc_values":true,"fielddata":{"loading":"lazy"}}}},{"template-doubles":{"match_mapping_type":"double","mapping":{"doc_values":true,"fielddata":{"loading":"lazy"}}}},{"template-geo_points":{"match_mapping_type":"geo_point","mapping":{"doc_values":true,"fielddata":{"loading":"lazy"}}}},{"template-booleans":{"match_mapping_type":"boolean","mapping":{"fielddata":{"format":"doc_values","loading":"lazy"}}}}]}}]
[2015-09-07 18:31:46,384][INFO ][org.elasticsearch.repositories.hdfs] [HdfsSnapshotRestoreTest#testSimpleWorkflow]: before test
[2015-09-07 18:31:46,706][INFO ][org.elasticsearch.repositories.hdfs] -->  creating hdfs repository with path [build/data/repo-737915553]
[2015-09-07 18:31:46,760][DEBUG][org.elasticsearch.repositories.hdfs] [node_t0] Using file-system [org.apache.hadoop.fs.LocalFileSystem@65e221bd] for URI [file:///], path [file:/E:/work/es/es-hadoop/repository-hdfs/build/data/repo-737915553]
[2015-09-07 18:31:46,791][INFO ][org.elasticsearch.repositories] [node_t0] put repository [test-repo]
[2015-09-07 18:31:46,826][DEBUG][org.elasticsearch.repositories.hdfs] [node_t1] Using file-system [org.apache.hadoop.fs.LocalFileSystem@2a1b7909] for URI [file:///], path [file:/E:/work/es/es-hadoop/repository-hdfs/build/data/repo-737915553]
[2015-09-07 18:31:46,877][WARN ][org.apache.hadoop.util.NativeCodeLoader] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[2015-09-07 18:31:46,914][INFO ][org.elasticsearch.repositories.hdfs] using custom data_path for index: [HiTvrCygQa]
...

The first exception has an issue associated with it here. The second is jar hell which acts as a warning not a fatal exception - ES keeps on running.

That's not to say it shouldn't be removed rather that it doesn't seem fatal.

Considering beta2 should be released soon, I'd rather test against that especially since Guava was/is removed.

[dadoonet]

This is expected. Tests relax the Security Manager by only WARNING the user because you could hit some issues when running from the IDE. The pb here is that the JarHell stops at the very first conflict instead of printing all conflicts at once. @rmuir Do you think we could print all JarHell errors in one go?

[rmuir]

no we shouldn't. It will make things too complicated (think thousands of conflicting class files just from one jar or so). Then there would be a problem that its too noisy and so on.

This code should stay simple... JarHell can be mean as hell: its the opposite of user-friendly :)

[costin]

This has been address in master (the upcoming v2.2.0-m2). It does however require the security manager to be disabled for the time being.

[karuppasamyit10]

me to got same issue. but i am using
compile group: 'org.springframework.data', name: 'spring-data-elasticsearch', version: '2.0.2.RELEASE'
springBootVersion = '1.4.2.RELEASE'

please help me some one