Failure seen while porting JanusGraph on ppc64le

[ArchaB]

Hi All,

I am trying run the JanusGraph build and automated tests on ubuntu 16.10 and RHEL 7.3 VM ppc64le and for the module janusgraph-hadoop-parent, test case verify-janusgraph-cassandra-test fails on both the distros.

The testcase summary file (/janusgraph/janusgraph-hadoop-parent/janusgraph-hadoop-1/target/failsafe-reports/failsafe-janusgraph-cassandra.xml) shows error in 21 tests.

Further analysis showed 7 testcases failed with IncompatibleClassChangeError and 3 testcases failed with IllegalStateException

Analysis of IncompatibleClassChangeError:
java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.cassandra.hadoop.HadoopCompat.(HadoopCompat.java:71)
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:120)
at org.janusgraph.hadoop.formats.cassandra.CassandraBinaryInputFormat.getSplits(CassandraBinaryInputFormat.java:62)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at org.janusgraph.hadoop.scan.HadoopScanRunner.runJob(HadoopScanRunner.java:136)
at org.janusgraph.hadoop.MapReduceIndexManagement.updateIndex(MapReduceIndexManagement.java:186)
at org.janusgraph.hadoop.AbstractIndexManagementIT.testRemoveGraphIndex(AbstractIndexManagementIT.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

Analysis on IllegalStateException
java.lang.IllegalStateException: java.lang.ExceptionInInitializerError
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep.processNextStart(ComputerResultStep.java:80)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:126)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:37)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:157)
at org.janusgraph.hadoop.CassandraInputFormatIT.testReadGraphOfTheGods(CassandraInputFormatIT.java:55) Caused by: java.lang.ExceptionInInitializerError: null
at org.apache.spark.storage.DiskBlockManager.addShutdownHook(DiskBlockManager.scala:147)
at org.apache.spark.storage.DiskBlockManager.(DiskBlockManager.scala:54)
at org.apache.spark.storage.BlockManager.(BlockManager.scala:75)
at org.apache.spark.storage.BlockManager.(BlockManager.scala:173)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:347)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
at org.apache.spark.SparkContext.(SparkContext.scala:450)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2258)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at org.apache.tinkerpop.gremlin.spark.structure.Spark.create(Spark.java:53)
at org.apache.tinkerpop.gremlin.spark.structure.io.SparkContextStorage.open(SparkContextStorage.java:60)
at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$28(SparkGraphComputer.java:138)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoSuchFieldException: SHUTDOWN_HOOK_PRIORITY
at java.lang.Class.getField(Class.java:1703)
at org.apache.spark.util.SparkShutdownHookManager.install(ShutdownHookManager.scala:220)
at org.apache.spark.util.ShutdownHookManager$.shutdownHooks$lzycompute(ShutdownHookManager.scala:50)
at org.apache.spark.util.ShutdownHookManager$.shutdownHooks(ShutdownHookManager.scala:48)
at org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:189)
at org.apache.spark.util.ShutdownHookManager$.(ShutdownHookManager.scala:58)
at org.apache.spark.util.ShutdownHookManager$.(ShutdownHookManager.scala)
at org.apache.spark.storage.DiskBlockManager.addShutdownHook(DiskBlockManager.scala:147)
at org.apache.spark.storage.DiskBlockManager.(DiskBlockManager.scala:54)
at org.apache.spark.storage.BlockManager.(BlockManager.scala:75)
at org.apache.spark.storage.BlockManager.(BlockManager.scala:173)

Steps I followed were:
git clone https://github.com/JanusGraph/janusgraph && cd janusgraph
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
mvn install
I am doing this build on Ubuntu 16.04 and RHEL 7.3 ppc64le. Please help me in resolving this issue.

Thanks,
Archa

[sjudeng]

JanusGraph is currently using TinkerPop 3.1.0 which doesn't look like it supports Hadoop1 anymore. See Hadoop-Gremlin section on the page below. The janusgraph-hadoop-1 module probably just needs to be removed.

http://tinkerpop.apache.org/docs/3.1.1-incubating/upgrade/#_tinkerpop_3_1_0_2

[ArchaB]

Thanks @sjudeng for the quick response.
Will try building JanusGraph by removing janusgraph-hadoop-1 module and update the behavior

[ArchaB]

Enabled Hadoop1 which resulted in 1 testcase (testReadWideVertexWithManyProperties) failure in janusgraph-hadoop-2 package
Failure seen:
<error type="java.lang.IllegalStateException" message="org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory output/~reducing already exists">java.lang.IllegalStateException: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory output/~reducing already exists at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep.processNextStart(ComputerResultStep.java:80) at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:126) at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:37) at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:157) at org.janusgraph.hadoop.CassandraInputFormatIT.testReadWideVertexWithManyProperties(CassandraInputFormatIT.java:81) Caused by: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory output/~reducing already exists at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1011) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:938) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:930) at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:809) at org.apache.tinkerpop.gremlin.spark.structure.io.OutputFormatRDD.writeMemoryRDD(OutputFormatRDD.java:65) at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$28(SparkGraphComputer.java:271) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) </error>

[sjudeng]

That's a known bug. I think the correct fix is to read gremlin.hadoop.outputLocation and create a new temporary directory based on that during test setup. I have the fix in the below branch, though it's part of a wider refactoring to use an abstract base class for these tests to support HBase tests.

https://github.com/ngageoint/janusgraph/blob/hbase-hadoop-fix/janusgraph-hadoop-parent/janusgraph-hadoop-core/src/test/java/org/janusgraph/hadoop/CassandraInputFormatIT.java

If you're okay with a quicker temporary solution/hack you could just add a setup method and remove the output directory.

https://github.com/ngageoint/janusgraph/blob/elasticsearch-geoshape-upgrade.orig/janusgraph-hadoop-parent/janusgraph-hadoop-core/src/test/java/org/janusgraph/hadoop/CassandraInputFormatIT.java#L53-L56

[ArchaB]

Thanks @sjudeng .
Will try on removing the output directory.

[sjudeng]

FYI the fix described above has now been merged into master so hopefully this issue is resolved.

[ArchaB]

Thanks @sjudeng for the intimation.
I tried to build the latest Janusgraph code, but it build failed for the 'JanusGraph-Core: Core Library for JanusGraph' package.

Failure seen:

[ERROR] Failed to execute goal on project janusgraph-core: Could not resolve dependencies for project org.janusgraph:janusgraph-core:jar:0.1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project janusgraph-core: Could not resolve dependencies for project org.janusgraph:janusgraph-core:jar:0.1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
        at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)
        at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
        at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project org.janusgraph:janusgraph-core:jar:0.1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
        at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:211)
        at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
        ... 23 more
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
        at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
        at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:205)
        ... 24 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
        at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
        ... 25 more
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
        at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:39)
        at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:355)
        at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[ERROR]

Am I missing something? I followed all the steps I had performed last time.

[pluradj]

@ArchaB Is your Maven configured to use a proxy? That pentho mirror doesn't seem to have the tinkerpop libraries. One option would be for you to build/install TinkerPop 3.2.3 on the same machine, then it would find it in your local repo.