Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TINKERPOP-1493 Groovy project doesn't build on Windows #457

Merged
merged 26 commits into from
Dec 2, 2016
Merged

TINKERPOP-1493 Groovy project doesn't build on Windows #457

merged 26 commits into from
Dec 2, 2016

Conversation

pauljackson
Copy link

Removed support for user.dir property as it was being prepended to a
fully qualified path and the second drive letter was making the path
illegal.

Made sure JarFile instances were being closed so that Groovy could
delete the directory without encountering file locked errors.

Exclude Unix scripts from RAT plugin

Tinkergraph integration tests fail but can build when skipping
integration tests. Hadoop fails either way. Could be environmental on my
end.

PaulJackson123 and others added 12 commits October 12, 2016 22:41
Removed support for user.dir property as it was being prepended to a
fully qualified path and the second drive letter was making the path
illegal.

Made sure JarFile instances were being closed so that Groovy could
delete the directory without encountering file locked errors.

Exclude Unix scripts from RAT plugin

Tinkergraph integration tests fail but can build when skipping
integration tests. Hadoop fails either way. Could be environmental on my
end.
Cherry-picked d0469a1 to backport from master and 3.2.3. CTR
These were requiring graphs to support integer and they were not marked as such. While most graphs support int, the Bolt protocol on neo4j does not so it caused some trouble over there. CTR
Strangely this method was deprecated back in 3.1.1, but the annotation was not added (just the javadoc). CTR
@pluradj
Copy link
Member

pluradj commented Oct 29, 2016

I verified multiple platforms to ensure the changes didn't have any adverse affects.

  • On Docker, this builds cleanly with docker/build.sh -t -n -i
  • On Mac, this builds cleanly with mvn clean install -DskipIntegrationTests=false
  • On Windows, this builds cleanly with mvn clean install -DskipTests. Following it up with mvn clean install -DskipIntegrationTests=false succeeds all the way through hadoop-gremlin, and then previously witnessed build failures on Windows start occurring in spark-gremlin. Skipping past spark-gremlin and giraph-gremlin, this builds cleanly to completion with mvn clean install -DskipIntegrationTests=false -DincludeNeo4j -rf :neo4j-gremlin.

Code changes look good to me.

VOTE: +1

@pluradj
Copy link
Member

pluradj commented Oct 29, 2016

I didn't encounter any failures with the TinkerGraph integration tests on Windows.

@spmallette
Copy link
Contributor

i'm having a tough time getting this to build with docker. not sure why....keep getting failures that seem unrelated to this PR (dies in spark).

java.lang.IllegalStateException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 4226.0 failed 1 times, most recent failure: Lost task 1.0 in stage 4226.0 (TID 4939, localhost): java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201610302243_9163_r_000001_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
    at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep.processNextStart(ComputerResultStep.java:81)
    at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:126)
    at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:37)
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:157)
    at org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.SubgraphStrategyProcessTest.shouldFilterEdgeCriterion(SubgraphStrategyProcessTest.java:166)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 4226.0 failed 1 times, most recent failure: Lost task 1.0 in stage 4226.0 (TID 4939, localhost): java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201610302243_9163_r_000001_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1914)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1055)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:938)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:930)
    at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:809)
    at org.apache.tinkerpop.gremlin.spark.structure.io.OutputFormatRDD.writeMemoryRDD(OutputFormatRDD.java:65)
    at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$1(SparkGraphComputer.java:271)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201610302243_9163_r_000001_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

@pluradj
Copy link
Member

pluradj commented Oct 31, 2016

@spmallette are you hitting that with docker/build.sh -t also or only with integration tests?

@spmallette
Copy link
Contributor

i didn't try with just -t - pretty sure it is just integration tests though. i found what i think is a fix to my problem though - i removed all my docker images and had them rebuild and i just got my first good build on docker with -t -i in a long time. i should be back to running it on this PR pretty soon. i have two separate systems doing builds right now on two different branches. hopefully they will both pass given the rebuilding of docker images.

While this is an improvement, it really sorta fixes a bug that can prevent certain tests from passing. By returning a confirmation of session close, it ensures that certain tests will always pass without having to rely on timers to simulate the situation. This change should be non-breaking to driver providers as to this point they had to have been doing simple fire-and-forget. CTR
@pluradj
Copy link
Member

pluradj commented Nov 5, 2016

any update on your Docker build @spmallette? still works ok for me.

@spmallette
Copy link
Contributor

Sorry to say I'm still getting errors.

Results :

Tests in error: 
  GroupTestV3d0$Traversals>GroupTestV3d0.g_V_group_byXoutE_countX_byXnameX:139->AbstractGremlinTest.printTraversalForm:252 ? IllegalState
  SideEffectCapTest$Traversals>SideEffectCapTest.g_V_chooseXlabel_person__age_groupCountXaX__name_groupCountXbXX_capXa_bX:64->AbstractGremlinTest.printTraversalForm:252 ? IllegalState
  GroovyHasTest$Traversals>HasTest.g_V_hasXname_markoX:135->get_g_V_hasXname_markoX:54 ? IllegalState
  GroovyHasTest$Traversals>HasTest.g_V_hasXage_gt_30X:152->get_g_V_hasXage_gt_30X:95 ? IllegalState
  GroovyGroupCountTest$Traversals>GroupCountTest.g_V_filterXfalseX_groupCount:142->get_g_V_hasXnoX_groupCount:59 ? IllegalState

You might want to rebase this on the latest from tp31, but i'm not so sure that will help as my problems are still in Spark. Here's a sample of the stacktrace from one of the errors:

g_V_filterXfalseX_groupCount(org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.GroovyGroupCountTest$Traversals)  Time elapsed: 0.215 sec  <<< ERROR!
java.lang.IllegalStateException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6870.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6870.0 (TID 8212, localhost): java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201611051613_15726_r_000000_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
    at org.apache.tinkerpop.gremlin.process.traversal.util.TraversalScriptHelper.compute(TraversalScriptHelper.java:46)
    at org.apache.tinkerpop.gremlin.process.traversal.util.TraversalScriptHelper.compute(TraversalScriptHelper.java:57)
    at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:228)
    at org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.GroovyGroupCountTest$Traversals.get_g_V_hasXnoX_groupCount(GroovyGroupCountTest.groovy:59)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6870.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6870.0 (TID 8212, localhost): java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201611051613_15726_r_000000_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1914)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1055)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:938)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
    at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:930)
    at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:809)
    at org.apache.tinkerpop.gremlin.spark.structure.io.OutputFormatRDD.writeMemoryRDD(OutputFormatRDD.java:65)
    at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$1(SparkGraphComputer.java:271)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201611051613_15726_r_000000_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

could just be environmental in some way, but i don't seem to get this on the tp31 branch itself. I just had a clean build of that this morning.

@dkuppitz could you give this a shot? Also, recognize that exception i pasted? it seems familiar to me. did we ever diagnose that?

@pluradj
Copy link
Member

pluradj commented Nov 9, 2016

I retested again overnight after rebasing the PR on tp31 and still having success with Docker build on Ubuntu and Mac. Looking at the code changes, I don't see how they would cause that Spark failure.

@spmallette
Copy link
Contributor

I finally got a good build on a different system this morning. Of course, I'm still in the weird position of the head of tp31 working on the failing system and consistent failures on this PR. I wonder what that means.

let's see what @dkuppitz says on this when he gets a chance to run it.

@dkuppitz
Copy link
Contributor

dkuppitz commented Nov 9, 2016

I just started a fresh build. Regarding the stacktraces: Wasn't the mkdirs() errors related to your outdated Docker version?

@spmallette
Copy link
Contributor

I remember that we talked about that, but I thought we'd said this version was ok

$ docker --version
Docker version 1.12.3, build 6b644ec

no?

@dkuppitz
Copy link
Contributor

dkuppitz commented Nov 9, 2016

Yep, this one is good (that's the one I have as well and I've never seen mkdirs() errors). I thought it only happened on the version you've had before.

This was the related Docker ticket: moby/moby#1295 (closed, but it seems that people occasionally still run into it).

@dkuppitz
Copy link
Contributor

dkuppitz commented Nov 9, 2016

All good here, docker/build.sh -t -i -n succeeded 2 times in a row for me.

VOTE: +1

@spmallette
Copy link
Contributor

I'm still in a position where tp31 is fine on docker but this PR is not. I rely on docker pretty heavily, so until I figure out what's wrong with my environment, I'm hesitant to +1 this though I admit that this change really shouldn't affect spark at all (yet somehow it does).

@dkuppitz
Copy link
Contributor

Hmm, that's very unlikely. Check the output of docker images. The TAG column shows the Hadoop version.

@dkuppitz
Copy link
Contributor

Btw. I don't think that this error is related to any of our components. The mkdirs problem has been reported in Docker's issue tracker several times and whenever they thought they solved it, someone else came up and ran into this issue again. I bet there's something on your system that interferes with Docker. SELinux has often been mentioned when this error popped up. Do you have any security related stuff to protect your file system?

@spmallette
Copy link
Contributor

i didn't think so. I purposely didn't turn anything on that I know of.

@spmallette
Copy link
Contributor

$ sestatus -v
SELinux status:                 disabled

@spmallette
Copy link
Contributor

I think we can get this merged once #498 goes in as it solves my problems with the build it seems. @pauljackson i will ping back here when that merges and request an additional rebase from you that I can then test. Thanks for your patience on this one.

@spmallette
Copy link
Contributor

Looks like #498 has merged. @pauljackson if you could kindly rebase this on tp31 for me I can give it a final test.

@pauljackson
Copy link
Author

Sorry, I’m fluent in git (shame). From reading, I gleaned that rebase is a type of merge from one branch to another. Since I created the pull request on tp31 I don’t understand what it means to rebase it on tp31. I’m happy to follow directions if you could give me more explicit steps.

Thanks,
-Paul

From: stephen mallette [mailto:notifications@github.com]
Sent: Saturday, November 19, 2016 7:52 AM
To: apache/tinkerpop
Cc: Paul A. Jackson; Mention
Subject: Re: [apache/tinkerpop] TINKERPOP-1493 Groovy project doesn't build on Windows (#457)

Looks like #498#498 has merged. @pauljacksonhttps://github.com/pauljackson if you could kindly rebase this on tp31 for me I can give it a final test.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//pull/457#issuecomment-261712200, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAUcSOuRaIqzUeUX5cxZ2PqRUqWOhi8Iks5q_vDngaJpZM4KVbcO.


@spmallette
Copy link
Contributor

well - your tp31 is now out of date because a number of changes (including the one that i need to test this branch with) have been added to that branch that are not in yours. If you were working in our repository I would say that to get those changes you should rebase which will effectively bring in the new changes and more your commits to the head of all those changes.

But it occurs to me now that you are working from a fork and perhaps you just need to merge upstream/tp31 (our repo - i assume you called the remote "upstream") into your fork. I don't think you need to rebase your fork.

@robertdale
Copy link
Member

Either one appears to work cleanly. Just update first - git fetch --all.
Obviously, make sure you're on your branch git checkout tp31.
Then either git rebase upstream/tp31 or git merge upstream/tp31.
Finally, push it back to your repo. git push or git push origin tp31 or however it's named.

@spmallette
Copy link
Contributor

good to know they both work. i sorta figured merge would be ok in this case as github would just figure out the diff and only include his one commit. when we merged that to our stuff, it wouldn't mess up history as it came from a fork. though i wonder if merge would bring an ugly "merge" commit with it. if it did, then i was wrong and git rebase would have been the way to go.

@pauljackson since @robertdale confirmed that git rebase is good, perhaps you should go that route to keep our history neat.

Removed support for user.dir property as it was being prepended to a
fully qualified path and the second drive letter was making the path
illegal.

Made sure JarFile instances were being closed so that Groovy could
delete the directory without encountering file locked errors.

Exclude Unix scripts from RAT plugin

Tinkergraph integration tests fail but can build when skipping
integration tests. Hadoop fails either way. Could be environmental on my
end.
@pauljackson
Copy link
Author

pauljackson commented Nov 27, 2016 via email

@robertdale
Copy link
Member

I didn't consider that your remote branch was behind. Duh. What should have been done is a forced push git push -f origin tp31 to replace your remote branch with your local branch. Instead git ended up doing a merge and the history here looks a little weird because it pulled in all the changes since your changes. But git does the right thing and it merges cleanly on tp31.

Build passes on Linux. Can't test on windows.

VOTE: +0.5

@pauljackson
Copy link
Author

pauljackson commented Nov 27, 2016 via email

@robertdale
Copy link
Member

@pauljackson It's all good. It's only the history here in the PR that looks weird. Since it's just the one commit, it'll be right in git history when it's actually merged.

@spmallette
Copy link
Contributor

Well - this isn't a rebase - it is a merge. See the ugly merge commit here: d49f692

For further explanation:

D:\Data\TEMP\@Other\tinkerpop>git push origin tp31
To https://github.com/pauljackson/tinkerpop.git
! [rejected]        tp31 -> tp31 (non-fast-forward)
error: failed to push some refs to 'https://github.com/pauljackson/tinkerpop.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

So you had it right up to here but then probably got rattled by the message above and decided to then do the following:

D:\Data\TEMP\@Other\tinkerpop>git pull
Merge made by the 'recursive' strategy.

which will do a git merge behind the scenes. Your git push command should have been:

git push origin tp31 --force

The --force is required because you have no re-written the git history for tp31 (i.e. you moved your commit forward to the head of tp31) and the server rejected it - "force" is a way of overriding that safeguard. So - anyway - this could be fixed, but i'm not going to worry about it. If only two commits come over in the merge (i.e. your original work and the merge commit) then I suppose that this is fine to merge.

@pluradj
Copy link
Member

pluradj commented Nov 29, 2016

I reconfirmed with the latest updates and reaffirm my +1 vote from before.

  • On Docker, this builds cleanly with docker/build.sh -t -n -i
  • On Mac, this builds cleanly with mvn clean install -DskipIntegrationTests=false
  • On Windows, this builds cleanly with mvn clean install -DskipTests
    • Following it up with mvn clean install -DskipIntegrationTests=false succeeds all the way until hadoop-gremlin
[INFO] Apache TinkerPop ................................... SUCCESS [ 11.578 s]
[INFO] Apache TinkerPop :: Gremlin Shaded ................. SUCCESS [  3.609 s]
[INFO] Apache TinkerPop :: Gremlin Core ................... SUCCESS [ 45.703 s]
[INFO] Apache TinkerPop :: Gremlin Test ................... SUCCESS [ 15.797 s]
[INFO] Apache TinkerPop :: Gremlin Groovy ................. SUCCESS [ 59.453 s]
[INFO] Apache TinkerPop :: Gremlin Groovy Test ............ SUCCESS [ 10.000 s]
[INFO] Apache TinkerPop :: TinkerGraph Gremlin ............ SUCCESS [04:31 min]
[INFO] Apache TinkerPop :: Hadoop Gremlin ................. FAILURE [06:13 min]

Although this doesn't completely solve TinkerPop builds on Windows, it passes a lot more than it used to.

@spmallette
Copy link
Contributor

All tests pass with docker/build.sh -t -n -i

VOTE +1

@pluradj
Copy link
Member

pluradj commented Dec 1, 2016

We have enough votes to commit this. I'll handle it today. Thanks everyone.

@asfgit asfgit merged commit d49f692 into apache:tp31 Dec 2, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants