Is there someone has met the issue when you use AQL queries in Spark, the query is normal after a while will report 404

[HansonZhu-bd]

My Environment

• ArangoDB Version: 3.3.16
• Storage Engine: RocksDB
• Deployment Mode: Cluster
• Deployment Strategy: ArangoDB Starter
• Configuration:
• Infrastructure:
• Operating System:
• Total RAM in your machine:
• Disks in use:
• Used Package:

Component, Query & Data

Affected feature:

AQL query (if applicable):
1.

 WITH Company, Person
          FOR v, e, p IN 1..3 ANY  'Company/${md5_comp}' invest,tradable_share,suspect_same_company OPTIONS {uniqueVertices: 'path'}
          FILTER p.vertices[-1].is_belong == "true" or p.vertices[-1].is_belong == true or ${ignoreIsBelong.toString}
          RETURN p

WITH Company, Person
          FOR v, e, p IN 4 ANY  'Company/${md5_comp}' invest,tradable_share,officer,person_merge,suspect_same_company OPTIONS {uniqueVertices: 'path'}
          FILTER split(p.vertices[2]._id, "/")[0] == "Person"
          FILTER p.vertices[-1].is_belong == "true" or p.vertices[-1].is_belong == true or ${ignoreIsBelong.toString}
          RETURN p

AQL explain (if applicable):
I want to query a company for all its affiliates in 3 degrees and wtih second vertex was people all its affiliates in 4 degrees.

Problem:

INFO org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) - ShuffleMapStage 5 (distinct at BlackListQueryFromArangoV1.scala:215) failed in 325.108 s due to Job aborted due to stage failure: Task 150 in stage 5.0 failed 1 times, most recent failure: Lost task 150.0 in stage 5.0 (TID 1150, localhost, executor driver): com.arangodb.ArangoDBException: Response: 404, Error: 1600 - cursor not found
	at com.arangodb.internal.velocystream.VstCommunication.checkError(VstCommunication.java:106)
	at com.arangodb.internal.velocystream.VstCommunicationSync.execute(VstCommunicationSync.java:134)
	at com.arangodb.internal.velocystream.VstCommunicationSync.execute(VstCommunicationSync.java:46)
	at com.arangodb.internal.velocystream.VstCommunication.execute(VstCommunication.java:96)
	at com.arangodb.internal.velocystream.VstProtocol.execute(VstProtocol.java:46)
	at com.arangodb.internal.ArangoExecutorSync.execute(ArangoExecutorSync.java:58)
	at com.arangodb.internal.ArangoExecutorSync.execute(ArangoExecutorSync.java:47)
	at com.arangodb.ArangoDatabase$1.next(ArangoDatabase.java:326)
	at com.arangodb.internal.ArangoCursorIterator.next(ArangoCursorIterator.java:66)
	at com.arangodb.ArangoCursor.next(ArangoCursor.java:117)
	at com.haizhi.blacklist.BlackListQueryFromArangoV1$.expandKLevelPath(BlackListQueryFromArangoV1.scala:134)
	at com.haizhi.blacklist.BlackListQueryFromArangoV1$$anonfun$8$$anonfun$9.apply(BlackListQueryFromArangoV1.scala:199)
	at com.haizhi.blacklist.BlackListQueryFromArangoV1$$anonfun$8$$anonfun$9.apply(BlackListQueryFromArangoV1.scala:199)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
this error appearance after program run about 10 minutes.

[2019-04-22 16:44:47,394] INFO org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) - Job 2 failed: collectAsMap at BlackListQueryFromArangoV1.scala:217, took 325.633962 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 150 in stage 5.0 failed 1 times, most recent failure: Lost task 150.0 in stage 5.0 (TID 1150, localhost, executor driver): com.arangodb.ArangoDBException: Response: 404, Error: 1600 - cursor not found
	at com.arangodb.internal.velocystream.VstCommunication.checkError(VstCommunication.java:106)
	at com.arangodb.internal.velocystream.VstCommunicationSync.execute(VstCommunicationSync.java:134)
	at com.arangodb.internal.velocystream.VstCommunicationSync.execute(VstCommunicationSync.java:46)
	at com.arangodb.internal.velocystream.VstCommunication.execute(VstCommunication.java:96)
	at com.arangodb.internal.velocystream.VstProtocol.execute(VstProtocol.java:46)
	at com.arangodb.internal.ArangoExecutorSync.execute(ArangoExecutorSync.java:58)
	at com.arangodb.internal.ArangoExecutorSync.execute(ArangoExecutorSync.java:47)
	at com.arangodb.ArangoDatabase$1.next(ArangoDatabase.java:326)
	at com.arangodb.internal.ArangoCursorIterator.next(ArangoCursorIterator.java:66)
	at com.arangodb.ArangoCursor.next(ArangoCursor.java:117)
	at com.haizhi.blacklist.BlackListQueryFromArangoV1$.expandKLevelPath(BlackListQueryFromArangoV1.scala:134)
	at com.haizhi.blacklist.BlackListQueryFromArangoV1$$anonfun$8$$anonfun$9.apply(BlackListQueryFromArangoV1.scala:199)
	at com.haizhi.blacklist.BlackListQueryFromArangoV1$$anonfun$8$$anonfun$9.apply(BlackListQueryFromArangoV1.scala:199)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Expected result:
company with its all associated companys or people.

[Simran-B]

Please edit your post and fill out the template. We need to know what your environment is, what you use in which version and how this problem can be reproduced.

[HansonZhu-bd]

@Simran-B I have update my arango cluster enviroment and the error information.thank you for answer my question.

[BCaxelbecker]

@zhuhan66 please add the driver version

[OmarAyo]

Hi @zhuhan66

As @BCaxelbecker reported in his previous message, could you please add the driver version

Thanks

[HansonZhu-bd]

I'm sorry about that I reply you guys too slowly.@OmarAyo @BCaxelbecker @Simran-B

driver version is below:

com.arangodb
arangodb-java-driver
4.2.2

And there is another problem I met recently.I use the cluster mode of arango with 3 computer,but when I delete data ,only the data of the primary node can be deleted, and the data of the two duplicate nodes cannot be deleted.please help me.

[dothebart]

Hi,
sorry for the late reply. There are two reasons for Cursor not found:

• You are talking to the wrong coordinator, the cursor lives on the other. Meanwhile later Arango-Releases coordinators will forward the cursor request to the proper coordinator.
• the cursor has timeouted. You need to specify a longer TTL so it lasts until you reach out for it again.

Anyways, ArangoDB 3.3 has reached EOL meanwhile; please upgrade your environment.

Please note that ArangoDB 3.7 has been released - closing this.
If you still encounter similar issues after an upgrade, please open a new issue.