NullPointerException during CommitAsync

[stephane-debart]

Hi,
We are facing some NullPointerExceptions during commit in our application.

java.lang.NullPointerException: null at org.neo4j.driver.internal.async.ResultCursorsHolder.retrieveFailure(ResultCursorsHolder.java:51) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:474) at org.neo4j.driver.internal.async.ResultCursorsHolder.retrieveNotConsumedError(ResultCursorsHolder.java:45) at org.neo4j.driver.internal.ExplicitTransaction.commitAsync(ExplicitTransaction.java:171)

We are using:
Neo4j 3.4.8 Enterprise Docker image.
java driver 1.6.3

All cypher queries are executed through transaction.runAsync.
The problem seems to occur when we are performing several parallel updates (of different nodes) in the same transaction.
All updates are successful but is some cases the commitAsync fails as mentioned.
Unfortunately we have not found a systematic repro, but the failure probability seems to increase with the number of parallel updates.

More detail available here: https://community.neo4j.com/t/errors-during-commitasync/3200

Stéphane Debart

[lutovich]

Hi @stephane-debart,

What do you mean by parallel updates? Transaction objects are not thread-safe and should not be used concurrently by multiple threads. Could you please share the code?

[stephane-debart]

Obviously we missed the point that transaction objects are not thread-safe and should not be used concurrently by multiple threads : this is exactly what we are doing through the runAsync method.

Looking back the documentation, this is only mentioned on Session object. This should probably be indicated explicitly on transactions and especially in javadoc of "Async" methods as these methods become then quite dangerous to use in asynchronous apps.

Stéphane

[zhenlineo]

Hi @stephane-debart
We are glad to know your problem is sorted out. Thanks for your feedback. We will improve docs accordingly.
Cheers,
Zhen

[stephane-debart]

Just an additional question: does this also mean that read-only queries must not be done concurrently on a transaction ?
This would definitely be a show stopper...

[zhenlineo]

Hi @stephane-debart
Neither Session or Transaction is thread safe.
https://github.com/neo4j/neo4j-java-driver#thread-safety

What is your usecase to have transactions across multiple threads? Why not just each thread start and commit its own transaction? What you want to achieve to access to one transaction in multiple threads? Could you share your cypher queries and how a transaction are used among multiple threads in your system?

Cheers,
Zhen

[stephane-debart]

Maybe I was just misunderstanding...
I want to have several concurrent queries open on the same transaction and their results consumed by different threads.
But if I understand correctly now, this doesn't mean executing transaction.runAsync in several threads. I can execute several transaction.runAsync in one thread and then consume their results in other thread as soon as they are available.
And the only things which become sequential are the calls to runAsync and not the full queries.
Am I right ?

[zhenlineo]

You will not get any performance benefits by consuming results in different threads, as the results of different runs arrives linearly in one transaction. That is to say, the result of second run will not arrive until the first one is fully arrived.

[stephane-debart]

Ok, we'll ensure our transactions are no more used by several threads concurrently and we'll see if there's any impact on performance.
Thanks for your answers.