ForsetiClient can't acquire ExclusiveLock

[xghrbc1001]

we use neo4j 2.2.5, ha mode. batch update data from mysql to neo4j ,a lot exception as below occurs. how to avoid this exception ?
maybe it's not an issue,just a question,but also need your hope. thank you.

org.neo4j.kernel.DeadlockDetectedException: ForsetiClient[153] can't acquire ExclusiveLock{owner=ForsetiClient[159]} on NODE(1892858739), because holders of that lock are waiting for ForsetiClient[153].
 Wait list:ExclusiveLock[
Client[159] waits for [153,159]]
        at org.neo4j.kernel.ha.lock.forseti.ForsetiClient.markAsWaitingFor(ForsetiClient.java:687)
        at org.neo4j.kernel.ha.lock.forseti.ForsetiClient.acquireExclusive(ForsetiClient.java:230)
        at org.neo4j.kernel.impl.api.LockingStatementOperations.nodeRemoveProperty(LockingStatementOperations.java:331)
        at org.neo4j.kernel.impl.api.OperationsFacade.nodeRemoveProperty(OperationsFacade.java:678)
        at org.neo4j.kernel.impl.core.NodeProxy.removeProperty(NodeProxy.java:320)
        at com.xxx.neo4j.plugin.knitter.service.KnitterService.setPro(KnitterService.java:385)
        at com.xxx.neo4j.plugin.knitter.service.KnitterService.getKniNode(KnitterService.java:412)
        at com.xxx.neo4j.plugin.knitter.service.KnitterService.setServe(KnitterService.java:525)
        at com.xxx.neo4j.plugin.knitter.service.KnitterService.realWork(KnitterService.java:296)
        at com.xxx.neo4j.plugin.knitter.immediate.UpdateImmediatelyCallable.call(UpdateImmediatelyCallable.java:25)
        at com.xxx.neo4j.plugin.knitter.immediate.UpdateImmediatelyCallable.call(UpdateImmediatelyCallable.java:9)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
ERROR 2016-01-19 12:55:49 neo4j Transaction Exception!! 
org.neo4j.kernel.DeadlockDetectedException: ForsetiClient[159] can't acquire ExclusiveLock{owner=ForsetiClient[153]} on NODE(1770669171), because holders of that lock are waiting for ForsetiClient[159].
 Wait list:ExclusiveLock[
Client[153] waits for [153,159]]
        at org.neo4j.kernel.ha.lock.forseti.ForsetiClient.markAsWaitingFor(ForsetiClient.java:687)
        at org.neo4j.kernel.ha.lock.forseti.ForsetiClient.acquireExclusive(ForsetiClient.java:230)
        at org.neo4j.kernel.impl.api.LockingStatementOperations$1.visit(LockingStatementOperations.java:256)
        at org.neo4j.kernel.impl.api.store.CacheLayer.relationshipVisit(CacheLayer.java:489)
        at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.relationshipVisit(StateHandlingStatementOperations.java:1238)
        at org.neo4j.kernel.impl.api.ConstraintEnforcingEntityOperations.relationshipVisit(ConstraintEnforcingEntityOperations.java:391)
        at org.neo4j.kernel.impl.api.LockingStatementOperations.relationshipDelete(LockingStatementOperations.java:251)
        at org.neo4j.kernel.impl.api.OperationsFacade.relationshipDelete(OperationsFacade.java:633)
        at org.neo4j.kernel.impl.core.RelationshipProxy.delete(RelationshipProxy.java:155)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
 INFO 2016-01-19 12:55:49 finally  beginTx.close(): 2016-01-19 12:55:49.425 
ERROR 2016-01-19 12:55:49 realtime error !! 
org.neo4j.kernel.DeadlockDetectedException: ForsetiClient[159] can't acquire ExclusiveLock{owner=ForsetiClient[153]} on NODE(1770669171), because holders of that lock are waiting for ForsetiClient[159].
 Wait list:ExclusiveLock[
Client[153] waits for [153,159]]
        at org.neo4j.kernel.ha.lock.forseti.ForsetiClient.markAsWaitingFor(ForsetiClient.java:687)
        at org.neo4j.kernel.ha.lock.forseti.ForsetiClient.acquireExclusive(ForsetiClient.java:230)
        at org.neo4j.kernel.impl.api.LockingStatementOperations$1.visit(LockingStatementOperations.java:256)
        at org.neo4j.kernel.impl.api.store.CacheLayer.relationshipVisit(CacheLayer.java:489)
        at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.relationshipVisit(StateHandlingStatementOperations.java:1238)
        at org.neo4j.kernel.impl.api.ConstraintEnforcingEntityOperations.relationshipVisit(ConstraintEnforcingEntityOperations.java:391)
        at org.neo4j.kernel.impl.api.LockingStatementOperations.relationshipDelete(LockingStatementOperations.java:251)
        at org.neo4j.kernel.impl.api.OperationsFacade.relationshipDelete(OperationsFacade.java:633)
        at org.neo4j.kernel.impl.core.RelationshipProxy.delete(RelationshipProxy.java:155)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[spacecowboy]

Could you be more specific as to what the batch update is doing?

• What cypher queries are you using?
• Are you writing to only the master?
• Are other transactions executing during the batch update?

[xghrbc1001]

we init data from mysql to neo4j, while mysql data changes, we need to synchronize this change data to neo4j.

we use neo4j plugin to update neo4j data,no cypher. like this:

try(Transaction beginTx=graphDatabaseService.beginTx()){
    Node nodeFrom=db.getNodeById(graphIdFrom);
    Node nodeTo=db.getNodeById(graphIdFrom);
    nodeTo.createRelationshipTo(nodeTo,RelationType.OWN);
    beginTx.success();
}

we not only write to master,sometimes write to slave,but we found HA's data are synchronized;

we use multi thread to update,beacause single thead is too slow.

special, in our data one node may have 30000 relations.

thank you.@spacecowboy

[spacecowboy]

@xghrbc1001 Do update nodes one by one as in the example? Not in batches?

[xghrbc1001]

@spacecowboy which example,can you give me the url? thank you. neo4j is an amazing graph database, I'm Chinese.

[spacecowboy]

@xghrbc1001 as in your example. Do you actually update each node one by one? That could be a source to your problems.

Instead, you should batch them. For example:

int batchSize = 10_000;
try(Transaction beginTx=graphDatabaseService.beginTx()){
    for (int i = 0; i < batchSize; i++) {
        Node nodeFrom=db.getNodeById(graphIdFrom[i]);
        Node nodeTo=db.getNodeById(graphIdTo[i]);
        nodeTo.createRelationshipTo(nodeTo,RelationType.OWN);
    }
    beginTx.success();
}

[xghrbc1001]

yes,the code like this:

int batchSize = 10_000;
try(Transaction beginTx=graphDatabaseService.beginTx()){
    for (int i = 0; i < batchSize; i++) {
        Node nodeFrom=db.getNodeById(graphIdFrom[i]);
        Node nodeTo=db.getNodeById(graphIdTo[i]);
        nodeTo.createRelationshipTo(nodeTo,RelationType.OWN);
    }
    beginTx.success();
}

dead lock occus also.some times org.jboss.netty.handler.queue.BlockingReadTimeoutException occus

[spacecowboy]

Have a look at our documentation regarding deadlocks here:
http://neo4j.com/docs/milestone/transactions-deadlocks.html#transactions-deadlocks-loop

If you have questions on how to customize that to your needs please ask on StackOverflow

[crazyyanchao]

The probability of this problem decreases after I index the attributes of the node.