Sync connections can become blocked on neo4j >=3.3, which will then cause them to be dropped due to inactivity.

[ecdavis]

References:

• An initial fix for the problem: discard unneeded query results. #108
• A description of the unusual behavior of the neo4j Python driver when this occurs: Session.__exit__ tries to sync using a broken connection. neo4j/neo4j-python-driver#301
• A description of the neo4j behavior: Bolt connection [/127.0.0.1:51026] will be closed because the client did not consume outgoing buffers for 00:15:00.000 which is not expected. neo4j/neo4j#11997

To summarize:

• When sending large quantities of data to neo4j without consuming the results of those queries, various buffers can fill up and cause the connection to get "stuck" in a send which won't succeed until a recv is completed.
• After 15 minutes in this state, neo4j will drop the connection (see stack trace from neo4j below).
• During cleanup, the Python driver will try to write to the dropped connection and this will cause a broken pipe error (see stack trace from cartography below).

A workaround for this issue is to consume or detach the results of all queries executed during sync. I would prefer to find a neater solution but this may be the only one available.

neo4j debug.log

2019-09-30 13:06:21.367+0000 ERROR [o.n.b.v.m.BoltRequestMessageReaderV3] Failed to write response to driver Bolt connection [/0:0:0:0:0:0:0:1%0:51612] will be closed because the client did not consume outgoing buffers for 00:15:00.000 which is not expected.
org.neo4j.bolt.messaging.BoltIOException: Bolt connection [/0:0:0:0:0:0:0:1%0:51612] will be closed because the client did not consume outgoing buffers for 00:15:00.000 which is not expected.
        at org.neo4j.bolt.v1.transport.ChunkedOutput.flush(ChunkedOutput.java:136)
        at org.neo4j.bolt.v1.transport.ChunkedOutput.messageSucceeded(ChunkedOutput.java:105)
        at org.neo4j.bolt.v1.messaging.BoltResponseMessageWriterV1.packCompleteMessageOrFail(BoltResponseMessageWriterV1.java:105)
        at org.neo4j.bolt.v1.messaging.BoltResponseMessageWriterV1.write(BoltResponseMessageWriterV1.java:79)
        at org.neo4j.bolt.v1.messaging.MessageProcessingHandler.onFinish(MessageProcessingHandler.java:102)
        at org.neo4j.bolt.v1.runtime.BoltStateMachineV1.after(BoltStateMachineV1.java:132)
        at org.neo4j.bolt.v1.runtime.BoltStateMachineV1.process(BoltStateMachineV1.java:97)
        at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:89)
        at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:191)
        at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:139)
        at org.neo4j.bolt.runtime.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:171)
        at org.neo4j.bolt.runtime.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$2(ExecutorBoltScheduler.java:154)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.neo4j.bolt.transport.TransportThrottleException: Bolt connection [/0:0:0:0:0:0:0:1%0:51612] will be closed because the client did not consume outgoing buffers for 00:15:00.000 which is not expected.
        at org.neo4j.bolt.transport.TransportWriteThrottle.acquire(TransportWriteThrottle.java:101)
        at org.neo4j.bolt.v1.transport.ChunkedOutput.flush(ChunkedOutput.java:132)
        ... 15 more

cartography stack trace

Traceback (most recent call last):
  ...
  File ".../lib/python3.6/site-packages/neo4j/__init__.py", line 498, in run
    self._connection.send()
  File ".../lib/python3.6/site-packages/neobolt/direct.py", line 394, in send
    self._send()
  File ".../lib/python3.6/site-packages/neobolt/direct.py", line 409, in _send
    self.socket.sendall(data)
  File "/usr/lib/python3.6/ssl.py", line 965, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.6/ssl.py", line 935, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.6/ssl.py", line 636, in write
    return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lib/python3.6/site-packages/neo4j/__init__.py", line 395, in close
    self._connection.sync()
  File ".../lib/python3.6/site-packages/neobolt/direct.py", line 505, in sync
    self.send()
  File ".../lib/python3.6/site-packages/neobolt/direct.py", line 394, in send
    self._send()
  File ".../lib/python3.6/site-packages/neobolt/direct.py", line 409, in _send
    self.socket.sendall(data)
  File "/usr/lib/python3.6/ssl.py", line 965, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.6/ssl.py", line 935, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.6/ssl.py", line 636, in write
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File ".../lib/python3.6/site-packages/neo4j/__init__.py", line 370, in __exit__
    self.close()
  File ".../lib/python3.6/site-packages/neo4j/__init__.py", line 397, in close
    ServiceUnavailable, SessionError):
NameError: name 'ServiceUnavailable' is not defined

[achantavy]

Closing as this can be solved with managed transactions. See follow-up discussion in #522, #702, and #705.