neobolt.exceptions.ServiceUnavailable in ecr.load_ecr_repository_images() when connecting to remote Neo4j

[achantavy]

Description:

What issue is being seen? Describe what should be happening instead of the bug, for example: Cartography should not crash, the expected value isn't returned, the data schema is wrong, etc.

When loading 50MB worth of data to a remote Neo4j server (i.e. not located on the same machine), ecr.load_ecr_repository_images() crashes with a neobolt.exceptions.ServiceUnavailable error after running for 2 hours.

To Reproduce:

Steps to reproduce the behavior. Provide all data and inputs required to reproduce the issue.

Run ecr.load_ecr_repository_images() with 50MB of data.

POC code:

from neo4j import GraphDatabase
import cartography.intel.aws.ecr
import time
# You will need to provide your own data here.
# data shape = [{'repo_uri': 'uri', 'repo_images': [{'imageDigest': 'mydigest', 'imageTag': 'mytag'}, ...]},...]
from image_data import image_list

neo4j_driver = GraphDatabase.driver("bolt://your-remote-endpoint:7687")
neo4j_session = neo4j_driver.session()
account_id = '1234'
aws_update_tag = int(time.time())
region = 'us-east-1'

common_job_parameters = {
    "UPDATE_TAG": aws_update_tag,
    "AWS_ID": account_id,
}

cartography.intel.aws.ecr.load_ecr_repository_images(neo4j_session, image_list, region, aws_update_tag)
cartography.intel.aws.ecr.cleanup(neo4j_session, common_job_parameters)

Logs:

If applicable, copy and paste your console log with the failing stack trace.

Traceback (most recent call last):
  File "/Users/achantavy/.pyenv/versions/3.7.9/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/Users/achantavy/.pyenv/versions/3.7.9/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 54] Connection reset by peer
Exception ignored in: 'neobolt.bolt._io.ChunkedInputBuffer.receive'
Traceback (most recent call last):
  File "/Users/achantavy/.pyenv/versions/3.7.9/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/Users/achantavy/.pyenv/versions/3.7.9/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 54] Connection reset by peer
Traceback (most recent call last):
  File "load_ecr_list_images.py", line 17, in <module>
    cartography.intel.aws.ecr.load_ecr_repository_images(neo4j_session, image_list, region, aws_update_tag)
  File "/Users/achantavy/lyftsrc/cartography/cartography/util.py", line 63, in timed
    return method(*args, **kwargs)
  File "/Users/achantavy/lyftsrc/cartography/cartography/intel/aws/ecr.py", line 122, in load_ecr_repository_images
    Region=region,
  File "/Users/achantavy/.virtualenvs/env11/lib/python3.7/site-packages/neo4j/__init__.py", line 503, in run
    self._connection.fetch()
  File "/Users/achantavy/.virtualenvs/env11/lib/python3.7/site-packages/neobolt/direct.py", line 414, in fetch
    return self._fetch()
  File "/Users/achantavy/.virtualenvs/env11/lib/python3.7/site-packages/neobolt/direct.py", line 431, in _fetch
    self._receive()
  File "/Users/achantavy/.virtualenvs/env11/lib/python3.7/site-packages/neobolt/direct.py", line 472, in _receive
    raise self.Error("Failed to read from defunct connection {!r}".format(self.server.address))
neobolt.exceptions.ServiceUnavailable: Failed to read from defunct connection Address(host={IP}, port=7687)

Please complete the following information::

• Cartography release version or commit hash [e.g. 0.12.0 or 95e8e11]

0c9a662

• Python version: [e.g. 3.7.4]

3.7.9

• OS (feel free to omit this if you don't think it's relevant to your issue): [e.g. Ubuntu bla bla, OSX bla bla]

Have observed this in a Docker container based on Debian as well as my OSX laptop. Neither of them appear to be resource constrained: CPU usage is around 0%, memory usage of the python process is about 200-300MB.

Additional context:

Add any other context about the problem here.

This appears related to

• ConnectionResetError and ServiceUnavailable exceptions thrown from ECR sync #440

• Sync connections can become blocked on neo4j >=3.3, which will then cause them to be dropped due to inactivity. #170

all of these issues involve sending fairly large objects over the Bolt connection.

---

Update:

I've also observed this issue on load_ecr_repositories().

[achantavy]

Actually going to reopen this issue as #523 does not completely resolve it.

[voutilad]

@achantavy I believe I know what the root cause is given my Neo4j experience. See my PR #526.

You were on the right track with UNWIND, but you need to still be cautious about batching. The transaction state in 3.5 by default is stored on-heap in the JVM and can create memory pressure when running large transactions. Worst case scenario is unresponsive network activity from the server causing the client to give up the connection.

(edited to explain batching since I hit comment too soon)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[achantavy]

Hey @voutilad, I put together a plan for improving write perf in this project. If you have 10 minutes would really appreciate your input: https://docs.google.com/document/d/1IZ12R3oROn11LcYj5XunokyOjJkKu-H2O1TEk065Dsk/edit#

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new change or reopen this one.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[jexp]

@achantavy I'm also happy to help if @voutilad is busy. If you want to we can have a look at the code together, just drop me an email to schedule a call.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[achantavy]

@voutilad @jexp Thanks for the help so far. I saw this issue happen on a different section of internal code and was able to resolve it by using explicit transactions!

Deployment information: we have a k8s cronjob running neo4j python driver 1.7.6, writing data to a Neo4j enterprise 3.5.19 database across an AWS Network Load Balancer.

To summarize, the code would

1. Instantiate a single neo4j session
2. Use that session to write some data to the graph
3. Do some more parsing/data transforms
4. Go to (2) until we are done

Most times, step (3) would take longer than 380 seconds and the code would work fine, which is not what I would expect because this is longer than the timeout from our AWS NLB and the value of our neo4j driver's max_connection_lifetime value. Anyway, we found that running this code with a specific set of data would cause step (4) to reliably fail with a ConnectionResetError, resulting in a neo4j.ServiceUnavailable exception.

To fix, I changed the code to explicitly use session.write_transaction() instead of auto-commit transactions with session.run() and now the code seems to work magically! I have not implemented any retry logic myself at all.

To get to this solution, I stumbled upon this section of the current driver doc: https://neo4j.com/docs/api/python-driver/current/api.html#managed-transactions-transaction-functions

[Managed transactoins] allow a function object representing the transactional unit of work to be passed as a parameter. This function is called one or more times, within a configurable time limit, until it succeeds.

Sure enough, this seemed encouraging and it worked! Prior to this I had only been reading the docs for the 1.7 driver but it seems that the docs have become more thorough for the current drivers.

I'll push out a similar fix to address this specific issue and other related ones. I guess to summarize Python driver best practices that we've learned in this project,

• Use the unwind pattern for speed and batching
• Use explicit transaction functions
• Consume the results within the transaction functions
• Be aware of max_connection_lifetime, especially if you're forced to deal with load balancers
• Ensure the size of the transaction is not too large

This has been bugging me for months and I'm glad to finally have forward movement on this problem. :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.