Question: moto_service s3 becoming slow and timing out as tests continue

[gmossessian]

I have a test suite of ~1000 tests. Previously we were using the mock_s3 decorator which only worked because our "integration tests" just imported modules from other services and mocked their functionality. This was always been undesirable but has not been a priority to fix until recently.

I'm attempting to use moto in standalone server mode, hackily starting the service in conftest.py using

def start_local_moto_server():
    global moto_subprocess
    moto_subprocess = subprocess.Popen(
        ['moto_server', 's3'],
        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        bufsize=0, universal_newlines=True,
    )
    print 'Waiting for moto local service to start...'
    while True:
        moto_log = moto_subprocess.stderr.readline()
        print moto_log
        if 'Running' in moto_log:
            break

    boto3.resource(
        's3', region_name='us-east-1', endpoint_url='http://127.0.0.1:5000'
    ).create_bucket(Bucket=TEST_S3_BUCKET_NAME)

As the tests progress, any calls to the s3 client become slower and slower, until finally I get

ReadTimeoutError: Read timeout on endpoint URL: "http://localbox:5000/my-bucket/my/key/file.tar"

Running those tests individually causes all of them to pass.

I don't really know where to begin debugging this or understanding the issue, or how to provide more information. Could someone please help identify the problem?

[gmossessian]

(cross-posting to SO here)

[spulec]

For the most part, moto will store S3 files in memory. It looks like anything greater than 16MB, we will flush to disk.

If you continue to write files under 16MB to moto s3, then the slowness would make sense.

A couple options:

• do you need to persist the same s3 backend for each test? If not, you can issue a reset to the backend to clear it (http://localhost:5000/moto-api/reset)
• Maybe our 16MB is a bad config and we should lower it

[gmossessian]

That did help with the speed issues! Thanks.

I wonder if that 16mb number could be configurable, as well as the total memory used before flushing all files to disk?

[spulec]

Yeah, we can make the individual file size to flush configurable, but we don't have an easy way to make this global.

If anyone has time to make a PR to do that, I would be happy to accept, but otherwise am going to close this out for now. Let me know if there is anything else!