A question about stream.read()

[llnancy]

        # get object from s3
        response = await client.get_object(Bucket=bucket, Key=key)
        # this will ensure the connection is correctly re-used/closed
        async with response['Body'] as stream:
            assert await stream.read() == data

For large files (more than GB), stream.read() will cause OOM. Are there any examples of segmented reading?

[jakob-keller]

Awaiting stream.read() will read the whole response’s body as bytes, potentially resulting in OOM issues.

You may want to look at stream.content and the StreamReader reading methods.

I hope this points you in the right direction.

[thehesiod]

there's also the StreamingBody helpers:

aiobotocore/aiobotocore/response.py

Line 19 in 9c6db8a

class StreamingBody(wrapt.ObjectProxy):

which mimics the version from botocore available via response['Body']