Saving the boto3 resource with aioboto3 8.x

[mvalkon]

After the upgrade of aiobotocore to v 1.x.x, the documentation suggests that the aioboto3 resource should be used with an async context manager. This raises a small concern on my side about saving the resource.

My use case is one in which I run a web server with aiohttp, and need to access dynamodb for every request made. Previously I was able to initiate the resource and subsequently the Table-object in my database abstraction object, which was then stored in the aiohttp request context at application startup. I would then call table.close() at application shutdown and cleanup. Creating a new resource and table object for each request is too slow.

It would be beneficial to add some documentation on how real world use cases are supported. Is there a way to maintain the Table-object with the new changes?

[terricain]

😄 I was waiting for an issue like this to appear. Yeah it can be done, https://docs.python.org/3/library/contextlib.html#contextlib.AsyncExitStack should be used as well, I'll look into making an aiohttp example this evening

[mvalkon]

Thanks @terrycain, much appreciated.

[terricain]

@mvalkon https://aioboto3.readthedocs.io/en/latest/usage.html#aiohttp-server-example does that help?

[mvalkon]

Thanks a lot @terrycain, good addition to the documentation.

[iameugenejo]

Can we please get a simpler approach for avoiding the use context manager at all? I believe long running application is a major use case as aiohttp is used for web servers.

[terricain]

Not really, because we're pretty much wrapping boto3 and aiobotocore, it is what it is. If you dont care about closing resources, you can just await aioboto3.resource(...).__aenter__() which mimics what async with does.

[julianogv]

I have a web app running inside an EC2 with an IAM Role attached to it and after some time running with the same client initialized with await aioboto3.client(...).__aenter__() it starts raising the error below.

@terrycain do you have any ideia if this error is related to not using the async with and using the same client all the time?

Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "/virtualenv/lib/python3.7/site-packages/aiobotocore/credentials.py", line 291, in _protected_refresh
    metadata = await self._refresh_using()
  File "/virtualenv/lib/python3.7/site-packages/aiobotocore/utils.py", line 104, in retrieve_iam_role_credentials
    token = await self._fetch_metadata_token()
  File "/virtualenv/lib/python3.7/site-packages/aiobotocore/utils.py", line 49, in _fetch_metadata_token
    async with session.put(url, headers=headers) as resp:
  File "/virtualenv/lib/python3.7/site-packages/aiohttp/client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "/virtualenv/lib/python3.7/site-packages/aiohttp/client.py", line 497, in _request
    await resp.start(conn)
  File "/virtualenv/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 844, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/virtualenv/lib/python3.7/site-packages/aiohttp/streams.py", line 588, in read
    await self._waiter
concurrent.futures._base.CancelledError

I'm using
aioboto3==9.2.2
aiobotocore==1.4.2
boto3==1.17.106
botocore==1.20.106

[terricain]

Its possible this is fixed in aioboto3 9.5.0, but there shouldn't be an issue reusing the same client. Realistically the part that fetches the metadata token should retry. If 9.5.0 doesn't work, see if you can replicate the issue just using the latest aiobotocore (also reusing the same client), as thats the library that handles all of the credentials logic.