web credentials not supported

[snstanton]

Describe the bug
I am attempting to connect in a context with web credentials by setting AWS_WEB_IDENTITY_TOKEN_FILE. This works with botocore. When I try the same code in aiobotocore, it fails.

Here is a sample script that demonstrates the failure:

import asyncio
import aiobotocore
import botocore.session

# SETUP:
#
# export AWS_DEFAULT_REGION=us-west-2
# export AWS_ROLE_ARN=arn:aws:iam::<projectId>:role/<role>
# export AWS_WEB_IDENTITY_TOKEN_FILE=/path/to/web/token/file

async def good():
    client = botocore.session.get_session().create_client('s3')
    print(client.list_buckets())

async def bad():
    async with aiobotocore.get_session().create_client('s3') as client:
        print(await client.list_buckets())

async def test():
    await good()
    await bad()

asyncio.get_event_loop().run_until_complete(test())

This results in the following stack trace:

Traceback (most recent call last):
  File "bug.py", line 28, in <module>
    asyncio.get_event_loop().run_until_complete(test())
  File "/Users/scotts/.pyenv/versions/3.8.0/lib/python3.8/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "bug.py", line 25, in test
    await bad()
  File "bug.py", line 20, in bad
    print(await client.list_buckets())
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/aiobotocore/client.py", line 76, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/aiobotocore/client.py", line 96, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/aiobotocore/endpoint.py", line 68, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/signers.py", line 152, in sign
    auth = self.get_auth_instance(**kwargs)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/signers.py", line 232, in get_auth_instance
    frozen_credentials = self._credentials.get_frozen_credentials()
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/credentials.py", line 591, in get_frozen_credentials
    self._refresh()
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/credentials.py", line 486, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/credentials.py", line 502, in _protected_refresh
    metadata = self._refresh_using()
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/credentials.py", line 643, in fetch_credentials
    return self._get_cached_credentials()
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/credentials.py", line 654, in _get_cached_credentials
    self._write_to_cache(response)
  File "/Users/scotts/.pyenv/versions/test38/lib/python3.8/site-packages/botocore/credentials.py", line 679, in _write_to_cache
    self._cache[self._cache_key] = deepcopy(response)
  File "/Users/scotts/.pyenv/versions/3.8.0/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'coroutine' object

pip freeze results
aiobotocore==0.12.0
aiohttp==3.6.2
aioitertools==0.5.1
async-timeout==3.0.1
attrs==19.3.0
botocore==1.15.15
chardet==3.0.4
docutils==0.15.2
idna==2.9
jmespath==0.9.5
multidict==4.7.5
python-dateutil==2.8.1
six==1.14.0
urllib3==1.25.8
wrapt==1.12.1
yarl==1.4.2

Environment:

• Python Version: 3.8.0
• OS name and version: Darwin Kernel Version 18.7.0

[snstanton]

It turns out the primary culprit is in AioEndpoint._send_request where it calls self.create_request. This is entering synchronous botocore code that ultimately needs to refresh the credentials in order to sign the request. At that point it tries to make an sts call, which re-enters the async session resulting in a coroutine result from a synchronous call. A full fix for this would require an async implementation of create_request down to that point, or a retry loop with an exception to force the refresh to happen back in the async code.

I worked around the problem by using a non-aio botocore session to refresh credentials with the following subclass:

class FixedAioSession(aiobotocore.AioSession):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.boto_session = botocore.session.get_session()

    def get_credentials(self):
        return self.boto_session.get_credentials()

This is a kludge, since it involves making a synchronous botocore api call in the middle of async code, but it seemed to get the job done in my case.

[thehesiod]

we're working on this, see #619 + #659 with https://gist.github.com/thehesiod/05298c1c89c7b5a38da0abc4ccbed7b7 workaround until we get this done

[terricain]

@snstanton you mind testing the changes in #659, in theory the web identity assume role auth should work with those changes

[snstanton]

I have applied the patch and can confirm that the test case above does appear to work now. Thanks for the update!

[terricain]

Nice! We'll merge the changes soon as I think we're pretty much there