Unable to locate credentials

[potatochip]

Describe the bug
I get the following exception when awaiting the s3_put coro below. This only happens intermittently when creating a k8s pod running the app. Credentials come from an iam role assigned to the pod. The pod will restart a couple times and eventually the error will not raise. The error did no raise in the same environment when using version 0 of aiobotocore (with a persistent client rather than context managed client). Maybe this is related to a change in pinned botocore version?

_REGION = None
_SESSION = None


def get_aws_region() -> str:
    """Get the aws region of the current ec2 machine."""
    global _REGION
    if _REGION is None:
        try:
            _REGION = ec2_metadata.region
        except (ConnectionError, ConnectTimeout):
            _REGION = 'us-east-1'
    return _REGION


def get_s3_client() -> aiobotocore.client.AioBaseClient:
    """Get async context client for s3."""
    global _SESSION
    if _SESSION is None:
        _SESSION = aiobotocore.get_session()
    return _SESSION.create_client(
        's3',
        region_name=get_aws_region(),
        config=Config(signature_version='s3v4'))


async def s3_put(bucket: str, key: str, body: str) -> None:
    """Async put body into key on s3.

    This is just a wrapper around put_object enforcing encryption.
    """
    async with get_s3_client() as client:
        await client.put_object(Bucket=bucket,
                                Key=key,
                                Body=body,
                                ServerSideEncryption='AES256') 

Traceback (most recent call last):
  File "x", line 45, in s3_put
    await client.put_object(Bucket=bucket,
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/client.py", line 111, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/client.py", line 131, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/endpoint.py", line 84, in _send_request
    request = await self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/endpoint.py", line 77, in create_request
    await self._event_emitter.emit(event_name, request=request,
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/hooks.py", line 27, in _emit
    response = await handler(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/signers.py", line 16, in handler
    return await self.sign(operation_name, request)
  File "/usr/local/lib/python3.8/site-packages/aiobotocore/signers.py", line 61, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.8/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

pip freeze results

aiobotocore==1.0.4
aiohttp==3.6.2
aioitertools==0.6.1
aioredis==1.3.1
async-timeout==3.0.1
asyncssh==2.1.0
attrs==19.3.0
boto3==1.12.32
botocore==1.15.32
cached-property==1.5.1
certifi==2020.4.5.1
cffi==1.14.0
chardet==3.0.4
cryptography==2.9.2
datadog==0.35.0
decorator==4.4.2
docutils==0.15.2
ec2-metadata==2.1.0
hiredis==1.0.1
idna==2.9
jmespath==0.9.5
multidict==4.7.5
plac==1.1.3
psycopg2-binary==2.8.5
pycparser==2.20
python-dateutil==2.8.1
python-json-logger==0.1.11
requests==2.23.0
s3transfer==0.3.3
six==1.14.0
SQLAlchemy==1.3.16
street-cred==0.0.2
urllib3==1.25.9
wrapt==1.12.1
yarl==1.4.2

Environment:

• Python Version: 3.8.2
• OS name and version: Debian GNU/Linux 10 (buster)

[thehesiod]

you can try the same code with a sync version of the same botocore to see if it's a botocore or aiobotocore issue

[thehesiod]

btw what do you mean by using version 0 of aiobotocore ? @terrycain any ideas?

[thehesiod]

btw @potatochip could you post an example using containers or something fully fleshed out? also can you test against pre 1.x version of aiobotocore to see if this is an injection?

[terricain]

Is something to do with the kube iam assume role thing I would guess. We'd need a way to reproduce it, my guess is it's triggered by a few env vars and files placed on the container. We'd need those (or at least those with the values somewhat anonymised)

[potatochip]

Pre 1.x did not experience this. Only started after upgrading to 1.x

[thehesiod]

@potatochip pre 1.x credential refresh was broken, so probably why you're starting to see this issue. If you could give us a full example with docker containers via moto or some cloudformation for us to repro that would be best otherwise there's no way for us to test.

[potatochip]

I understand. I think best to close then. I can’t replicate it reliably.

[potatochip]

For future viewers, I’m attempting to solve it by increasing the timeout / number of retries botocore allocates to connecting to the metadata server.

[thehesiod]

btw at FBN we're running into an issue where botocore first tries to use retrieve the a task based IAM, times out, then falls back to the EC2 IAM. So that scenario seems like it's expected via botocore. I'm going to close for now but please keep us apprised and let us know if this is specific to aiobotocore or happens with botocore as well. I'm interested either way. Thanks! btw metadata retries/timeout is deep in botocore, it has separate constants for the metadata calls.

[thehesiod]

I'm definitely interested in ensuring we don't have a bug, so please re-open when you have more details! Probably worth setting botocore to DEBUG level logging for more details so you can see what's going on.

[fox91]

I have the same issue.

Package version that I'm using:

aioboto3==8.2.0
aiobotocore==1.1.2
aiohttp==3.7.3
boto3==1.14.44
botocore==1.17.44

I need to download one hundred files from S3 and I'm using aioboto3 (that use aiobotocore under the hood).
After the first 50-60 files (the number also varies from time to time) I get this error for some files:

Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/REDACTED: 
Traceback (most recent call last):
  File "/home/ubuntu/ENV/lib/python3.8/site-packages/aiobotocore/utils.py", line 79, in _get_request
    async with session.get(url, headers=headers) as resp:
  File "/home/ubuntu/ENV/lib/python3.8/site-packages/aiohttp/client.py", line 1117, in __aenter__
    self._resp = await self._coro
  File "/home/ubuntu/ENV/lib/python3.8/site-packages/aiohttp/client.py", line 619, in _request
    break
  File "/home/ubuntu/ENV/lib/python3.8/site-packages/aiohttp/helpers.py", line 656, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError

repeated many times and than:

Max number of attempts exceeded (1) when attempting to retrieve data from metadata service

I'm using it in the wrong way or we are hitting some limit of the metadata service?

[thehesiod]

@fox91 try with botocore instead of aiobotocore to see if it's related to aiobotocore

[terricain]

Also are you opening a new client for each file? as opening 1 client and using that for all the 60 s3 files would be better.