Aioboto3 not working on eks.

[abhiamnyuMt]

• Async AWS SDK for Python version:

aioboto3==6.4.1
aiobotocore==0.10.4
boto3==1.9.252
botocore==1.12.252

• Python version:

Python 3.7.1

• Operating System:

Docker Image python:3.7.1-alpine3.8

Description

We are using aioboto3==5.0.0 for reading files from s3 bucket inside Kubernetes(EKS) pod. Now we wanted to try out IAM Role for Service Account (IRSA) and as aioboto3==5.0.0 doesn't support IRSA check here. We updated aioboto3==5.0.0 to aioboto3==6.1.0 and then tried all the version above, but it's not working.

What I Did

import asyncio
import aioboto3
async def get_file_text(key):
    """
    get text present inside of file with s3_path
    :param s3_bucket:exit string s3 bucket name
    :param s3_path: string s3 path
    :return: string
    """
    async with aioboto3.resource('s3', endpoint_url="http://s3-ap-southeast-1.amazonaws.com") as s3:
        s3_object = s3.Object("call-ai-staging-notification-templates",
                              Key)
        return (await (await s3_object.get())['Body'].read()).decode("utf-8")

asyncio.get_event_loop().run_until_complete(get_file_text(key))

ERROR:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "<stdin>", line 11, in get_file_text
  File "/usr/local/lib/python3.7/site-packages/aioboto3/resources.py", line 292, in do_action
    response = await action.async_call(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/aioboto3/resources.py", line 65, in async_call
    response = await getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 92, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 113, in _make_request
    request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 226, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 149, in sign
    auth = self.get_auth_instance(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 229, in get_auth_instance
    frozen_credentials = self._credentials.get_frozen_credentials()
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 577, in get_frozen_credentials
    self._refresh()
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 472, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 488, in _protected_refresh
    metadata = self._refresh_using()
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 629, in fetch_credentials
    return self._get_cached_credentials()
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 640, in _get_cached_credentials
    self._write_to_cache(response)
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 665, in _write_to_cache
    self._cache[self._cache_key] = deepcopy(response)
  File "/usr/local/lib/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle coroutine objects

[terricain]

Ugh this an evil one. It'll need to be fixed in aiobotocore basically by aio-libs/aiobotocore#659 - once I've dealt with some patches here, i'll look into helping the aiobotocore guys out

[terricain]

Yeah so digging deeper. It gets here - https://github.com/boto/botocore/blob/99a72e5cc689c86419f3cc0d6847da3d8f425148/botocore/credentials.py#L856 and as self._client_creator will return an async version of the botocore client, client.assume_role_with_web_identity would return a coroutine instead of the credentials its expecting hence deepcopy fails. Basically we gotta patch pretty much the entire file... I'll have a think if it can be worked around

[terricain]

So aio-libs/aiobotocore#659 should be at a point where in theory it'll solve that error you had, do you mind seeing if you can take that pr's code, and see if a simple aiobotocore example works inside eks?

[abhiamnyuMt]

aio-libs/aiobotocore#659
Let me check if it works.

[terricain]

Bear in mind its has 0 testing, so I would not be surprised if you get even more exceptions :D

[terricain]

Is now tracked here - aio-libs/aiobotocore#779

[fractal-joe]

any progress on this?

[terricain]

This should be solved