Connecting to SQS in docker after assume role/kubernetes IAM role not working

[eldarnegrinperion]

Please fill out the sections below to help us address your issue.

What issue did you see ?
logs-from-kubernetes.txt
when inside docker, can't access role assumed on computer/iam role on kubernetes
from my computer it works fine, it finds the credential and config files.
when creating s3 client all works fine. this happens only in sqs client..

Steps to reproduce
If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples.
simple python (3.7.4) code, boto3 (1.14.2), just creating a client for sqs.
if __name__ == '__main__': boto3.set_stream_logger('') sqs = boto3.client('sqs')

Debug logs
Full stack trace by adding boto3.set_stream_logger('') to your code.
here is local docker, and attached kubernetes logs file

2020-07-02 07:05:24,593 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2020-07-02 07:05:24,597 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2020-07-02 07:05:24,598 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2020-07-02 07:05:24,602 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2020-07-02 07:05:24,602 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2020-07-02 07:05:24,604 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2020-07-02 07:05:24,605 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2020-07-02 07:05:24,612 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: env
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: sso
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: config-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2020-07-02 07:05:24,634 botocore.credentials [DEBUG] Looking for credentials via: container-role
2020-07-02 07:05:24,634 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2020-07-02 07:05:24,635 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80
2020-07-02 07:05:25,646 urllib3.connectionpool [DEBUG] Starting new HTTP connection (2): 169.254.169.254:80
2020-07-02 07:05:26,660 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 342, in _get_request
    response = self._session.send(request.prepare())
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2020-07-02 07:05:26,669 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2020-07-02 07:05:26,671 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json
2020-07-02 07:05:26,681 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f503ec53b00>
2020-07-02 07:05:26,696 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/sqs/2012-11-05/service-2.json
2020-07-02 07:05:26,701 botocore.hooks [DEBUG] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f503eca0f80>
Traceback (most recent call last):
  File "EnrichmentWorkerService.py", line 88, in <module>
    sqs = boto3.client('sqs')
  File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 835, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 85, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 287, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 73, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 153, in compute_client_args
    s3_config=s3_config,
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 218, in _compute_endpoint_config
    return self._resolve_endpoint(**resolve_endpoint_kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 301, in _resolve_endpoint
    service_name, region_name, endpoint_url, is_secure)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 361, in resolve
    service_name, region_name)
  File "/usr/local/lib/python3.7/site-packages/botocore/regions.py", line 134, in construct_endpoint
    partition, service_name, region_name)
  File "/usr/local/lib/python3.7/site-packages/botocore/regions.py", line 148, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

[swetashre]

@eldarnegrinperion - Thank you for your post. It looks like this issue might be related to your network policies for the particular container. https://kubernetes.io/docs/concepts/services-networking/network-policies/
Do you have any restrictive security group for the particular container because of which it is not able to connect to IMDS ?
How have you configured your container to use assume role ?

This stack overflow post might help in debugging the issue:
https://stackoverflow.com/questions/22409367/fetching-aws-instance-metadata-from-within-docker-container

[eldarnegrinperion]

no special configuration. it happens in my local environment as well - when i run the code directly (virtualenv) all is good.
when i run it in a docker (very simple python docker, no special rules) it doesn't.
when i change the code to S3 client, all is good and working as expected
why does it happen when trying to create a sqs client?

[eldarnegrinperion]

Dockerfile.txt

[swetashre]

@eldarnegrinperion - Thanks for responding. I am not able to reproduce the issue with the Dockerfile you provided. I am assuming you are running this docker container in an ec2 instance as ec2 instance metadata can't be used from your local environment.
Are you able to use any api call with the s3 client ?
What is the response you are getting when using curl http://169.254.169.254/latest/meta-data/iam/security-credentials from your docker container ?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[eldarnegrinperion]

sorry i missed your response. i can't even load the docker as it fails and exits as the log above i sent. i will try to run the docker with just an endless loop so i can get in exec mode and do the CURL and update results here.
thanks

[eldarnegrinperion]

this is what i get when exec CURL to the above mentioned url from within the docker on my machine.

curl: (56) Recv failure: Connection reset by peer

[eldarnegrinperion]

and here's my code, pretty simple:
`
if name == 'main':
# boto3.set_stream_logger('')
print('starting polling...')
Path('/tmp/healty').touch() # health check for probe

while True:
    sleep_timer=random() * 5 * 60
    print(f'{datetime.utcnow()} loop {sleep_timer} seconds...')
    time.sleep(sleep_timer)

# sqs_polling(queue_url=settings.SQS_URL,
#             callback=sqs_message_callback,
#             max_workers=settings.MAX_WORKERS,
#             process_worker=settings.PROCESS_WORKER,
#             interval_seconds=settings.INTERVAL_SECONDS
#             )

`

[swetashre]

Sorry for late reply. Here the problem is that you are trying to use an assume role from the docker container but when boto3 is not able to find that role it is trying to use ec2 instance metadata service and then it is giving you error.

Have you tried by adding the credential file to your docker container root folder and see if that works for you ?
Can you please follow these steps and let me know your result?

1. docker images
2. docker run -it -d <your image id>
3. docker ps # you will get container id for you image
4. docker attach <container id>

after running this command you will get something like this:
 root@a1bf5930c4ce:/# 

then execute this code
root@a1bf5930c4ce:/# python
Python 2.7.17 (default, Apr 15 2020, 17:20:14) 
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>>boto3.set_stream_logger('')
>>> client = boto3.client('sqs')

If you are getting the same credentials error can you add your ~/.aws folder to the root folder and again run these command ?

[eldarnegrinperion]

I will check and get back here.

One note though: I don't understand why for S3 it works ok and for SQS not?
If the AWS session is expired S3 fails as well, then I assume role and all works fine..

[eldarnegrinperion]

still no good.
root@7a7c1bcb49b7:/usr/src/proj# ls /.aws
config credentials
root@7a7c1bcb49b7:/usr/src/proj# python
Python 3.7.4 (default, Oct 17 2019, 05:59:21)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import boto3
boto3.set_stream_logger('')
client = boto3.client('sqs')
2020-07-23 15:17:37,467 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2020-07-23 15:17:37,470 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2020-07-23 15:17:37,472 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2020-07-23 15:17:37,476 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2020-07-23 15:17:37,477 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2020-07-23 15:17:37,478 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2020-07-23 15:17:37,479 botocore.hooks [DEBUG] Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section
2020-07-23 15:17:37,486 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2020-07-23 15:17:37,486 botocore.hooks [DEBUG] Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section
2020-07-23 15:17:37,486 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2020-07-23 15:17:37,486 botocore.hooks [DEBUG] Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section
2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: env
2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: sso
2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2020-07-23 15:17:37,494 botocore.credentials [DEBUG] Looking for credentials via: config-file
2020-07-23 15:17:37,495 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2020-07-23 15:17:37,495 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2020-07-23 15:17:37,495 botocore.credentials [DEBUG] Looking for credentials via: container-role
2020-07-23 15:17:37,495 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2020-07-23 15:17:37,496 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80
2020-07-23 15:17:38,510 urllib3.connectionpool [DEBUG] Starting new HTTP connection (2): 169.254.169.254:80
2020-07-23 15:17:39,524 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 1336, in getresponse
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

[swetashre]

@eldarnegrinperion - Are you running this docker inside a ec2 instance or from your local environment ?

[eldarnegrinperion]

i tried two configurations:

1. local docker on my machine, with credential files (temp session)
2. inside kubernetes cluster in AWS, which has access to anything inside the account through role (and this is tested in other containers in different languages)
   both yield the same result.

[swetashre]

When you are running in local environment did you add your credentials file to your docker container root folder ? Are you also getting the same error even after adding the credentials file to your docker container root folder ?

[eldarnegrinperion]

yes i am bringing it in, /.aws/credentials + /.aws/config, and still get the error
my credentials file holds:
[default]
aws_access_key_id = xxx
aws_secret_access_key = xxx
region = us-east-1
aws_session_token = xxx

my config file holds:
[default]
output = json
region = us-east-1

[eldarnegrinperion]

if the .aws folder wasn't in the correct folder (i.e /.aws | /home/ubuntu/.aws | /home/airflow/.aws depending on image) nothing works - S3 and all other services as well.
the issue here is even if the folder is in the correct place (verified by logging into S3 and performing actions) the default region from config file isn't being taken and hence causing the exception when trying to work with region specific services such as SQS.

[stobrien89]

Hi @eldarnegrinperion,

the issue here is even if the folder is in the correct place (verified by logging into S3 and performing actions) the default region from config file isn't being taken and hence causing the exception when trying to work with region specific services such as SQS.

In this case, were you still using an instance profile for credentials? or were you using the credentials file? Would you be able to share what your config file looks like?

[eldarnegrinperion]

yes i am bringing it in, /.aws/credentials + /.aws/config, and still get the error
my credentials file holds:
[default]
aws_access_key_id = xxx
aws_secret_access_key = xxx
region = us-east-1
aws_session_token = xxx

my config file holds:
[default]
output = json
region = us-east-1

it's here above :)

[stobrien89]

Hi @eldarnegrinperion,

Sorry for missing that. It's quite a long thread 😄. The only thing I see that's out of the ordinary is declaring a region in the credentials file (usually only specified in the config file), but I've tested it and that seems to work fine. As an alternative, have you tried setting the AWS_DEFAULT_REGION environment variable when using your instance profile?

[eldarnegrinperion]

with AWS default region it works fine.
again, the issue here is that it's not consistent.
it SHOULD support the configuration file, as stated in the docs, and it doesn't, which is what caused the confusion that started this thread :)

[tim-finnigan]

Hi @eldarnegrinperion, this issue was recently assigned to me so I wanted to check in. Is this still an issue, and if so do you have any updates as far as what you’ve tried doing?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[eldarnegrinperion]

it's such an old issue, but still exists. eventually it is up to you guys to decide if you want to solve it. thanks for your time :)

[kdaily]

Hi @eldarnegrinperion,

I took your Dockerfile and can reproduce your error. I've determined the error is indeed caused by missing AWS configuration and credentials - they are not in the right place. The location of the AWS config and credentials file will change depending on the Docker container image you're using, so they'll need to be moved to the home directory of the user who is running the command. When using the base container image python:3.7.4, the container is run as root, and the home directory is /root/. This means your config and credentials files must be in /root/.aws/config and /root/.aws/credentials. I verified this by adding in the following to the Dockerfile, assuming your config and credential files are present in the current local directory (CAUTION: I would NOT recommend this method going forward, only to illustrate that this is the correct location):

## Add AWS config and credentials
RUN mkdir /root/.aws/
COPY config /root/.aws/
COPY credentials /root/.aws/

If I do not copy them, or copy them to /.aws/, I receive the same error as you, NoRegionError. I could also provide them by mounting the directory containing the config and credential files when running docker run via the -v flag.

There was some confusion as to why you seemed to get different behavior depending on which service client you were using - S3 or SQS. You can instantiate an S3 client without a region because it assumes you would be using the global region, so you will not receive a NoRegionError. However, you would eventually get a NoCredentialsError if you tried to run an S3 client command if your configuration file is missing (or, you haven't set a region using another method). Other clients, like SQS, require a region when constructing the client; if it cannot find a region using any of the methods available (IMDS, environment variable, parameter, or config file) you'll receive the NoRegionError.

I've ruled out any reason why boto3 would not be working, and this is an issue with your specific environment. We aren't able to assist on the details related to Kubernetes - you would be better off asking in a forum like repost.aws or Stack Overflow.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.