fix connect timeout while getting IAM creds in Docker #941
Conversation
While trying to use S3FS in a Docker container on my EC2, I found that `AioIMDSFetcher._fetch_metadata_token()` was throwing a `ConnectTimeoutError` trying to connect to the metadata service with `PUT http://169.254.169.254/latest/api/token/`. This came from an `aiohttp.ServerTimeoutError` while trying to submit the PUT. When I compared the logs to plain botocore, I found that it instead encountered a `ReadTimeoutError`, which it ignored and moved on to trying `GET http://169.254.169.254/latest/meta-data/iam/security-credentials/`, which succeeded. This error was redirected from a `urllib3.ReadTimeoutError` instead of a `ConnectTimeoutError`. I believe this particular difference comes from aiohttp and urllib3 handling the server connection issue differently. By handling the `ConnectTimeoutError`, aiobotocore is able to successfully get the credentials from the metadata service while in Docker. Note that this issue does not occur outside of Docker, even with an identical virtual env.
`aiohttp.ServerTimeoutError` should be mapped to `ReadTimeoutError`, not `ConnectTimeoutError` (aio-libs#941)
|
This pull request introduces 1 alert when merging 215089f into a5920f6 - view on LGTM.com new alerts:
|
`aiohttp.ServerTimeoutError` should be mapped to `ReadTimeoutError`, not `ConnectTimeoutError` (aio-libs#941)
aiobotocore/httpsession.py
Outdated
| except ServerTimeoutError as e: | ||
| raise ConnectTimeoutError(endpoint_url=request.url, error=e) |
There was a problem hiding this comment.
we still do need to support ConnectTimeoutError: https://github.com/boto/botocore/blob/develop/botocore/httpsession.py#L478. so probably need something a little more refined
There was a problem hiding this comment.
Digging through the source code, it looks like we can differentiate by looking at the message:
- Connection timeout - https://github.com/aio-libs/aiohttp/blob/8e92c4c7c305f914c17f41539da20822ea7baddf/aiohttp/client.py#L505
- Read timeout - https://github.com/aio-libs/aiohttp/blob/8e92c4c7c305f914c17f41539da20822ea7baddf/aiohttp/client_proto.py#L197
How about:
if str(e).lower().startswith('connect'):
raise ConnectTimeoutError(...)
else:
raise ReadTimeoutError(...)
|
@thehesiod Are there any other changes you would like me to make? |
|
my apologies for delay, just been crazy here |
Codecov Report
@@ Coverage Diff @@
## master #941 +/- ##
==========================================
- Coverage 86.77% 86.74% -0.04%
==========================================
Files 55 55
Lines 5295 5297 +2
==========================================
Hits 4595 4595
- Misses 700 702 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
Description of Change
While trying to use S3FS in a Docker container on my EC2, I found that
AioIMDSFetcher._fetch_metadata_token()was throwing aConnectTimeoutErrortrying to connect to the metadata service withPUT http://169.254.169.254/latest/api/token/. This came from anaiohttp.ServerTimeoutErrorwhile trying to submit the PUT.When I compared the logs to plain botocore, I found that it instead encountered a
ReadTimeoutError, which it ignored and moved on to tryingGET http://169.254.169.254/latest/meta-data/iam/security-credentials/, which succeeded. This error was redirected from aurllib3.ReadTimeoutErrorinstead of aConnectTimeoutError.I believe this particular difference comes from aiohttp and urllib3 handling the server connection issue differently. By handling the
ConnectTimeoutError, aiobotocore is able to successfully get the credentials from the metadata service while in Docker.Note that this issue does not occur outside of Docker, even with an identical virtual env.
Assumptions
ConnectTimeoutErrorfrom the metadata service shouldn't bubble up and end attempts to resolve credentialsChecklist for All Submissions
Checklist when updating botocore and/or aiohttp versions