aws s3 is not compatible with VPC - S3 endpoints

[tatobi]

After we configured AWS VPC -> S3 endpoint in our VPC and started to test on a VPC machine without public IP / EIP (http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html ) the basic awscli s3 usage stopped to working. However, if we define the --endpoint-url , it works again but evidently we wouldn't define the --endpoint-url every time when we use aws cli s3 on any machine. Moreover it may break the data route from VPC to S3 without public internet paths on a machine which has public IP/EIP.

Log:

aws s3 ls s3://s3-policytest/ --debug
2015-05-14 09:24:36,776 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.7.26 Python/2.7.6 Linux/3.13.0-48-generic, botocore version: 0.107.0
2015-05-14 09:24:36,776 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', 's3://s3-policytest/', '--debug']
2015-05-14 09:24:36,776 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f3c6f7ea9b0>
2015-05-14 09:24:36,776 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider at 0x7f3c6fa20b18>
2015-05-14 09:24:36,777 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f3c6fa26938>
2015-05-14 09:24:36,777 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x7f3c6fba1f50>
2015-05-14 09:24:36,778 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler <function add_waiters at 0x7f3c6fa26938>
2015-05-14 09:24:36,778 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <function uri_param at 0x7f3c6fba1f50>
2015-05-14 09:24:36,779 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <function uri_param at 0x7f3c6fba1f50>
2015-05-14 09:24:36,779 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthand object at 0x7f3c6f7efc10>
2015-05-14 09:24:36,779 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-05-14 09:24:36,779 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <function uri_param at 0x7f3c6fba1f50>
2015-05-14 09:24:36,779 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <function uri_param at 0x7f3c6fba1f50>
2015-05-14 09:24:36,779 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthand object at 0x7f3c6f7efc10>
2015-05-14 09:24:36,780 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-05-14 09:24:36,780 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <function uri_param at 0x7f3c6fba1f50>
2015-05-14 09:24:36,780 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2015-05-14 09:24:36,780 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2015-05-14 09:24:36,780 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2015-05-14 09:24:36,781 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2015-05-14 09:24:36,807 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2015-05-14 09:24:36,814 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjects: calling handler <function add_expect_header at 0x7f3c6ff68230>
2015-05-14 09:24:36,814 - MainThread - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x7f3c6f53b8d0> (verify_ssl=True) with params: {'query_string': {u'prefix': u'', u'delimiter': '/'}, 'headers': {}, 'url_path': u'/s3-policytest', 'body': '', 'method': u'GET'}
2015-05-14 09:24:36,814 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjects: calling handler <bound method s3._sign_request of <botocore.client.s3 object at 0x7f3c6f53b0d0>>
2015-05-14 09:24:36,814 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjects: calling handler <function fix_s3_host at 0x7f3c6ff67e60>
2015-05-14 09:24:36,814 - MainThread - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-eu-west-1.amazonaws.com/s3-policytest?prefix=&delimiter=%2F
2015-05-14 09:24:36,815 - MainThread - botocore.handlers - DEBUG - URI updated to: https://s3-policytest.s3.amazonaws.com?prefix=&delimiter=%2F
2015-05-14 09:24:36,815 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2015-05-14 09:24:36,815 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2015-05-14 09:24:36,815 - MainThread - botocore.auth - DEBUG - StringToSign:
GET

Thu, 14 May 2015 09:24:36 GMT
/s3-policytest/
2015-05-14 09:24:36,820 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [GET]>
2015-05-14 09:24:36,820 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): s3-policytest.s3.amazonaws.com

..... and it stuck here, because the s3-policytest.s3.amazonaws.com is evidently not accessible via the VPC endpoint because of the VPC endpoint for Ireland region is: s3-eu-west-1.amazonaws.com

.....
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 265, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 313, in call
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 222, in call
return self._check_caught_exception(attempt_number, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 355, in _check_caught_exception
raise caught_exception
ConnectTimeout: HTTPSConnectionPool(host='s3-policytest.s3.amazonaws.com', port=443): Max retries exceeded with url: /?prefix=&delimiter=%2F (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f3c6f508c50>, 'Connection to s3-policytest.s3.amazonaws.com timed out. (connect timeout=60)'))

---

After the --endpoint-url defined, it works well.
root@_:~# aws s3 ls s3://s3-policytest/ --endpoint-url http://s3-eu-west-1.amazonaws.com
PRE *__/
PRE *_*/
PRE man/
PRE */

---

root@**********:~# ping s3-policytest.s3.amazonaws.com
PING s3-1-w.amazonaws.com (54.231.12.153) 56(84) bytes of data.
--- s3-1-w.amazonaws.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1008ms

root@**********:~# ping s3-eu-west-1.amazonaws.com
PING s3-eu-west-1.amazonaws.com (54.231.130.196) 56(84) bytes of data.
64 bytes from 54.231.130.196: icmp_seq=1 ttl=60 time=0.919 ms
64 bytes from 54.231.130.196: icmp_seq=2 ttl=60 time=1.03 ms
2 packets transmitted, 2 received, 0% packet loss, time 1001ms

---

In my opinion, the aws cli s3 should not try to connect the s3-policytest.s3.amazonaws.com if it is already resolved that the DNS for a given region well, because it breaks the usage in EC2 machines connected to S3 via VPC endpoints only.
( 2015-05-14 09:24:36,815 - MainThread - botocore.handlers - DEBUG - URI updated to: https://s3-policytest.s3.amazonaws.com?prefix=&delimiter=%2F )

It may break the data route also, because if anybody configure the VPC endpoint on a machine with EIP/public IP, to avoid route through the public Internet the sensitive S3 data, the aws cli still brake it.

[kyleknap]

@tatobi

So with the CLI, we try to help the customer out with s3 by sending all requests to the global endpoint s3.amazonaws.com if the bucket name is dns compatible. This allows users to avoid having to know and specify the region that their bucket is in because s3 will route the request to correct region. So that is why you are running into this issue.

Fortunately I believe there is a workaround this. If you force the signature version of your request to be v4, we will always send your request to the endpoint of the region you specified due to the requirements of v4 signing. You can set the signature version by running:

$ aws configure set s3.signature_version s3v4

Let me know how that works.

[tatobi]

Thank you for your help. I found the followings:
1.) the above problem fixed somehow automatically by today, now it works without any extra configuration you proposed above. I guess it might be an internal AWS S3-DNS network route problem with newly created buckets and VPC - S3 endpoint policy.
2.) I created a new bucket namely "s3-policytest-2" now to ascertain, with the very same policy like applied before to the VPCE, and it does not work as I expected. Then I applied your suggestion:

$ aws configure set s3.signature_version s3v4

and it solves the problem immediately.
In summary, I guess it is not explicitly an awscli bug, but the awscli doesn't work well until the AWS applies the correct network route path to the newly created S3 bucket's DNS IP address.

[kyleknap]

@tatobi
Thanks for looking into the issue. How long would you say it takes for scenario one to happen after creating the bucket?

[tatobi]

~1-2hours, I dont know exactly but at the afternoon, the second config turned to be good without extra S3 settings, as well.

[jamesls]

Closing out issue, glad you were able to get this working.

[Downchuck]

For others coming through on this thread, here's a us-west-2 example:

./venv/bin/aws s3 ls s3://bucketname/ --endpoint-url https://s3-us-west-2.amazonaws.com --region=us-west-2