cannot open a connection to S3 in ap-northeast-2

[eniskonuk]

Trying to open a S3 connection to ap-northeast-2 times out with the Seahorse::Client::NetworkingError/execution expired exception. I have a snippet of code that opens a connection to EC2 successfully and then tries the same w/ S3 and times out. The buckets on S3 exist and I can see and manipulate them in the AWS console. Tried this with 2.2.13, 14, 15, 17 of the SDK and all fail the same way. Running Ruby 2.1.7.

Here is a wire trace.

opening connection to ec2.ap-northeast-2.amazonaws.com:443...
opened
starting SSL for ec2.ap-northeast-2.amazonaws.com:443...
SSL established
<- "POST / HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded; charset=utf-8\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.2.13 ruby/2.1.7 x86_64-darwin15.0\r\nX-Amz-Date: 20160218T023056Z\r\nHost: ec2.ap-northeast-2.amazonaws.com\r\nX-Amz-Content-Sha256: 672499f98b79d4003cccbe13a6cbf7075942acc15b97f573064bc60e117baddc\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAJKNKRDVBVPNFNJBA/20160218/ap-northeast-2/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=3850ae3941b0626d805a4c33043a70e034b1f5d547c997343b8403acdfcf479c\r\nContent-Length: 41\r\nAccept: /\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Content-Type: text/xml;charset=UTF-8\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Vary: Accept-Encoding\r\n"
-> "Date: Thu, 18 Feb 2016 02:30:56 GMT\r\n"
-> "Server: AmazonEC2\r\n"
-> "\r\n"
-> "703\r\n"
reading 1795 bytes...
-> "\n<DescribeRegionsResponse xmlns="http://ec2.amazonaws.com/doc/2015-10-01/\">\n 14f3449e-2574-4236-b35c-4799954eb13b\n \n \n eu-west-1\n ec2.eu-west-1.amazonaws.com\n \n \n ap-southeast-1\n ec2.ap-southeast-1.amazonaws.com\n \n \n ap-southeast-2\n ec2.ap-southeast-2.amazonaws.com\n \n \n eu-central-1\n ec2.eu-central-1.amazonaws.com\n \n \n ap-northeast-2\n ec2.ap-northeast-2.amazonaws.com\n \n \n ap-northeast-1\n ec2.ap-northeast-1.amazonaws.com\n \n \n us-east-1\n ec2.us-east-1.amazonaws.com\n \n \n sa-east-1\n ec2.sa-east-1.amazonaws.com\n \n \n us-west-1\n ec2.us-west-1.amazonaws.com\n \n \n us-west-2\n ec2.us-west-2.amazonaws.com\n \n \n"
read 1795 bytes
reading 2 bytes...
-> "\r\n"
read 2 bytes
-> "0\r\n"
-> "\r\n"
Conn keep-alive
ap-northeast-2
opened connection to S3
opening connection to ap-northeast-2-qwiklab-asset.s3.ap-northeast-2.amazonaws.com:443...
opening connection to ap-northeast-2-qwiklab-asset.s3.ap-northeast-2.amazonaws.com:443...
opening connection to ap-northeast-2-qwiklab-asset.s3.ap-northeast-2.amazonaws.com:443...
opening connection to ap-northeast-2-qwiklab-asset.s3.ap-northeast-2.amazonaws.com:443...
Error: Updating bucket ap-northeast-2-qwiklab-asset => Seahorse::Client::NetworkingError/execution expired
opening connection to ap-northeast-2-qwiklab-release.s3.ap-northeast-2.amazonaws.com:443...
opening connection to ap-northeast-2-qwiklab-release.s3.ap-northeast-2.amazonaws.com:443...
opening connection to ap-northeast-2-qwiklab-release.s3.ap-northeast-2.amazonaws.com:443...
opening connection to ap-northeast-2-qwiklab-release.s3.ap-northeast-2.amazonaws.com:443...
Error: Updating bucket ap-northeast-2-qwiklab-release => Seahorse::Client::NetworkingError/execution expired

[trevorrowe]

I just ran the following code without issue:

s3 = Aws::S3::Client.new(http_wire_trace:true, region:'ap-northeast-2')
s3.list_objects(bucket:'aws-sdk-ap-northeast-2')

Which generated the following HTTP wire trace:

opening connection to aws-sdk-ap-northeast-2.s3.ap-northeast-2.amazonaws.com:443...
opened
starting SSL for aws-sdk-ap-northeast-2.s3.ap-northeast-2.amazonaws.com:443...
SSL established
<- "GET /?encoding-type=url HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.2.17 ruby/2.2.2 x86_64-darwin13\r\nX-Amz-Date: 20160218T192931Z\r\nHost: aws-sdk-ap-northeast-2.s3.ap-northeast-2.amazonaws.com\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAJUNH63P3WCTAYHFA/20160218/ap-northeast-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=6c0f73f010cb260de922d5194305840f7d1d7759b332c4113dc43d4fef9596f6\r\nContent-Length: 0\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "x-amz-id-2: 73iBOoaEYCcsMQGt5fYC/Lde2l4YL9GB9lHE/TiFY6ARlONIF43puCH1bExXxfDLfjeRiVfmke0=\r\n"
-> "x-amz-request-id: B780DC7D2C8A8819\r\n"
-> "Date: Thu, 18 Feb 2016 19:29:33 GMT\r\n"
-> "x-amz-bucket-region: ap-northeast-2\r\n"
-> "Content-Type: application/xml\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Server: AmazonS3\r\n"
-> "\r\n"
-> "118\r\n"
reading 280 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListBucketResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Name>aws-sdk-ap-northeast-2</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>"
read 280 bytes
reading 2 bytes...
-> "\r\n"
read 2 bytes
-> "0\r\n"
-> "\r\n"
Conn keep-alive

I'm not sure why Net::HTTP is unable to open a connection to the endpoint. Can you ping s3.ap-northeast-2.amazonaws.com from the same host?

[eniskonuk]

Very strange. The ping doesn’t work for me. I get an ICMP timeout. If I try it from an EC2 instance it works fine. So, something is wrong w/ the routing somehow. The trace route shows that the request makes it to Korea but then dies.

traceroute to s3.ap-northeast-2.amazonaws.com (52.92.8.9), 64 hops max, 52 byte packets
1 router.asus.com (192.168.1.1) 2.384 ms 2.097 ms 1.977 ms
2 10.0.0.1 (10.0.0.1) 2.809 ms 2.940 ms 2.746 ms
3 96.120.69.61 (96.120.69.61) 11.261 ms 19.999 ms 11.987 ms
4 ge-6-1-ur01.woburn.ma.boston.comcast.net (68.87.158.197) 12.586 ms 11.783 ms 20.213 ms
5 be-117-ar01.woburn.ma.boston.comcast.net (68.85.106.161) 17.395 ms 16.265 ms 17.972 ms
6 he-0-11-0-0-ar01.needham.ma.boston.comcast.net (162.151.112.17) 15.197 ms 16.947 ms 17.525 ms
7 be-7015-cr02.newyork.ny.ibone.comcast.net (68.86.90.217) 21.266 ms 20.903 ms 23.006 ms
8 be-10103-cr02.ashburn.va.ibone.comcast.net (68.86.85.161) 27.581 ms 28.110 ms 29.739 ms
9 be-10114-cr02.56marietta.ga.ibone.comcast.net (68.86.85.10) 51.819 ms 51.451 ms 54.761 ms
10 be-11424-cr02.dallas.tx.ibone.comcast.net (68.86.85.22) 68.878 ms 70.202 ms 74.977 ms
11 be-11524-cr02.losangeles.ca.ibone.comcast.net (68.86.87.173) 99.990 ms 99.257 ms 95.023 ms
12 he-0-0-0-0-pe01.losangeles.ca.ibone.comcast.net (68.86.85.14) 94.996 ms 109.233 ms 99.985 ms
13 50.248.117.82 (50.248.117.82) 92.609 ms 94.291 ms 92.778 ms
14 121.78.120.225 (121.78.120.225) 94.021 ms 96.099 ms 96.147 ms
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *

I thought that it was an aws-sdk issue as everything seemed to work fine through the console. It looks like it’s something else.

Thx
Enis Konuk
enis@qwiklab.com
978-760-0732

On Feb 18, 2016, at 2:31 PM, Trevor Rowe notifications@github.com wrote:

I just ran the following code without issue:

s3 = Aws::S3::Client.new(http_wire_trace:true, region:'ap-northeast-2')
s3.list_objects(bucket:'aws-sdk-ap-northeast-2')
Which generated the following HTTP wire trace:

opening connection to aws-sdk-ap-northeast-2.s3.ap-northeast-2.amazonaws.com:443...
opened
starting SSL for aws-sdk-ap-northeast-2.s3.ap-northeast-2.amazonaws.com:443...
SSL established
<- "GET /?encoding-type=url HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.2.17 ruby/2.2.2 x86_64-darwin13\r\nX-Amz-Date: 20160218T192931Z\r\nHost: aws-sdk-ap-northeast-2.s3.ap-northeast-2.amazonaws.com\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAJUNH63P3WCTAYHFA/20160218/ap-northeast-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=6c0f73f010cb260de922d5194305840f7d1d7759b332c4113dc43d4fef9596f6\r\nContent-Length: 0\r\nAccept: /\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "x-amz-id-2: 73iBOoaEYCcsMQGt5fYC/Lde2l4YL9GB9lHE/TiFY6ARlONIF43puCH1bExXxfDLfjeRiVfmke0=\r\n"
-> "x-amz-request-id: B780DC7D2C8A8819\r\n"
-> "Date: Thu, 18 Feb 2016 19:29:33 GMT\r\n"
-> "x-amz-bucket-region: ap-northeast-2\r\n"
-> "Content-Type: application/xml\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Server: AmazonS3\r\n"
-> "\r\n"
-> "118\r\n"
reading 280 bytes...
-> "\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/\">aws-sdk-ap-northeast-21000urlfalse"
read 280 bytes
reading 2 bytes...
-> "\r\n"
read 2 bytes
-> "0\r\n"
-> "\r\n"
Conn keep-alive
I'm not sure why Net::HTTP is unable to open a connection to the endpoint. Can you ping s3.ap-northeast-2.amazonaws.com from the same host?

—
Reply to this email directly or view it on GitHub #1096 (comment).

[trevorrowe]

No problem. Please feel free to re-open if there is anything else.