aws s3 doesn't follow TemporaryRedirect

[grobie]

error message

aws s3 cp vagrant/foo.box to s3://foo/vagrant/foo.box
upload failed: vagrant/foo.box to s3://foo/vagrant/foo.box
HTTPSConnectionPool(host='foo.s3.amazonaws.com', port=443): Max retries exceeded with url: /vagrant/foo.box?partNumber=6&uploadId=TQEd8x5XRHKjrHo366qad.......-- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

curl https://foo.s3.amazonaws.com
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message><RequestId>63B...</RequestId><Bucket>foo</Bucket><HostId>S3nu....</HostId><Endpoint>foo.s3-ap-southeast-1.amazonaws.com</Endpoint></Error>

[jamesls]

What version of the cli are you using aws --version? I ask because I believe this is fixed in the latest versions.

[grobie]

aws-cli/1.2.7 Python/2.7.3 Linux/2.6.32-5-xen-amd64

[grobie]

Just to be clear, that happened for a new bucket. Once the DNS changes trickled through (some hours) it worked fine. Steps to reproduce:

1. create new bucket (I think it has to be in a non us zone, like ap-southeast-1)
2. try to upload to that bucket immediately

[prognostikos]

I can confirm that I'm seeing the same issue with a newly created bucket in eu-west-1. Just tried with aws-cli/1.2.9 Python/2.7.5 Darwin/13.0.0 - will wait to see if it eventually works.

[prognostikos]

A followup - a few hours later the cp completed with no issues

[shallker]

The newly created bucket will have a redirect, for example:
hit new-bucket.s3.amazonaws.com will redirect to new-bucket.s3-us-west-2.amazonaws.com

But old bucket won't have this redirect, for example:
hit old-bucket.s3.amazonaws.com will stay at old-bucket.s3.amazonaws.com

What's s3 team's plan?

[jamesls]

I'm going to close this, as it seems like this is fixed now. I confirmed that I'm able to create buckets outside of US classic and immediately upload data to the bucket in the latest version of the CLI. From the debug logs it shows that we're following 307 redirects:

$ aws s3 mb s3://test-eu-west-1-jamesls/ --region eu-west-1
make_bucket: s3://test-eu-west-1-jamesls/
$ aws s3 cp NOTICE s3://test-eu-west-1-jamesls/ --region eu-west-1 --debug
...
2014-02-20 13:51:12,543 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): test-eu-west-1-jamesls.s3.amazonaws.com
2014-02-20 13:51:12,602 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "PUT /NOTICE HTTP/1.1" 307 None
2014-02-20 13:51:12,606 - botocore.awsrequest - DEBUG - Redirect received, rewinding stream: <open file u'NOTICE', mode 'rb' at 0x108c8a0c0>
2014-02-20 13:51:12,607 - botocore.awsrequest - DEBUG - Rewinding stream: <open file u'NOTICE', mode 'rb' at 0x108c8a0c0>
2014-02-20 13:51:12,607 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): test-eu-west-1-jamesls.s3-external-3.amazonaws.com
2014-02-20 13:51:13,299 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "PUT /NOTICE HTTP/1.1" 200 0
...

If anyone is still seeing this issue feel free to post a set of repro steps and I'd be happy to investigate further.

[jamesls]

Reopening, I can trigger this error when we reach the threshold for switching to multipart uploads, so if I try the same test with a 10mb file, I can repro the issue. Investigating...

[rptb1]

I am having this issue right now with a newly-created bucket in eu-west-1. It does not occur with an identical bucket in us-east-1. I checked lib/py*/site-packages/botocore/awsrequest.py includes the change above.

I am unable to reproduce this problem reliably. It seems to occur for some buckets and not others.

$ pip freeze
awscli==1.3.6
bcdoc==0.12.2
botocore==0.40.0
colorama==0.2.5
docutils==0.11
jmespath==0.3.1
ply==3.4
pyasn1==0.1.7
python-dateutil==2.2
rsa==3.1.2
six==1.6.1
wsgiref==0.1.2
$ curl https://[MY BUCKET].s3.amazonaws.com      
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message><RequestId>[REDACTED]</RequestId><Bucket>[MY BUCKET]</Bucket><HostId>[REDACTED]</HostId><Endpoint>[MY BUCKET].s3-external-3.amazonaws.com</Endpoint></Error>

The HTTPSConnectionPool Max retries exceeded error occurs with a command like

aws s3 sync --acl private --storage-class REDUCED_REDUNDANCY /path/to/files s3://[MY BUCKET]/to/files

Edit: Two hours later, the problem has gone away, with the "curl" command above now producing an "AccessDenied" response. Presumably this problem has to do with bucket setup at the server end, and so it hard to reproduce reliably.

[smremde]

This is not fixed in aws-cli/1.3.6 Python/2.7.4 Linux/3.8.13-xxxx-grs-ipv6-64

[pesterhazy]

I'm seeing the same problem here, also with eu-west-1. The TemporaryRedirect seems to be the problem.

On my box this also triggers a bug in awscli: you get negative numbers of files remaining as the number of the chunk increases.

[pesterhazy]

As expected, after around 2 hours the problem has disappeared, I can now upload large files without problems. It's still a bug that a bucket in eu-west-1 cannot be uploaded to in the first couple of hours after its creation (in some circumstances).

[kevin-canadian]

I get this problem as well. Steps to reproduce are simple.

1. Create a NEW bucket in eu-west-1 (using aws s3 mb).
2. Upload a large file (e.g. 10-15MB) (using aws s3 sync).

[kevin-canadian]

Is there a way to disable multi-part uploading? A flag, an environment variable, etc.?

[jamesls]

This issue is the same as: #634, with an overview of the situation here: #634 (comment)

There's a fix for this (#634 (comment)), we just need to do more testing to ensure there are no regressions, and then we'll release the fix.

There's no way to currently turn off multipart upload, but I believe the problem still happens with large files even without multipart upload.