Skip to content

Accessing non-US S3 buckets causes S3ResponseError #443

Closed
russss opened this Issue Dec 21, 2011 · 15 comments
@russss
russss commented Dec 21, 2011

The following code, where eu-west-bucket is a bucket in the eu-west region:

import boto
suri = boto.storage_uri('s3://eu-west-bucket/test')
suri.new_key()

...raises this error:

boto.exception.S3ResponseError: S3ResponseError: 301 Moved Permanently
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message <RequestId>06660583263444FC</RequestId><Bucket>smarkets-db</Bucket><HostId>TCZJTKZ8wo8V1h0xjkOI6grojs/r9IBkhMOcvolXv06QEtxTX89M55aLTPGOo/ht</HostId><Endpoint>eu-west-bucket.s3.amazonaws.com</Endpoint></Error>

This appears to have been introduced between 2.0b4 and 2.0rc1. Before 2.0rc1, s3 requests were sent to bucketname.s3.amazonaws.com, but now they seem to all go to s3.amazonaws.com. The redirect isn't handled by the HTTP code because that 301 response doesn't appear to include a Location header - it requires you to parse the body of the response to get the URL.

@garnaat
the boto project member
garnaat commented Dec 21, 2011

What happens if you just do this:

import boto
s3 = boto.connect_s3()
s3.lookup('test')

@garnaat
the boto project member
garnaat commented Dec 21, 2011

I think the problem can be explained in this code comment:

    # Use OrdinaryCallingFormat instead of boto-default
    # SubdomainCallingFormat because the latter changes the hostname
    # that's checked during cert validation for HTTPS connections,
    # which will fail cert validation (when cert validation is enabled).

Because it is explicitly using OrdinaryCallingFormat, this will cause problems. Let me do a bit more investigation and see if this can be resolved. This was introduced when client cert validation was added.

@hochidsp

Any news on this bug?

@roncohen
roncohen commented Dec 9, 2012

+1

@sebdah
sebdah commented Jan 2, 2013

Ran in to a similar problem. It works well when connecting with boto.connect_s3 but not with boto.s3.connect_to_region.

@victorhooi

Hi,

I can report that I seemed to his this bug as well, using Duplicity to attempt to backup to a S3 bucket located in Sydney.

Cheers,
Victor

@ispivey
ispivey commented Apr 2, 2013

+1

@yang
yang commented May 7, 2013

+1

@lyrixx
lyrixx commented May 15, 2013

+1

@izquierdo

+1

Anything I can help with?

@toastdriven

In trying to address this, I added a test case in SHA: 084dd57. I can't get it to fail the same way though. Running that test (python test.py -s -v -x integration/storage_uri/test_storage_uri.py) passes.

And while it is the us-west-2 region instead of the eu-west-1 region, it still should have exhibited the same behavior (being in a non-classic/us-east-1 region). I also tried it with eu-west-1, which made no difference, as the tests passed there too.

If others could either verify (http://docs.pythonboto.org/en/latest/contributing.html#setting-up-a-development-environment, using the above command) or modify that testcase so it fails, that would be appreciated. Be warned that running it will incur a small cost to your AWS account.

Closing until we have a way to replicate.

@shyamnambiar

I am new to this and getting error

LOG: archive command failed with exit code 2
DETAIL: The failed archive command was: envdir /etc/wal-e.d/env /usr/bin/wal-e wal-push pg_xlog/000000010000000000000001
2013-06-11T11:14:39.023+00 pid=4291 wal_e.worker.s3_worker INFO MSG: begin archiving a file
DETAIL: Uploading "pg_xlog/000000010000000000000001" to "s3://pgsqlbackups/wal_005/000000010000000000000001.lzo".
2013-06-11T11:14:39.116+00 pid=4291 wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
DETAIL: Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/WAL_E-0.5.12-py2.6.egg/wal_e/cmd.py", line 334, in main
backup_cxt.wal_s3_archive(args.WAL_SEGMENT)
File "/usr/lib/python2.6/site-packages/WAL_E-0.5.12-py2.6.egg/wal_e/operator/s3_operator.py", line 402, in wal_s3_archive
wal_path)
File "/usr/lib/python2.6/site-packages/WAL_E-0.5.12-py2.6.egg/wal_e/worker/s3_worker.py", line 308, in do_lzop_s3_put
k = uri_put_file(s3_url, tf)
File "/usr/lib/python2.6/site-packages/WAL_E-0.5.12-py2.6.egg/wal_e/worker/s3_worker.py", line 165, in uri_put_file
k = suri.new_key()
File "/usr/lib/python2.6/site-packages/boto/storage_uri.py", line 156, in new_key
bucket = self.get_bucket(validate, headers)
File "/usr/lib/python2.6/site-packages/boto/storage_uri.py", line 141, in get_bucket
bucket = conn.get_bucket(self.bucket_name, validate, headers)
File "/usr/lib/python2.6/site-packages/boto/s3/connection.py", line 391, in get_bucket
bucket.get_all_keys(headers, maxkeys=0)
File "/usr/lib/python2.6/site-packages/boto/s3/bucket.py", line 360, in get_all_keys
'', headers, **params)
File "/usr/lib/python2.6/site-packages/boto/s3/bucket.py", line 327, in _get_all
response.status, response.reason, body)
S3ResponseError: S3ResponseError: 301 Moved Permanently
<?xml version="1.0" encoding="UTF-8"?>
PermanentRedirectThe bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.F333FDD2E99800A3pgsqlbackupsPVonXWNfvXyKXHae034+C1Xc71xICtH6Q8lLRL/Hm3CxLZzkqMnsH6g7njcvBoQipgsqlbackups.s3.amazonaws.com

please help me

@toastdriven

@shyamnambiar It appears that bucket was created in the standard ("us-east-1") so subsequent requests to put data in that bucket must also try to use that region. I'm not familiar with wal_e, so you may want to contact the developer.

@bshi
bshi commented Jun 14, 2013

@shyamnambiar Please try using the latest release of wal_e on master (0.5.12 is quite old and not even the latest in the 0.5 series - 0.5.14). It should have resolved most edge cases with respect to bucket/region access.

WAL-E project is here:

https://github.com/wal-e/WAL-E/issues?state=open

@tail tail pushed a commit to tail/wal-e that referenced this issue Dec 15, 2013
@bshi bshi Allow use of S3 buckets in regions other than us-east-1.
Work around region issue in S3Backup.new_connection and
boto.storage_uri.  When attempting to connect to a bucket in a
non-standard (non us-east-1) region, boto complains

  S3ResponseError: 301 Moved Permanently
  The bucket you are attempting to access must be addressed using the
  specified endpoint. Please send all future requests to this endpoint.

Naming collision fix contributed by Toby Collier (@tobyw4n)

Add a unit test and integration test.  This adds a testing dependency on
unittest2 which is a Python 2.6 backport of the unittest module
improvements in Python 2.7.

Having AWS credentials in your environment along with the
WALE_S3_INTEGRATION_TESTS (any arbitrary value) will trigger inclusion
of the integration tests.

References:

  boto/boto#443
  https://github.com/heroku/WAL-E/issues/24

Signed-off-by: Bo Shi <bs@alum.mit.edu>
Signed-off-by: Daniel Farina <drfarina@acm.org>
832e5ab
@zbenjamin zbenjamin pushed a commit to zbenjamin/wal-e that referenced this issue Feb 7, 2014
@bshi bshi Allow use of S3 buckets in regions other than us-east-1.
Work around region issue in S3Backup.new_connection and
boto.storage_uri.  When attempting to connect to a bucket in a
non-standard (non us-east-1) region, boto complains

  S3ResponseError: 301 Moved Permanently
  The bucket you are attempting to access must be addressed using the
  specified endpoint. Please send all future requests to this endpoint.

Naming collision fix contributed by Toby Collier (@tobyw4n)

Add a unit test and integration test.  This adds a testing dependency on
unittest2 which is a Python 2.6 backport of the unittest module
improvements in Python 2.7.

Having AWS credentials in your environment along with the
WALE_S3_INTEGRATION_TESTS (any arbitrary value) will trigger inclusion
of the integration tests.

References:

  boto/boto#443
  https://github.com/heroku/WAL-E/issues/24

Signed-off-by: Bo Shi <bs@alum.mit.edu>
Signed-off-by: Daniel Farina <drfarina@acm.org>
5d7b73a
@mattse
mattse commented Mar 5, 2015

I posted a workaround to this problem here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.