backup-push works but not backup-list or backup-fetch in europe bucket

[InformatiQ]

boto==2.9.6
wal-e==0.6.2
gevent==0.13.6

backup-push works fine
backup-list or backup-fetch returns error

wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
DETAIL: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/wal_e/cmd.py", line 315, in main
pool_size=args.pool_size)
File "/usr/local/lib/python2.7/dist-packages/wal_e/operator/s3_operator.py", line 248, in database_s3_fetch
backups = list(bl.find_all(backup_name))
File "/usr/local/lib/python2.7/dist-packages/wal_e/worker/s3_worker.py", line 454, in find_all
for backup in iter(self):
File "/usr/local/lib/python2.7/dist-packages/wal_e/worker/s3_worker.py", line 478, in iter
bucket = self.s3_conn.get_bucket(self.layout.bucket_name())
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 431, in get_bucket
bucket.get_all_keys(headers, maxkeys=0)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 375, in get_all_keys
'', headers, **params)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 342, in _get_all
response.status, response.reason, body)
S3ResponseError: S3ResponseError: 301 Moved Permanently

PermanentRedirectThe bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.C173EF42B319E30Fbackups-eu+IFsO55g1dUxAZ89lLtLFoV6KWDQGOzVaO4Yt1rr9LbM5lAzzUKBYkWsYvAe87GAbackups-eu.s3.amazonaws.com

    STRUCTURED: time=2013-07-01T11:14:12.825963-00 pid=12210

any help is much appreciated

[InformatiQ]

wal-fetch works

[fdr]

Try the thing in the v0.6 maintenance branch and let me know how it is. I'm very close to tagging this as 0.6.3, modulo other problems. If it works, your kudos are best directed to @bshi.

[InformatiQ]

i tried the branch v0.6 from git but no luck still the same issue
This basically means that wal-e is not useable for european buckets

[fdr]

Yeah; as has been the case for some time...other people have worked around this by teaching wal-e to follow the redirect, or consider the boto behavior a bug.

If you submit a patch to fix this, I'll review it.

[InformatiQ]

would help me very much if you could point me to where this happens, my python skills are not very good so i get lost easily in code

[fdr]

The problem is the naive way the connection is constructed, using the OrdinaryCallingFormat: https://github.com/wal-e/wal-e/blob/master/wal_e/operator/s3_operator.py#L50-L65

Sad to say, the grotesque situation with S3 bucket names and their corresponding endpoints have led to code like this:

https://github.com/wal-e/wal-e/blob/master/wal_e/worker/s3_worker.py#L51-L140

With a tiny bit of refactoring it should be possible to teach the connection-convention and location-resolution code to be held in greater commonality, as s3_endpoint_for_uri only relies on a URI insomuch as the netloc is a bucket name; so the bucket-name part of the code can be broken out methinks.

This is basically whack-a-mole because some stuff has been taught to use those operators already, gaining alternate-location support, but not every operator has...

[InformatiQ]

I am still trying to figure out the need for OrdinaryCallingFormat as when removing that from the connection all works for me
but again i have only tested fetch push and list for eu and us buckets

[fdr]

The need is for people using S3 classic with upper case bucket names (or,
really, any bucket name not following the preferred naming guidelines),
e.g., me. A historical accident, but I'm not going to drop support for
those in a point release for sure.

They can also hypothetically be useful to someone with a dns-compatible
bucket name that contains dots, and who also wishes to use SSL, since the
dots in the name have been known to introduce certificate validation
problems in the past.

[bshi]

Mole whacker here. @InformatiQ - I looked through the v0.6 branch and it does not look like it has merged the set of region fixes that are in master (@fdr can you confirm?). I suspect that if you install from master things should work as we have some anecdotal evidence that things work in the australian region (#3 (comment)).

I have not had time to do a full end-to-end test, but a dry-run using the appropriate connection constructor arguments against a test bucket in EU worked for me:

In [2]: from boto.s3.connection import SubdomainCallingFormat as SCF, OrdinaryCallingFormat as OCF

In [40]: from boto.s3.connection import S3Connection

In [41]: tc = S3Connection(calling_format=SCF())

In [42]: tc.get_bucket('ireland-abc123').get_location()
Out[42]: u'eu-west-1'

In [43]: import boto

In [44]: suri = boto.storage_uri('s3://ireland-abc123/testbar')

In [45]: suri.connection_args = {'calling_format': OCF(), 'host': 's3-eu-west-1.amazonaws.com'}

In [47]: for x in suri.list_bucket(): print x

<Key: ireland-abc123,foo>
<Key: ireland-abc123,gorilla-away.gif>

In [48]: suri.set_contents_from_string('blah')
Out[48]: 4

In [49]: suri.list_bucket()
Out[49]: <boto.s3.bucketlistresultset.BucketListResultSet instance at 0x10a9f0e60>

In [50]: for i in suri.list_bucket(): print i

<Key: ireland-abc123,foo>
<Key: ireland-abc123,gorilla-away.gif>
<Key: ireland-abc123,testbar>

[bshi]

Erm, scratch that - looks like v0.6 is indeed up to date. Still digging around. @InformatiQ - would it be possible for you to try running with boto debug logging on to provide some more context?

[fdr]

I've been looking into this a couple of days ago. I was scratching around trying to reproduce bucket failures in https://github.com/fdr/wal-e/commits/bucket-naming before doing something about normalizing the endpoint/calling convention guesswork.

[bshi]

OrdinaryCallingFormat should handle any bucket name (us-east-1 included) with SSL verification but the catch is that your S3Connection must be set to the right region (hence the gymnastics we go through currently).

I still need to prove this to myself with exhaustive system test for all regions. The original error suggests to me that somehow the region (aka "host" named argument in S3Connection.init parlance) was mismatched.

Finding the region given a bucket name is tricky and my original approach was probably not optimal. We may be able to simplify slightly. If the bucket name has an uppercase or consecutive periods, it can only be a US east bucket (all other regions follow DNS friendly rules) and no further checking is necessary. If not, the name is SubdomainCallingFormat-friendly and you can discover the region through a connection to s3-us-east so long as SSL verification is turned off for this discovery process (otherwise the host name might not match the ssl wildcard and an exception would be triggered).

[fdr]

I've provoked both the invalid certificate error and 301 redirect in the test suite now committed to that branch. It relies on tox, which for now I think I'm going to make the preferred testing shell.

[fdr]

Notably also: boto 2.0 fails horribly with cert validation on, even though the commit log makes it look as though it is supported. boto 2.2+ (which is required by wal-e 0.7 for multi-delete functionality) doesn't have this problem (it dies with the expected InvalidCertificateException)

[fdr]

Finding the region given a bucket name is tricky and my original approach was probably not optimal. We may be able to simplify slightly. If the bucket name has an uppercase or consecutive periods, it can only be a US east bucket (all other regions follow DNS friendly rules) and no further checking is necessary. If not, the name is SubdomainCallingFormat-friendly and you can discover the region through a connection to s3-us-east so long as SSL verification is turned off for this discovery process (otherwise the host name might not match the ssl wildcard and an exception would be triggered).

This is exactly right given my understanding. DNS lets one avoid the problem of knowing the region of the bucket, hence the reticence of S3 to put locations in ARNs (which I still view as pretty silly to date in that those are not optionally interpreted). So, there's no need to do get_location in that case.

[fdr]

I'm also whacking around a region guesser in that branch. See fdr@8672965, which is totally non-functional and not hooked up, but I'm adding rules to detect bad bucket names.

[fdr]

See #24

[fdr]

Well, given that this is merged, until I hear complaint, closing.