Can't use bucket names with dots

[lewisdiamond]

Using boto for a s3 bucket named with dots, e.g. my.bucket.s3.amazonaws.com fails:

ssl.CertificateError: hostname 'my.bucket.s3.amazonaws.com' doesn't match either of '*.s3.amazonaws.com', 's3.amazonaws.com'

[krallin]

Having the same issue here, running on Python 2.7.9. Python 2.7.9 introduced strict certificate checking. This might be why the error is happening.

@lewisdiamond — are you also running Python 2.7.9?

[krallin]

Note: setting the following configuration solves the issue for me:

[s3]
calling_format = boto.s3.connection.OrdinaryCallingFormat

[lewisdiamond]

@krallin yes, 2.7.9

[kmarekspartz]

Same here and @krallin's config change fixes it for me.

[starrify]

Same issue experienced here. Solved by @krallin's fix.

[jbmartin]

Thanks @krallin, your fix works for me on Python 2.7.9.

[alex]

This looks like an AWS bug to me, as far as I can tell from the various RFCs, *.domain.com in a SAN should not match domain1.domain2.domain.com.

[oberstet]

This workaround does not work for me.

Test program:

from boto.s3.connection import S3Connection
conn = S3Connection()
print conn.get_bucket("web-autobahn-ws")
print conn.get_bucket("autobahn.ws")

Without the workaround in .boto:

$ python test.py
<Bucket: web-autobahn-ws>
Traceback (most recent call last):
  File "test.py", line 4, in <module>
    print conn.get_bucket("autobahn.ws")
  File "c:\Python27\lib\site-packages\boto\s3\connection.py", line 502, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "c:\Python27\lib\site-packages\boto\s3\connection.py", line 521, in head_bucket
    response = self.make_request('HEAD', bucket_name, headers=headers)
  File "c:\Python27\lib\site-packages\boto\s3\connection.py", line 664, in make_request
    retry_handler=retry_handler
  File "c:\Python27\lib\site-packages\boto\connection.py", line 1068, in make_request
    retry_handler=retry_handler)
  File "c:\Python27\lib\site-packages\boto\connection.py", line 942, in _mexe
    request.body, request.headers)
  File "c:\Python27\lib\httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "c:\Python27\lib\httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "c:\Python27\lib\httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "c:\Python27\lib\httplib.py", line 850, in _send_output
    self.send(msg)
  File "c:\Python27\lib\httplib.py", line 812, in send
    self.connect()
  File "c:\Python27\lib\httplib.py", line 1216, in connect
    server_hostname=server_hostname)
  File "c:\Python27\lib\ssl.py", line 350, in wrap_socket
    _context=self)
  File "c:\Python27\lib\ssl.py", line 566, in __init__
    self.do_handshake()
  File "c:\Python27\lib\ssl.py", line 796, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "c:\Python27\lib\ssl.py", line 269, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname 'autobahn.ws.s3.amazonaws.com' doesn't match either of '*.s3.amazonaws.com', 's3.amazonaws.com'

With the workaround:

$ python test.py
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    print conn.get_bucket("web-autobahn-ws")
  File "c:\Python27\lib\site-packages\boto\s3\connection.py", line 502, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "c:\Python27\lib\site-packages\boto\s3\connection.py", line 549, in head_bucket
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 301 Moved Permanently

[oberstet]

FWIW, here is how to monkey patch away hostname verification:

import ssl
if hasattr(ssl, '_create_unverified_context'):
   ssl._create_default_https_context = ssl._create_unverified_context

Other than that, it seems, migrating to Cloudfront (which doesn't require source S3 buckets to be dotted), might be an option.

[oberstet]

Here is a more specific monkey patch:

import ssl

_old_match_hostname = ssl.match_hostname

def _new_match_hostname(cert, hostname):
   if hostname.endswith('.s3.amazonaws.com'):
      pos = hostname.find('.s3.amazonaws.com')
      hostname = hostname[:pos].replace('.', '') + hostname[pos:]
   return _old_match_hostname(cert, hostname)

ssl.match_hostname = _new_match_hostname

[krallin]

@alex,

I imagine S3 can't (or just doesn't) generate certificates on the fly for buckets that don't match the generic certificate.

However, it might make sense for boto to default to the ordinary calling format (vs. the subdomain calling format)? At least for "dotted" buckets where the subdomain calling format will not work?

The default might have to do with #443, though.

@oberstet,

Is your bucket located outside of us-east-1? Your issue looks a lot like #443.

[oberstet]

@krallin yes, my bucket is in EU west. and yes, the workaround with the calling format stuff triggers an error that looks very similar to #443. currently, the only thing that works for me is monkey patching ..

[krallin]

@oberstet

Using a patched HTTP Connection seems to work too. The following code uses the standard library for SSL cert validation, but submits a different hostname for validation to pass (one that matches S3's cert).

It appears on Python 2.7.9 (though it will not work on an earlier version of Python, since those don't have ssl.SSLContext, so some conditionals would be required).

import logging
import socket, ssl
import re

import boto
from boto.https_connection import CertValidatingHTTPSConnection

logging.basicConfig(level=logging.WARNING)

class TestHttpConntection(CertValidatingHTTPSConnection):
    # !! Unsafe on Python < 2.7.9
    def __init__(self, *args, **kwargs):
        CertValidatingHTTPSConnection.__init__(self, *args, **kwargs)  # No super, it's an old-style class
        self.ssl_ctx = ssl.create_default_context(cafile=self.ca_certs)  # Defaults to cert validation
        if self.cert_file is not None:
            self.ssl_ctx.load_cert_chain(certfile=self.cert_file, keyfile=self.key_file)

    def connect(self):
        "Connect to a host on a given (SSL) port."
        if hasattr(self, "timeout"):
            sock = socket.create_connection((self.host, self.port), self.timeout)
        else:
            sock = socket.create_connection((self.host, self.port))

        if re.match(".*\.s3.*\.amazonaws\.com", self.host):
            patched_host = ".".join(self.host.rsplit(".", 4)[1:])
        boto.log.warn("Connecting to '%s', validated as '%s'", self.host, patched_host)
        self.sock = self.ssl_ctx.wrap_socket(sock, server_hostname=patched_host)


def main():
    from boto.s3.connection import S3Connection
    conn = S3Connection(https_connection_factory=(TestHttpConntection, ()))

    print conn.get_bucket("boto2836.us-east-1.test")
    print "Standard OK"

    print conn.get_bucket("boto2836.eu-west-1.test")
    print "EU OK"

if __name__ == "__main__":
    main()

Output:

WARNING:boto:Connecting to 'boto2836.us-east-1.test.s3.amazonaws.com', validated as 'test.s3.amazonaws.com'
<Bucket: boto2836.us-east-1.test>
Standard OK
WARNING:boto:Connecting to 'boto2836.eu-west-1.test.s3.amazonaws.com', validated as 'test.s3.amazonaws.com'
WARNING:boto:Connecting to 'boto2836.eu-west-1.test.s3-eu-west-1.amazonaws.com', validated as 'test.s3-eu-west-1.amazonaws.com'
<Bucket: boto2836.eu-west-1.test>
EU OK

[krallin]

Note that I'm still getting a 400 when connecting to S3 in Frankfurt, but I think that's because S3 in Frankfurt requires a different signature format.

[krallin]

I'm working on a patch here https://github.com/krallin/boto/compare/fix-2836

First, I'm making sure that cert validation is left up to Boto regardless of the Python version (which means that when validate_certs is None, as is in the case in S3Connection, certs are indeed not validated). This is done, so and I'm adding tests (I have a few done manually, just need to convert those to integration tests)

Finally, I'll also try and add an option for boto to accept certs for "dotted" buckets on S3.

Cheers,

[excieve]

Same issue here with 2.7.9. However aws-cli, which is based on the new botocore works fine in case a certain way of passing the args is used. There's a related issue in aws-cli.

[NamanJn]

Yeah same here, getting an error when accessing buckets with dots.

[m-tse]

I'm having similar issues.

With this code:

conn = S3Connection(awsAccessKeyID, awsSecretKey)

It works fine with a bucket name that has no periods in it. Like matt-test
But if a bucket name has a period name in it, like matt.test, I'll get the following error:

InvalidCertificateException: Host matt.test.s3.amazonaws.com returned an invalid certificate (remote hostname \"matt.test.s3.amazonaws.com\" does not match certificate): {'notAfter': 'Apr  9 23:59:59 2015 GMT', 'subjectAltName': ((u'DNS', '*.s3.amazonaws.com'), (u'DNS', 's3.amazonaws.com')), 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'Washington'),), (('localityName', u'Seattle'),), (('organizationName', u'Amazon.com Inc.'),), (('commonName', u'*.s3.amazonaws.com'),))}

And if I change the code to:

conn = S3Connection(awsAccessKeyID, awsSecretKey, calling_format=OrdinaryCallingFormat())

it now works when there are periods in the name, but fails when there are no periods in the name. Here's the failure for when there are no periods in the name:

<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>matt-test</Bucket><Endpoint>matt-test.s3.amazonaws.com</Endpoint><RequestId>DB89A58C5FFB8B2E</RequestId><HostId>ODxMzw0brxB4PyqpmGD+Ecff8lak6DuULecHrt3S6PHcRclft8tFaDjUXRXd62dm</HostId></Error>"

The only solution I've found to work is to if/else based on the type of bucket:

if '.' in bucketName:
    conn = S3Connection(awsAccessKeyID, awsSecretKey, calling_format=OrdinaryCallingFormat())
else:
    conn = S3Connection(awsAccessKeyID, awsSecretKey)

Hope that helps.

EDIT: It's still failing for international buckets (anything other than US Standard). I'm trying to figure that out next.

[gholms]

I'll post some info here to save a little time: the ordinary calling format uses older, "path-style" URLs like https://s3-us-west-2.amazonaws.com/bukkit/key to address things, which means when you create the S3Connection you have to ensure you point it at the right endpoint or the service will reply with a redirect that is intentionally difficult to handle automatically. As long as you use the DNS name that matches the region of the bucket you want to work with it should work just fine.

[m-tse]

@gholms thanks for the insight!

So I have buckets located in various regions with and without dots in their name that I have to upload to. I would like a clean solution to be able to handle them all. My current solution is now:

conn = boto.s3.connect_to_region(
    region,
    aws_access_key_id=awsAccessKeyID,
    aws_secret_access_key=awsSecretKey,
    calling_format=OrdinaryCallingFormat()
    )

But this requires me to map a region, (us-east-1, us-west-1, etc.) to each bucket, which is something I haven't had to do before. Previously the default calling format worked fine for buckets with dots in their name. Looking at my logs, it seems that starting February 13th, I began getting the ssl.CertificateError error mentioned in the first post of this issue thread for buckets with a dot in their name. Nothing in the code changed, although it's possible that some software on the box got updated.

If my goal is to not require a region with each bucket, is my only option to wait for @krallin 's PR?

[krallin]

@MATTSE ,

Unfortunately that PR doesn't really seem to be making much progress :( It's been lingering there for a while and I haven't really heard back ever.

As explained a bit above, the change might have been caused by you upgrading to Python 2.7.9, which broke a lot of stuff that used to silently work (albeit insecurely!) as far as SSL is concerned.

Cheers,

[Koff]

If you are experiencing this issue in airflow. Go the Admin -> Connection settings for your S3 connection, and add an extra key calling_format to your connection dictionary with boto.s3.connection.OrdinaryCallingFormat as value.

Your Extra field should look like:

{"aws_access_key_id":"_your_key_", "aws_secret_access_key": "_your_secret_", "calling_format": "boto.s3.connection.OrdinaryCallingFormat"}

[tolgahanuzun]

@YunanHu Thanks.
My problem is solved.

[benlk]

If you're looking for a solution that does not depend upon global configs, here's the code for connecting to a bucket in a non-us-east-1 region that has periods in its name:

import boto

def get_bucket( bucket_name ):
    """
    Establish a connection and get an S3 bucket
    """
 
    s3 = boto.s3.connect_to_region(
        'us-east-2',
        host='s3-us-east-2.amazonaws.com', # endpoint name from https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
        calling_format=OrdinaryCallingFormat()
    )

    return s3.get_bucket(bucket_name)

[ykhrustalev]

Based on @oberstet solution but with support of region zones

import ssl

_old_match_hostname = ssl.match_hostname


def remove_dot(host):
    """
    >>> remove_dot('a.x.s3-eu-west-1.amazonaws.com')
    'ax.s3-eu-west-1.amazonaws.com'
    >>> remove_dot('a.s3-eu-west-1.amazonaws.com')
    'a.s3-eu-west-1.amazonaws.com'
    >>> remove_dot('s3-eu-west-1.amazonaws.com')
    's3-eu-west-1.amazonaws.com'
    >>> remove_dot('a.x.s3-eu-west-1.example.com')
    'a.x.s3-eu-west-1.example.com'
    """
    if not host.endswith('.amazonaws.com'):
        return host
    parts = host.split('.')
    h = ''.join(parts[:-3])
    if h:
        h += '.'
    return h + '.'.join(parts[-3:])


def _new_match_hostname(cert, hostname):
    return _old_match_hostname(cert, remove_dot(hostname))


ssl.match_hostname = _new_match_hostname

[UrosOgrizovic]

@benlk Thanks for the function. For future reference, I'm writing to confirm that it's working in Python 3.7.4 and boto 2.49.0.