UnboundLocalError: local variable 'conn' referenced before assignment

[andygock]

I have a problem when working with:

• CentOS Linux release 7.2.1511 (Core)
• Python 2.7.5

Problem does not occur with:

• CentOS release 6.7 (Final)
• Python 2.6.6

This is with Dreamhost Dreamobjects, and have not tried with Amazon S3.

All with the same config file:

[default]
access_key = XXX
secret_key = XXX
host_base = objects.dreamhost.com
host_bucket = %(bucket)s.objects.dreamhost.com
enable_multipart = True
multipart_chunk_size_mb = 15
use_https = True

On the CentOS 7.x machine, when running:

s3cmd -c ~/.s3cfg_dream ls s3://BUCKETNAME

I get (replaced personal info):

Invoked as: ./s3cmd -c /root/.s3cfg_dream ls s3://BUCKET
Problem: CertificateError: hostname 'BUCKET.objects.dreamhost.com' doesn't match either of '*.objects.dreamhost.com', 'objects.dreamhost.com'
S3cmd:   1.6.1
python:   2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
environment LANG=en_US.UTF-8

Traceback (most recent call last):
  File "./s3cmd", line 2919, in <module>
    rc = main()
  File "./s3cmd", line 2841, in main
    rc = cmd_func(args)
  File "./s3cmd", line 120, in cmd_ls
    subcmd_bucket_list(s3, uri)
  File "./s3cmd", line 153, in subcmd_bucket_list
    response = s3.bucket_list(bucket, prefix = prefix)
  File "/root/s3cmd/S3/S3.py", line 297, in bucket_list
    for dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params):
  File "/root/s3cmd/S3/S3.py", line 324, in bucket_list_streaming
    response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params)
  File "/root/s3cmd/S3/S3.py", line 343, in bucket_list_noparse
    response = self.send_request(request)
  File "/root/s3cmd/S3/S3.py", line 1081, in send_request
    conn = ConnMan.get(self.get_hostname(resource['bucket']))
  File "/root/s3cmd/S3/ConnMan.py", line 192, in get
    conn.c.connect()
  File "/usr/lib64/python2.7/httplib.py", line 1198, in connect
    server_hostname=sni_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/usr/lib64/python2.7/ssl.py", line 588, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 818, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 269, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
CertificateError: hostname 'BUCKET.objects.dreamhost.com' doesn't match either of '*.objects.dreamhost.com', 'objects.dreamhost.com'

So I try instead:

s3cmd -c ~/.s3cfg_dream --no-check-certificate --no-check-hostname ls s3://BUCKET

And I get:

Invoked as: ./s3cmd -c /root/.s3cfg_dream --no-check-certificate --no-check-hostname ls s3://BUCKET
Problem: UnboundLocalError: local variable 'conn' referenced before assignment
S3cmd:   1.6.1
python:   2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
environment LANG=en_US.UTF-8

Traceback (most recent call last):
  File "./s3cmd", line 2919, in <module>
    rc = main()
  File "./s3cmd", line 2841, in main
    rc = cmd_func(args)
  File "./s3cmd", line 120, in cmd_ls
    subcmd_bucket_list(s3, uri)
  File "./s3cmd", line 153, in subcmd_bucket_list
    response = s3.bucket_list(bucket, prefix = prefix)
  File "/root/s3cmd/S3/S3.py", line 297, in bucket_list
    for dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params):
  File "/root/s3cmd/S3/S3.py", line 324, in bucket_list_streaming
    response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params)
  File "/root/s3cmd/S3/S3.py", line 343, in bucket_list_noparse
    response = self.send_request(request)
  File "/root/s3cmd/S3/S3.py", line 1106, in send_request
    conn.counter = ConnMan.conn_max_counter
UnboundLocalError: local variable 'conn' referenced before assignment

Looks like problem may be related to this though I don't know how to reopen a closed issue.

I'm using s3cmd version 1.6.1

I also tried a few older versions, but with no success.

All works fine with Centos 6.x and Python 2.6

It use to work on my CentOS 7 machines, but stopped working some time after mid December 2015 for some reason, probably after certain distro updates.

Any help would be appreciated. Thanks.

[mjtko]

Hi @andygock: can you try my fix here and see if that works for you please? #714

[mjtko]

Just to provide a bit more background: the latest fix special-cased s3.amazonaws.com names to allow the *.s3.amazonaws.com wildcard certificate to match in a more forgiving way (as per RFC, while *.example.com does match x.example.com it does not match y.x.example.com).

The fix I propose in #714 generalizes the special-case for *.s3.amazonaws.com and extends it to all wildcard certificates when dealing with bucket names that contain the . character.

[mdomsch]

How is that different than --no-check-certificate?
On Mar 7, 2016 6:00 AM, "Mark J. Titorenko" notifications@github.com
wrote:

Just to provide a bit more background: the latest fix special-cased
s3.amazonaws.com names to allow the *.s3.amazonaws.com wildcard
certificate to match in a more forgiving way (as per RFC, while *.
example.com does match x.example.com it does not match y.x.example.com).

The fix I propose in #714 #714
generalizes the special-case for *.s3.amazonaws.com and extends it to all
wildcard certificates when dealing with bucket names that contain the .
character.

—
Reply to this email directly or view it on GitHub
#712 (comment).

[mjtko]

--no-check-certificate remains broken for anything other than s3.amazonaws.com due to having special-casing only apply for .amazonaws.com @ https://github.com/s3tools/s3cmd/blob/master/S3/ConnMan.py#L118

        if (hostname.endswith('.amazonaws.com') or hostname.endswith('.amazonaws.com.cn')):
                # this merely delays running the hostname check until
                # after the connection is made and we get control
                # back.  We then run the same check, relaxed for S3's
                # wildcard certificates.
                debug(u'Recognized AWS S3 host, disabling initial SSL hostname check')
                check_hostname = False
                if context:
                    context.check_hostname = False
            conn = httplib.HTTPSConnection(hostname, port, context=context, check_hostname=check_hostname)

If you're using, say, storageapis.google.com it all falls apart when establishing the HTTPSConnection.

The fix in #714 applies the same principle to all buckets with . in their name. As an added bonus, it doesn't apply the same principle to S3 buckets without a . in them.

Note that this failure requires a "modern" python to reproduce correctly, such as that included in EL7.2.

[mjtko]

I should also point out that the certificates are actually still being checked, it's only the match against a wildcard certificate that is being relaxed (in the same way as has previously been relaxed for *.s3.amazonaws.com), which also makes it quite different from --no-check-certificate which skips certificate verification entirely!

[andygock]

Thanks @mjtko - I tried the #714 fix, and that works now (with --no-check-certificate as well)

[iby]

Also getting this with 1.6.1 on mac, earlier versions used to work. @mjtko thanks for providing a fix, any chance you can respond to comments in your pull request?

[mjtko]

@ianbytchek: I've (finally!) found a moment to respond. Sorry for delay. If you have any suggestions or input that will harden up this change, please also comment in #714.

[fviard]

Certificate issues should be completely resolved in the latest master I hope, don't hesitate to give it a try and re-open in case of issue.
If there was any issue, at least the "UnboundLocalError" should not be encountered anymore and the real error being displayed.