Make boto.ec2 work with Python 3.2 and Python 3.3 #1156

Closed
wants to merge 24 commits into from

3 participants

@msabramo

A bunch of changes to make boto.ec2 tests pass on Python 3.2 and Python 3.3 while all boto tests continue to pass on Python 2. There has been some manual testing as well to verify that basic functionality works (e.g.: printing all available images) when connecting to Eucalyptus Community Cloud (see here) and EC2 (see here).

Passing Travis CI build: https://travis-ci.org/msabramo/boto/builds/3465072

This is the work that I alluded to in #1127.

msabramo added some commits Nov 25, 2012
@msabramo msabramo Update boto/pyami/config.py for Python 3 c3d1f67
@msabramo msabramo Update boto/exception.py for Python 3 6f43e29
@msabramo msabramo Add stuff to tox.ini for testing py32 and py33 with boto.ec2 alone 8fe81b3
@msabramo msabramo Remove simplejson from requirements.txt as it causes a failure with p…
…y33 (and I think it's not needed with py26 and py27?)
23bb84e
@msabramo msabramo Make tests/unit/__init__.py Python 3 friendly by abstracting out that…
… stuff moved from httplib to http.client in Python 3
17f11d7
@msabramo msabramo Make boto/ec2/connection.py Python 3 friendly by changing "except com…
…ma" syntax to "except as" syntax
9a7ad50
@msabramo msabramo Make boto/connection.py more Python 3 friendly d363a67
@msabramo msabramo Make boto/utils.py more Python 3 friendly e26f487
@msabramo msabramo Make boto/auth_handler.py Python 3 friendly 2ad1ff1
@msabramo msabramo Make test commands in tox.ini consistent 779736d
@msabramo msabramo Use newer o0 notation for octal number in boto/ec2/keypair.py 3d12d0c
@msabramo msabramo Put parentheses around print calls in boto/ec2/reservedinstance.py 0e6823c
@msabramo msabramo Tweak HTTPS detection to work in Python 3 in boto/connection.py and b…
…oto/http_connection.py
9dfc91e
@msabramo msabramo Paper over lack of `long` type in Python 3 in boto/connection.py 2bd3784
@msabramo msabramo Update boto/__init__.py for Python 3 4a0debc
@msabramo msabramo A bunch of changes to support Python 3. Tests passing in Python 3 now…
… for boto.ec2!
5c91d91
@msabramo msabramo Use same default testenv command as on `develop` branch. Add a testen…
…v for testing boto.ec2 only with py26
364c980
@msabramo msabramo .travis.yml: Test Python 3.2 but only run tests for boto.ec2 since th…
…at's all that I've ported to Python 3 so far.
93f54ed
@msabramo

Has anyone gotten a chance to try this out? I don't actually have a use case for boto at the moment, so I haven't gotten a chance to use this code for real.

@skiold

py27 and py33_ec2 test passing with msabramo@0223797

But that code tree breaks when trying to open a connection to the api (see https://gist.github.com/skiold/5006231). I get the same error with py33 and py27.

@msabramo

Thanks, @skiold, I think I see what the problem might be. Hoping to have a fix soon.

@msabramo msabramo Fix infinite recursion error reported by @skiold on GitHub at
boto#1156 (comment)

I reverted back to qualifying stuff from `httplib` using `httplib.`;
because otherwise `HTTPResponse` is ambiguous and may refer to itself
instead of the version in `httplib`.
d2654f0
@msabramo

@skiold et al, I added commit d2654f0 which should fix the issue reported by @skiold.

@msabramo

Tox passing:

marca@marca-mac:~/dev/git-repos/boto$ tox
...
  py26: commands succeeded
  py27: commands succeeded
  py32_ec2: commands succeeded
  py33_ec2: commands succeeded
  congratulations :)

Passing Travis CI build: https://travis-ci.org/msabramo/boto/builds/4988307

Bug reported by @skiold seems to be fixed -- now it prints a proper HTTP 401 authorization failure message; (I don't have a real EC2 key and don't feel like registering for 1 year of free tier just for this):

marca@marca-mac:~/dev/git-repos/boto$ AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="yyy" .tox/py27/bin/python -c 'import boto.ec2; boto.ec2.regions()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "boto/ec2/__init__.py", line 40, in regions
    return c.get_all_regions()
  File "boto/ec2/connection.py", line 2665, in get_all_regions
    [('item', RegionInfo)], verb='POST')
  File "boto/connection.py", line 989, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>0e8a7e6e-d4f9-49cb-834e-9e63e306f355</RequestID></Response>

marca@marca-mac:~/dev/git-repos/boto$ AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="yyy" .tox/py32/bin/python -c 'import boto.ec2; boto.ec2.regions()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "boto/ec2/__init__.py", line 40, in regions
    return c.get_all_regions()
  File "boto/ec2/connection.py", line 2665, in get_all_regions
    [('item', RegionInfo)], verb='POST')
  File "boto/connection.py", line 989, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>56687567-1e72-474a-aad0-ff48645e61ef</RequestID></Response>

marca@marca-mac:~/dev/git-repos/boto$ AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="yyy" .tox/py33/bin/python -c 'import boto.ec2; boto.ec2.regions()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "./boto/ec2/__init__.py", line 40, in regions
    return c.get_all_regions()
  File "./boto/ec2/connection.py", line 2665, in get_all_regions
    [('item', RegionInfo)], verb='POST')
  File "./boto/connection.py", line 989, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>fe1fb9d1-c107-4d7c-a067-96bb74bb0495</RequestID></Response>

If folks could try this out with real AWS credentials, that would be great.

@msabramo

OK, I've managed to get this to connect to the Eucalyptus Community Cloud. I signed up for an account and then used the instructions at this blog post to configure it.

marca@marca-mac:~/dev/git-repos/boto$ cat print_euca_images.py 
import boto

euca = boto.connect_euca()

print(euca.get_all_images())

marca@marca-mac:~/dev/git-repos/boto$ .tox/py26/bin/python print_euca_images.py 
[Image:emi-F86C3FE3, Image:emi-51DD3B86, ...]

marca@marca-mac:~/dev/git-repos/boto$ .tox/py27/bin/python print_euca_images.py 
[Image:emi-F86C3FE3, Image:emi-51DD3B86, ...]

marca@marca-mac:~/dev/git-repos/boto$ .tox/py32/bin/python print_euca_images.py 
[Image:emi-F86C3FE3, Image:emi-51DD3B86, ...]

marca@marca-mac:~/dev/git-repos/boto$ .tox/py33/bin/python print_euca_images.py 
[Image:emi-F86C3FE3, Image:emi-51DD3B86, ...]

Note that I can connect to euca and print a list of images with Python 2.6, 2.7, 3.2, and 3.3.

@skiold

Working against ec2 also
👍

@msabramo

For adventurous folks who are interesting in using boto.s3 on Python 3, have a look at https://github.com/msabramo/boto/tree/pr_python3_boto_s3_2013-02-26 -- in that tree I have 8 out of 9 of the boto.s3 unit tests passing. I have not tried to connect to Eucalyptus Community Cloud or anything though.

@kurin
@msabramo

Interesting. I wonder if our changes are similar. It does seem like a good idea to have a central repo where folks can collaborate.

I wonder if @garnaat is open to merging this stuff -- I just recalled that he had been toying with the idea of doing a rewrite of boto as boto3. Not sure if that is still the thought.

I'll be at PyCon US in March -- I wonder if this is something that folks might want to sprint on.

@kurin
@msabramo

That's great news that @garnaat is open to merging our changes. Yeah, a fork was tried before (the "neo" branch of this repo) and the problem was that it just got out of sync with the develop and master branches). Getting it merged into the mainline I think is the way to go and glad that @garnaat will consider it.

Yeah, most of my changes deal with bytes vs. text. We probably made a lot of the same changes.

If your fork passes integration tests as well, then maybe your fork is the better one to start with? I didn't try integration tests although there was a bit of manual testing that you can see above.

@kurin
@msabramo

The major change between your fork and mine is that you used 2to3 and I opted not to use it, because I wanted to see if I could make it work without the preprocessing. It does work, though it requires a lot more changes of mundane stuff that 2to3 could handle.

I wonder if @garnaat et al want to use 2to3 or not? If they do, then your fork is probably a better base.

@msabramo

@garnaat: Do you have any interest in this PR? The reason that I ask is I think that with the new version of six, which now has abstractions for various urllib and urlparse functions, it might be possible to revise this and simplify a bit. But I don't want to invest further time in this if it's not something that you're interested in merging.

@msabramo

@garnaat: Are you open to using something like this?

@msabramo

ping

@msabramo

OpenStack wants boto for Python 3. They are currently sprinting on Python 3 at Pycon and they have a dependency on boto. Do they have to wait for boto3? Maybe chat with them if you're at Pycon or ping @haypo who is spearheading the Python 3 effort...

@kurin

I'm not sure if you're pinging me or @garnaat, but for my part unfortunately I don't have any time to work on this.

@msabramo

Maybe @garnaat and @haypo might want to chat about what's the best thing for OpenStack to do. Perhaps OpenStack should be moving to something else like boto3 (currently has "experimental" and "not production-ready" stamped all over it) or botocore or aws-cli? @garnaat (or someone else) might have good advice here.

@msabramo msabramo closed this Jan 13, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment