Modifications for python3 support. #1345

wants to merge 53 commits into


None yet

5 participants

kurin commented Feb 21, 2013

This creates a subdirectory under which the code can be converted to python3 compatible with 2to3.

It also makes several changes to the existing code base -- some substantial -- mostly to ensure that strings are in the correct format for python3.

redsymbol and others added some commits Jan 29, 2013
@redsymbol redsymbol Infrastructure and initial dev docs for python 3 porting work 1cebc38
@kurin kurin Fix rfc822 import errors in python3 3c1be09
@kurin kurin Intercept calls to the hmac module
This monkeypatches the hmac module so that calls to .new() and
.update() are done, in python3, with actual bytes objects.
@kurin kurin Modify sha256 always to pass bytes in python3
The same as the previous patch.

Also this patch monkeypatches hashlib.  This code should probably
be pulled and put elsewhere, but I don't know where else is good.
@kurin kurin Patch the sax parser for python3 strings. 82c3dde
@kurin kurin Ensure byte strings actually are byte strings 6e29ec0
@kurin kurin Change the rounding type
From what I can tell the rounding type doesn't matter, but who knows
@kurin kurin Convert to bytes 0ebadc1
@kurin kurin cmp is deprecated, use cmp_to_key bea296b
@kurin kurin Change hex and string conversions.
The vault code at one point (line 59, here) makes a point of encoding
unicode strings into utf-8.  This breaks the unit tests in python3,
which aren't explicitly expecting byte strings.  They almost certainly
should be, but I didn't feel qualified to make such sweeping changes
to the unit tests, so for now I've put in a workaround that causes
python3 strings to remain as they are.

Also, python3 no longer has str.encode('hex')
@kurin kurin Change the Binary class and _encode_s method
Bring these in line with the other python3 encoding changes.
@kurin kurin Tag objects cannot also be None
This I think is a straight-up bug.
@kurin kurin Change the test for sets of strings.
The original code assumed that iterating through a set() would throw
up the contents in a predictable order.  This is not the case, and
in python3 apparently these specific strings are returned in the
opposite order.
@kurin kurin Turn a string into a byte literal.
This is required to work in python3.  It should not affect the test.
@kurin kurin Change the way the hex is expressed.
python3 doesn't have str.encode('hex'), and reaches those values
with a different expression.  I have replaced the longhand with the
specific values, which should not alter the test.
@kurin kurin Modify the test to also accept python3 exceptions.
In python3, IOError (which is apparently also OSError) does not
have a message attribute.
@kurin kurin Pass bytes to md5(), not a string, in python3. aef21ec
@kurin kurin Assist 2to3 in dealing with unittest.mock
Apparently "import mock" and "from mock import whatever" are used
all over the place.  As this is pretty experimental, I've just done
a simple find job to fix it.  This works for FreeBSD's sed(1) but
hasn't been checked against whatever else is out there.
@kurin kurin Adjust the output of base64.encodestring in p3k
Apparently a string of the form "%s:%s" % (x, y), when y is a byte
literal, will be formatted as "this-is-x:b'this-is-y'", which
obviously AWS isn't expecting.  So this patch is further insanity
to ensure that the output of base64.encodestring is a utf-8 encoded
string, not a byte literal.
@kurin kurin Python3 needs BytesIO not StringIO
Doubtless I'll have to change this in a number of places.
@kurin kurin *Don't* encode each unicode header.
This probably breaks stuff, which is why it's only commented out.
@kurin kurin Again, BytesIO and not StringIO 1f57139
@kurin kurin Re-dencode encoded and quoted header strings. b034808
@kurin kurin Only encode headers that have non-ascii characters.
So, we have to be even clever-er to avoid running certain headers
(namely the MD5 checksup) through urllib.quote_plus.

Honestly at this point we should probably just check for it explicitly.
@kurin kurin Remove a hack.
So in 2.5 and below, apparently "host" had to be amended to "host:port"
for compatibility with httplib.  The code would check to see if we
were running 2.6 and 2.7, which in 3.3 we *aren't*, but I guess it
never got the memo.

2.5 and older aren't supported now anyway, so remove the check entirely.
@kurin kurin Properly handle the mock -> unittest.mock conversion. 8cf4f8e
@kurin kurin python 2.6 doesn't have functools.cmp_to_key 739261a
@kurin kurin Clean up the exception assertions. 964e4b9
@kurin kurin Merge branch 'develop' of into develop 70ec23c
@kurin kurin Fix tests to conform to python3
httpretty returns byte literals, which need to be decoded
@kurin kurin Fixes to handle byte literal returns 2915451
@kurin kurin now with httpretty
Which does support python3, thank goodness

Hi Toby,

I notice the list of files where you made mods doesn't include any of the Google Cloud Storage code (boto/boto/gs/)
We'd like to also get the GCS code to work under Python 3. We could help with that work (unless you'd prefer to make all the changes in this CL, to keep them uniform).


Mike Schwartz

kurin commented Feb 22, 2013

Sure, I'd be happy to look at that. The commits here really only bring the unit tests and my personal use of S3 and EC2 into line; I hadn't looked at GCS (or any other module) yet.


Great. Let me know if you'd like any help from us, otherwise I'll just look
forward to reviewing / testing your changes.



On Thu, Feb 21, 2013 at 5:53 PM, Toby Burress notifications@github.comwrote:

Sure, I'd be happy to look at that. The commits here really only bring the
unit tests and my personal use of S3 and EC2 into line; I hadn't looked at
GCS (or any other module) yet.

Reply to this email directly or view it on GitHub

kurin added some commits Feb 22, 2013
@kurin kurin Another instance of StringIO -> BytesIO
I should probably put a function in but this pattern shows
up rarely enough.
@kurin kurin Merge remote-tracking branch 'upstream/develop' into develop
Keeping up with upstream.
kurin commented Feb 22, 2013

I don't know how that merge made it into the pull request; I'll probably have to close this one and open a new one.

But I can now do a gsutil perfdiag without failures, which I don't know how much of GCS's functionality that uses, but it's at least some of it.

kurin commented Feb 24, 2013

The s3 integration tests now pass successfully in 2.6, 2.7, and 3.3.

garnaat commented Feb 24, 2013

I was just trying this out but ran into an issue with pyami/ I don't think that's been updated yet. I'm wondering how you are running your integration tests without it, though.

Also, we should look at some of the existing directories and decide whether they need to be there anymore or not. For example, boto/roboto, boto/manage, boto/mashups, etc. that are really old and not really used. This would be a good time to break those out as separate packages or just get rid of them altogether. Certainly no reason to try to port them to Python 3.x.

kurin commented Feb 24, 2013

So far I've only done the unit tests and the s3 integration tests. I was going to tackle ec2's integration tests next; I haven't touched anything under pyami/.

kurin commented Feb 24, 2013

I also don't want to wholesale remove any parts of the library, because I don't think I'm qualified to make that decision. But I mean if they happened to disappear the next time I merged from develop, I wouldn't mourn them.

kurin added some commits Feb 27, 2013
@kurin kurin Merge remote-tracking branch 'upstream/develop' into develop 9470a4f
@kurin kurin Add port to server_name method.
Apparently this is used to create HTTP objects, instead of passing
the port explicitly.
@kurin kurin Update test to expect bytes. 139f0f4
@kurin kurin Update the text/bytes handling in SQS messages.
SQS message are always unicode, but they have to be converted to/from
bytes to make it through the base64 encoder.
@kurin kurin Modify to build both python3 and python2
Now calls to ` bdist`, in python3, will run 2to3 on the
source.  This requires a pretty recent install of Distribute (0.6.31

This will allow a single source package to install in both python2
and python3.
kurin commented Feb 28, 2013

For the record, this now also passes the ec2 and sqs integration tests.

It also has a modification to so that when installed by python3, it automatically converts with 2to3. This way you can distribute a single source dist.


Ok, thanks Toby.


On Wed, Feb 27, 2013 at 7:16 PM, Toby Burress notifications@github.comwrote:

For the record, this now also passes the ec2 and sqs integration tests.

It also has a modification to so that when installed by python3,
it automatically converts with 2to3. This way you can distribute a single
source dist.

Reply to this email directly or view it on GitHub

extesy commented Mar 28, 2013

+1 to get boto working on python 3.3.

kurin commented Mar 28, 2013

Hi extesy, there's actually a branch now for python3, py3kport. It's a few bugfixes behind develop but nothing major. Please check it out and let me know if you run into any issues. You can build it by running install (it will automatically run 2to3 when installing) and the 3.3 requirements are in py3kport/requirements.txt-3.3. Let me know if you hit any bugs!

extesy commented May 20, 2013

@kurin Thanks for your work on this port. The problem with manually building from a separate branch is that it's not updated as often as a master. I would really prefer to simply use pip for install and get all new fixes as they are released.

@garnaat I see that you have just pulled the latest changes into py3kport branch. Are you going to merge everything into master any time soon?

kurin commented May 22, 2013

Yeah. It's pretty close to that now, but some patches coming in to develop need some massaging to get them to work in python3. I don't know when the python3 changes (which, in sum, aren't really trivial, although new modifications are) are going to make it into develop, but for my part I'll try merging from develop into py3kport every release or so.

kurin commented May 22, 2013

Also also, this pull request is no longer valid. Python3 support should be pulled in from the py3kport branch now.

@kurin kurin closed this May 22, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment