Add support for Python 3.x

[garnaat]

This is being actively worked in the neo branch.

[hroncok]

Hi, this issue is 2 years old. How are we standing now with Python 3?

Would you like some help with that?

[kurin]

I have a branch that supports Python 3, but I don't really keep it up to date because the demand for Python 3 is far, far, far less than you might think. It's the py3kport branch here.

[hroncok]

Is the Python3 branch working with Python 2.7 as well? Or is there some reason why not to merge it?

[kurin]

Yeah it works with 2.6, 2.7, and 3.3. It doesn't work with 2.5. You'd have to ask Mitch re: merging, but personally I wouldn't want to merge it yet either, until it'd seen a lot of usage in the real world, which again see above re: demand. You're the second or maybe third person to ask about it in ~6 months.

[hroncok]

Well, we want it for Fedora. So I guess that would bring some users.

https://fedoraproject.org/wiki/Changes/Python_3_as_Default

Boto is needed in our cloud image.

Thanks for info.

[kurin]

I can update my branch tonight/this week and send you a link when it's passing all the unit tests again. All the heavy lifting is pretty much done.

[hroncok]

Thanks a lot, that would be great.

[kurin]

Okay, so a possible 2.15 is available in my repository here: https://github.com/kurin/boto/tree/py3kport

Give that a shot. Open bugs against it if it doesn't work, etc. Let me know how it goes.

[kurin]

I meant to add: it still fails 3 unit tests, because python 3 randomizes the order of dictionary keys, and python 2 (by default) doesn't. Maybe there's a graceful way around it, but I dunno it yet, and fixing these is otherwise a pita, so I didn't.

[hroncok]

So the question here is:

Is the order of that dict important in the real word?

If it is, then the fix is needed indeed. If it is not, then the tests should be fixed, right?

[kurin]

Well, you can run the tests and see, but they are:

File "/usr/home/kurin/repos/boto/py3kport/build/tests/unit/dynamodb2/test_table.py", line 450, in test_prepare_full

File "/usr/home/kurin/repos/boto/py3kport/build/tests/unit/sns/test_connection.py", line 229, in test_publish_with_json

and at least one other that isn't failing just now.

The upshot is that I don't believe that they're indicative of actual failures, but as I said the effort/reward formula for this isn't really in my favor.

[hroncok]

How do you run the tests? Cause test.py in your branch is failing with print statement, so I guess it's not that way.

[kurin]

Just run tox, or tox -e py33 if you just want to test the python 3.3 branch.

[kurin]

@hroncok, is this sufficient, or how is it re: your shift to Python3?

[hroncok]

I'm a bit surprised how the Python 3 branch is done. Doing changes and than calling 2to3. However I haven't time to look at it carefully yet.

[kurin]

I think of all the alternatives, this one required the least number of
changes (and least invasive changes) to the 2.x code base itself.

[wlonk]

I'd like to voice more support for getting a working py3 version of Boto.

[kurin]

Try the branch I linked, and let me know of any bugs that aren't in the current development version of boto on 2.x.

[wlonk]

I will, thanks!

[ztane]

I have used the py3 branch a bit, we're interested in converting everything to py3, so far the boto is almost the only component that is holding us back, not the other way.

[ztane]

That is, whenever ppl start a new project, they see if boto supports py3, if not, then they silently agree that "it can't be done effortlessly" and use py2 instead.

[kurin]

@ztane You mean you used the py3 branch of boto? How was it? Did it meet your needs?

[ztane]

In my only boto-using py3 project I used only S3, and for it it did work correctly.

However, we have another project that we would be liking to port to python 3, for that we would also use ELB, instance and image mgmt from boto. Currently we have some other libraries as partial showstoppers (natural language processing), so we haven't migrated it yet.

[hroncok]

I don't know what I'm doing wrong, but I cannot get httpretty to work with Python 3. It's listed in your Python 3 requirements.

[ztane]

I don't think the requirements is right at all for py3, argparse for example is a builtin module already.

[ztane]

Anyway, it is used by the tests only, so who cares.

[hroncok]

I care, I want to see the test pass :)

[kurin]

I think you are installing the python2 requirements, which are what is
in requirements.txt.

You want py3kport/requirements.txt-3.3, which doesn't have argparse and
which bumps the httpretty req to 0.5.10.

[hroncok]

Well, I cannot pass httpretty's tests on 0.5.10 either. And I also thinks that requiring one version specifically is not a long term solution.

[kurin]

Can you be more specific about which tests are failing and how?

What version of Python are you using?

What is the output of pip freeze in your virtual environment?

Thanks

[kurin]

How are you installing the library? You should just run setup.py with a python3 binary, and it will automatically run 2to3 over the code base. If you're running into issues that 2to3 would normally catch, either you're not running setup.py with python3, or something's busted in the 2to3 step.

[danielgtaylor]

Botocore is being actively worked on and as @kurin said it will be the next generation and does already support Python 3. It is currently considered _unstable_ because the API will probably change before we recommend that anybody seriously use it, but the AWS CLI is already using it successfully. Python 3 support is an important feature to us.

[jamesshannon]

Hi Daniel. Thanks for the note. I ended up giving up on the py3k branches and at someone's suggestion on #boto I tried botocore. It's obviously a bit rough around the edges, but seems to be working acceptably.

With that being said, and I realize this is "free" software (though clearly, Amazon makes a lot of money from AWS) and that I'm brand new to the Python community, but with all due respect it just doesn't feel like py3 support is that important. It appears that Python 3 was released over 5 years ago. IE8 was released more recently.

Before I found botocore (which I wouldn't have tried based on the github description), I thought I'd have to backport my entire project (or switch to PHP) -- just because of an ancillary service. If the client didn't explicitly want AWS (being an existing customer for other things), I'd have pushed hard to use alternative services.

[danielgtaylor]

@jamesshannon I completely understand the disappointment and frustration you (and many others) are feeling. Boto is an interesting project for many reasons, one of which is that it has had many contributions from many owners over many years and was only recently officially sponsored by Amazon. While that's certainly no excuse, it does mean that we've had to make tough decisions about where to focus our efforts and how to move forward, the result of which is and will be Botocore, which was built from the ground up to support both Python 2 and Python 3, and we take great pains to test on multiple Python versions.

I can't comment on specifics yet, but I can say that things in this regard should be getting better soon.

[meawoppl]

Another +1 for py3 here. Our entire tool chain switched over, and I was suprised to find this (largest) module was still straggling. Our core functionality tests in ok on the py3 branch.

[darkness51]

+1 for py3k here

[arthurprs]

+1

[nicksloan]

@danielgtaylor Can we close this issue since it is irrelevant to THIS repository, and there isn't likely to be any further practical discussion here. +1s aren't helpful. Seems like it should be closed with a final message of "Follow boto/botocore for further information on Python 3 support."

[gotpredictions]

Should we assume that the recommendation for Python 3'ers is to port their code to botocore instead of using boto? If this issue is going to be closed, it would good to have a firm recommendation for people like us using boto today on our python 3 approach.

[nicksloan]

@gotpredictions The current boto API will never support Python 3. For Python 3 support, you have to port your code to one of two options: botocore (low-level, mostly available now) or boto3 (high-level, still a ways off). As I understand it, both projects will support Python 2.7 and reasonable versions of Python 3.x. If you want to port your project to Python 3 soon, you should plan on switching to botocore. If you can wait for boto3, it will likely offer a simpler API. There will be no Python 3 version that supports boto's current API.

That is why I advocate for closing this issue. Python 3 support is being worked on, but not in this repository. +1-ing is useless because the project is already underway. It is happening now. The most important takeaway is that for Python 3 support, you will have to port your code to a new API no matter what.

That's my understanding of things. If any of that is wrong, hopefully someone can correct me.

[ztane]

There has been a python 3 port for years in "experimental branch" that we use in production without any problems... only we cannot install it from pypi bc "boto will not support python 3". This is bullshit.

There are also no commits to boto 3 in like 5 months! So telling to wait for boto 3 is bullshit too, "until what day?" "IDK".

Telling to write ones own api over botocore to get started with python 3 is bullshit too.

The problem is for deciding against merging anything related to python 3 to boto main branch because "well we don't."

[meawoppl]

Wow. I didn't realize that boto3 was becoming a stale project, can anyone give an update on it? I figured given its explicit support by AWS, it would be a bit more expedited.

I did end up writing a wrapper over botocore, and I am happy with its performance. Getting s3 working (fairly extensive use) took only an hour or so.

[nicksloan]

@ztane I'm not affiliated with the boto project or Amazon in any way, so take what I am saying with a grain of salt. That said, +1s are worthless, on the other hand, I think what IS worthwhile is to argue against the notion of creating an additional burden in the transition to Python 3.x by creating a new, incompatible API, rather than updating the old, widely used API, and I'd be inclined to agree with such an argument. I think the boto project has treated Python 3 with outright hostility, and it has been a gross disservice to users.

With that said, boto is an open source project, so you are not without recourse. You seem to have experience with an "experimental branch" that does support Python 3. I'd propose that you fork the repository, and merge that into master. You could publish your forked version on PyPi, thereby scratching your own itch, while providing a valuable service for many, many other people who are in your shoes. With luck, your efforts might light a fire under the asses of those working on boto3 to try to save what little Python 3 users they will be able to muster once your fork begins siphoning them away.

No need for outrage. No one owes you a thing. If you want something done, contribute.

[ztane]

The downside of such a fork would be that it would necessarily fall behind (as it has happened with these experimental branches) as it would become more burdensome to merge patches from python 2 only repositories. And also, its existence and low download numbers could be used again as an excuse to further delay work on other Python 3 supporting projects, possibly backed by more resources. As such I am very happy with the experimental branches myself - they do the work for me; I just wish the official stance would change to merge them into the master branch as it is the only really sustainable way forward - or, if the boto3 is the future then lets concentrate the effort there.

[danielgtaylor]

Python 3 support has been an important feature request from our customers, and today we're happy to announce new plans for Python 3 in Boto.

With the help of our developer community, we plan to add support for Python 3.3+ to Boto while continuing to support Python 2.6 and 2.7. We will add support on a module by module basis, with our initial implementation covering a couple of services and all of the common Boto code. We will accept pull requests for other service modules, provided that they follow our contributing guidelines. In a nutshell, this means they pass the unit and integration tests for Python versions 2.6, 2.7, 3.3, and 3.4. Adding new tests for uncovered modified code is encouraged.

This new Python 3.3 support is being developed now and will be landing in Boto within a few weeks. A side effect of this is that Boto will no longer be able to support Python 2.5, and as a result, we will not be able to accept any more patches related to Python 2.5.

Our long-term plans remain largely unchanged. Botocore is going to be the basis of the next major version of Boto, Boto 3, as well as the underlying library on which the AWS CLI is built. Boto 3 will be adding high-level, object-style interfaces on top of Botocore, similar to what is available in Boto today but also data-driven like Botocore. The data-driven architecture will allow us to provide complete coverage and timely support for every new update and service coming from AWS. We hope that this and other advantages will encourage you to adopt Boto 3 for new projects in the future.

[jcdyer]

Will you be using six to support both versions of python? I've got a fork where I did a fair amount of work toward a single-codebase port to adding python3 support using six. It worked very well in my experience. I'm happy to share what I've done, and contribute more. I'm not sure what I have right now is pull-ready, though.

Also, thank you for this announcement. I'm very excited to see python3 supported in boto. This is the last piece to being able to upgrade some of my django projects.

[danielgtaylor]

@jcdyer, I plan to use the six module, which I will be vendoring into the Boto codebase so that our customers don't need to install any extra dependencies. I'm also updating the boto.compat module to help abstract away differences so that six doesn't need to be used everywhere, similar to how Botocore is set up. I currently have a significant portion of the unit test suite working, as well as a couple service module integration tests.

I'll take a look at your fork and see if there's anything I can borrow. I expect to push up a branch of my work within the next few days so that people can start playing with it.

[nicksloan]

@danielgtaylor This is VERY welcome news! I was disappointed by the previous plans to only provide Python 3 support against a new API. This will be a huge boon to anyone who has ever hoped to port a large boto-dependent app to Python 3, and this marks one of the last major milestones on the road to Python 3 adoption. I've never been happier to be totally wrong.

[dukedougal]

So what's the point of botocore if Python 3 support is being added to Boto? Does this mean that Botocore has failed as a project? Will both Boto and Botocore be maintained into the future?

[dukedougal]

What's the timeframe on this? How long till botocore is production ready with documentation, and won't this announcement to support Python 3 into Boto slow down the botocore project?

[jamesls]

@dukedougal Daniel addressed these questions in his last paragraph. Our long term plans remain largely unchanged, botocore is still under active development, and will be the basis for boto3, the next version of boto.

The timeframe for the initial python3 support in boto will be in a few weeks. Because we are using six to maintain a single codebase that works in python2/python3, python3 support does not have to come all at once. We can incrementally merge in support to the develop branch for python3, and python2 users will be unaffected.

As for the timeframe for botocore, I don’t have any specific dates, but I can say that it is being actively worked on and we still plan on getting it production ready with complete documentation.

[danielgtaylor]

I've gone ahead and put up my work in progress here in case anybody wants to play with it over the weekend:

https://github.com/danielgtaylor/boto/tree/py2and3

Initial testing is going well. Integration tests for common code, S3, SQS, Kinesis and CloudTrail are mostly passing. I've also started a new porting guide which should help others to port modules once this branch is merged into develop.

Keep in mind that this is a work in progress. Consider it experimental until it has been merged. It still needs to be thoroughly tested and reviewed.

[dukedougal]

Out of interest - how many hands on developers does Amazon have working on Python boto botocore full time? Just Daniel and James?

[danielgtaylor]

I've submitted a pull request with my work in #2344. Please take a look and let me know any feedback you might have, and feel free to test the code in both Python 2 and Python 3.

[dukedougal]

Hi Daniel
Would you mind providing specific instructions please on how to test a version of boto that works with Python 3? I've tried a few different approaches but nothing seems to work. What's the right way?

thanks

[danielgtaylor]

Sure, you can try this all out in an isolated environment with the following, for example (assuming you have virtualenv, if not then pip install virtualenv):

virtualenv -p python3 py3boto

# Mac and Linux:
. py3boto/bin/activate

# Windows:
py3boto/bin/activate.bat

# All systems
git clone -b py2and3 https://github.com/danielgtaylor/boto.git
cd boto

# You can also install boto into the environment, so that you can
# use other packages that might depend on boto
python setup.py install

Now you can use Boto like normal inside of the virtualenv, e.g. run python:

>>> import boto
>>> s3 = boto.connect_s3()
>>> buckets = s3.get_all_buckets()
[<Bucket: bucket-1>, <Bucket: bucket-2>, ...]

This will become much easier once the branch is merged.

[danielgtaylor]

My branch has now been merged into develop. Please take a look and test it out, both to ensure Python 2 continues to work as expected and that Python 3 support now works for S3, SQS, Kinesis, and CloudTrail.

Setting up VirtualEnv (optional, but recommended):

virtualenv -p python3 py3boto

# Mac and Linux:
. py3boto/bin/activate

# Windows:
py3boto/bin/activate.bat

Installing the latest Boto

git clone https://github.com/boto/boto.git
cd boto
python setup.py install

Any bugs you find can be reported directly as Boto issues.