Python 3 support

[goertzenator]

I have Paramiko running on Python 3 at https://github.com/goertzenator/paramiko

All unit tests, except for sftp, pass. Needs documentation updates for the many str -> bytes changes.

Looking for advice on what to do with this. Adding a python 3 branch to paramiko would have a lot of complications...

[robey]

there's no way to get code that will run on both python 2 and 3, is there? :(

[goertzenator]

Paramiko is much too large and io heavy to make that realistic. You could get the two codebases (paramiko py2 and py3) very similar if you started with a py3 version and downported to 2.6 or 2.7. But the current paramiko works well on 2.6 and older, so I don't see a lot of value in that.

[ramon-garcia]

Hello,

You can have a single codebase for python 2 and python 3. In the setup.py add the conversion 2to3. This is the recommended way. Add the following code to the setup.py

try:
from distutils.command.build_py import build_py_2to3 as build_py
except ImportError:
# 2.x
from distutils.command.build_py import build_py
...
setup(...
cmdclass = {'build_py':build_py}
)

This is the recommended way: http://wiki.python.org/moin/PortingPythonToPy3k

[goertzenator]

Using 2to3 was the first step I took for the port and it did convert a lot of things, but it didn't do anything for the massive str/bytes separation that had to take place.

Now I don't know a lot about it, but 3to2 might have more potential for paramiko.

[ramon-garcia]

I am working with it now. Since paramiko development seems to be stopped (no reply from the author for months), I am using the ssh fork https://github.com/bitprophet/ssh

[goertzenator]

Cool, let me know how you make out or if you have any questions.

[sciunto]

Hi,

Is there any progress on that? Is https://github.com/bitprophet/ssh the best implementation in python3?

Thank you!

[bitprophet]

@sciunto That 'ssh' fork will merge back into this one shortly, for what it's worth. It's also not Python3 compat yet, though there are patches.

Speaking personally, Python 3 support is a medium priority -- my hope is for Fabric 2.x to be Python 2.6=>3.x compatible, and that will require doing the same for some Paramiko version.

For Fabric and my other projects, I'm hoping to go the "one codebase that works on both Python versions" approach. As mentioned above, I'm not sure if that is feasible for Paramiko due to it being bigger, more complex and more IO oriented. I also haven't experienced the joy/sadness of 2to3/3to2 yet -- if they work and don't make maintenance a huge pain, awesome. If not, hard to say what will happen.

[sciunto]

Alright. I'm going to limit my codes to some popen calls until things evolve.

Thanks a lot for your reply and your work

[bitprophet]

For the record, the python 'ssh' lib has now merged back into Paramiko (as of tonight).

Nothing else has changed yet re: investigating the least painful Py3k route for Paramiko itself -- but as I start to work on Fabric 2.x, this will become increasingly important to me, so look for updates in the nearish future.

[timdawborn]

Any updates or an ETA on a py3k version?

[hamilton-z]

any news about py3k version？

[bitprophet]

No py3k news yet, it is still definitely planned (see my above comment), but other priorities have to come first :(

[jlegewie]

I will just add my encouragement here. paramiko is great and I would really like to see it for Python 3.

[aspineux]

I wrote a python library pyzmail that work for both python 2 and 3.
I wrote it for 2.x then ran py2to3 on it. I made some changes to make it work for 2.x and be convertible by py2to3.
Then I have updated my setup.py to make it compatible with py2to3 ( py2to3 is run automatically when installing on 3.X, then no need to have two source tree )

The most difficult part is about unicode string and unicode literal. pyzmail use a lot of both. At 2 or 3 place I had no other choice than test check the python version and write specific code for both version.
The second one was to write doctest that work for both, because py2to3 don't convert comments.

paramiko don't make an heavy use of unicode string then it should be possible to use the same way.

[mvanderkolff]

Not quite: Particularly in the agent stuff, it should be using
bytestrings.

Cheers,

Michael

On Mon, Mar 4, 2013 at 1:26 PM, Alain Spineux notifications@github.comwrote:

I wrote a python library pyzmail that work for both python 2 and 3.
I wrote it for 2.x then ran py2to3 on it. I made some changes to make it
work for 2.x and be convertible by py2to3.
Then I have updated my setup.py to make it compatible with py2to3 ( py2to3
is run automatically when installing on 3.X, then no need to have two
source tree )

The most difficult part is about unicode string and unicode literal.
pyzmail use a lot of both. At 2 or 3 place I had no other choice than test
check the python version and write specific code for both version.
The second one was to write doctest that work for both, because py2to3
don't convert comments.

paramiko don't make an heavy use of unicode string then it should be
possible to use the same way.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/16#issuecomment-14361382
.

[aspineux]

By "using unicode", I mean having bytestring to convert to 'human' readable strings and vice versa.
98% of the string in paramiko are bytestrings; hostname, login and password are from the user and en then must be converted appropriately. SSH Keys must be loaded and saved the right way.
Thats all. This is not a complete rewrite.

[geertj]

I'd like to voice my support for this issue as well.

@bitprophet You should definitely go for the single code base approach. If you only need to support 2.6, 2.7 and 3.3+ this is very feasible.

[e0ne]

It looks like need to support two different code bases: for python 2.x and 3.7. I'm trying to port paramiko to 3.3 but it really hard to suport 2.x and 3.x in one codebase.

[aspineux]

What is your main problem ? Do you have some recurring piece of code that require some changes ? Does this make the code difficult to read ? Are you using 2to3 in POST-process (after you have made the original code for 2.x , 3.x ready )?

[e0ne]

Yes, I'm using 2to3. Main problem is new types: no long and unicode more. I'm pretty sure that that it will broke 2.x code or code will be difficult to read with a lot of try-except blocks

[bitprophet]

I've been actively porting my (much smaller & newer) projects Spec and Invoke to a shared 2.6+3.3 style codebase using six, an approach that not only everybody who's already done this has recommended, but which does seem to work quite well in practice. The transition's been thorny but only because I hadn't done any work with Python 3.x prior so I'm both porting + learning.

I expect any final Paramiko Python 3.x support will end up using six and similar approaches, even if an initial 2to3 pass is used to help highlight trouble spots.

[FirefighterBlu3]

@e0ne it's not that hard. instead of making crazy amounts of try/except blocks, make a function that has that in it. i maintain a daily use set of software that is approaching 10K lines and has to work on everything from 2.4 to 3.3

[FirefighterBlu3]

also, i find it much easier to write for py3 and then handle legacy breakage as exceptions to the norm. it makes dealing with issues a lot easier.

[e0ne]

@FirefighterBlu3 thanks for advise. I'll try it. I'm newbie in porting to python 3.x. I'm working on Lettuce and wont to get fabric/paramiko on python 3.3 in my project.
I don't like six because it makes python not so clear. Have a two branches for 2.x and 3.x is harder but code is better for me.

[FirefighterBlu3]

no problem. i'd love to see a py3 capable paramiko. i'd port it but i'm really really pressed for time. i prefer a single branch

[sybrenstuvel]

Please update the Fabric documentation; it now reads "Fabric is a Python (2.5 or higher) library", which implies 3.x support. Only after installing with pip and seeing syntax errors fly by, did I get suspicious about the 3.x support...

[bitprophet]

@sybrenstuvel Most folks realize Python 3 is a big break from Python 2...but clearly not everybody. Fixed: fabric/fabric@a722a7c - thanks!

I just completed #256 so it is now officially PYTHON 3 TIME and I will dive into the above PRs next time I have OSS time.

[asmeurer]

When you start actually using Python 3 you realize that it's not actually that big of a break.

[bitprophet]

Looks like #236 is the most recent attack on this so I will start by updating a copy of it, w/ latest master (main pain point is likely to be the docs changes from #256).

[bitprophet]

@asmeurer Sure, it's no Perl 6 ;) but it's enough of a barrier that I would never expect a project to default to supporting 3 if it just says "Python 2.5+". (Especially if it says 2.5+, given supporting 2.5 and 3 at the same time is even harder ;))

[bitprophet]

Mostly done applying #236's changes on top of latest master, taking notes as well. Plan is to present a new, clean PR with that changeset; discuss my notes on it (probably in that PR, just because it makes line comments easier); then iterate until we're satisfied.

[bitprophet]

#276 is that PR. Please put any interested eyes on that copy of the code, and I will be posting progress there as I hack on things.

[bitprophet]

We're basically done with #276, got test suite parity + did some cleanup + a little remaining that is being handled now.

What's left, IMO:

• Changelog entry
• Modify README/website/etc as required to note that Py3 support is now implemented
• Possibly wait for a few more invested parties (that would be: you) to report that yes, the post-merge master branch works well for them.
• Release Paramiko 1.13 (now that's an auspicious version number)
• Handle any bugs-in-the-wild as they come up, in their own tickets.

[bitprophet]

Was randomly poking https://caniusepython3.com/ and realized it's asserting ecdsa is not Py3 compatible. Should check that out; will be a hard blocker if it cannot be installed and/or imported (if it's only broken when actually used, then we can just deal for now as ECDSA isn't always required.)

[yegle]

https://github.com/warner/python-ecdsa
It says it support 3.2 and 3.3.

[tshepang]

The latest released version (0.10) has no trove classifiers indicating which Python versions are supported, but there is a commit claiming support for both 2.x and 3.x.

[bitprophet]

Well that's good to hear then, thanks @yegle @tshepang!

Back on OSS today (it's...been a long week), hopefully will have this out on PyPI by EOD if I'm lucky.

[bitprophet]

Merged into master!

Can't see a great reason to hold off on a 1.13 release; if it breaks, it breaks, and we bugfix.

[sybrenstuvel]

Nice work guys!

The reason I posted a comment about the "2.5+" thing, is that I'm trying to get "X and newer" to actually mean "X and newer", and not "X and newer except fo other new stuff" ;-)

[jaraco]

Smart move to release and move forward. Too many projects get stymied by fear of issues.

[travis-bear]

Congrats. And thank you all.

[tomoh1r]

👍 💯

[leafonsword]

Hope py3 supporting will come soon!

[tshepang]

@leafonsword there is, since 1.13.0.

[leafonsword]

@tshepang
haha,Thx~

[ghost]

Error from python 3.4.3:
from Crypto.Util import _counter
ImportError: Cannot load specified object

[Russell-Jones]

@TeeBSD could you
a) provide more context. e.g. Why is an error from Crypto to do with paramiko? Which OS and version of paramiko? How was it installed?
b) open a new ticket for this issue if there isn't one already (I couldn't find one)

[Lazik]

Paramiko uses pyCrypto.
It's more about how/which version of pyCrypto you installed.