New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to Python 3 #326

Closed
cortesi opened this Issue Aug 14, 2014 · 53 comments

Comments

Projects
None yet
@cortesi
Member

cortesi commented Aug 14, 2014

Creating this as a catch-all ticket to elicit some discussion of Python3 support, which has become something of an FAQ.

We're going to have to make the jump to Python3 at some point. All our dependencies (last time I checked) were Python 3-ready, so in theory we could make the leap. If we do it, we would do Python 3-only - I don't think there's a benefit to trying to maintain Python 2/3 Frankenstein that gives us the worst of both worlds.

This leaves us with two issues. First, there's a legacy concern - quite a few people use libmproxy to build things, and I know that I personally would probably have a few weeks of work ahead of me just in converting existing related projects over.

The second thing has to do with Python 3 and its treatment of Unicode/strings. Mitmproxy is "special", in the sense that it deals with loosely typed, untrusted and possibly intentionally corrupted data flows. It's a boundary layer between a weird world where string encodings can't be assumed, and the higher levels like the UI and so forth, where Python 3 forces us to have unicode. In this sense, it's similar to other programs that deal with on-the-wire data, like web frameworks (though our problem is even worse). Have a look at Armin Ronacher's experience with Python 3 Unicode and Flask/Jinja2/etc.:

http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/

I know that this has frustrated some people working on similar programs enough that they've moved away from Python altogether....

Where does this leave us? Well, clearly we clearly can't stay on Python2 in the longer term. The Python3 situation is pretty dire, but has very gradually been getting very slightly better. 3.4 included some improvements, and I think 3.5 will have a few more. The key issues are intrinsic to the language design, though, and will never go away.

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Aug 17, 2014

Member

This leaves us with two issues. First, there's a legacy concern - quite a few people use libmproxy to build things, and I know that I personally would probably have a few weeks of work ahead of me just in converting existing related projects over.

I can't add much to this point - a few weeks sounds fairly bad though.

The second thing has to do with Python 3 and its treatment of Unicode/strings. Mitmproxy is "special", in the sense that it deals with loosely typed, untrusted and possibly intentionally corrupted data flows. It's a boundary layer between a weird world where string encodings can't be assumed, and the higher levels like the UI and so forth, where Python 3 forces us to have unicode. In this sense, it's similar to other programs that deal with on-the-wire data, like web frameworks (though our problem is even worse). Have a look at Armin Ronacher's experience with Python 3 Unicode and Flask/Jinja2/etc.:

As much as I respect Armin Ronacher for Flask, jinja2 and werkzeug, I feel that his rant on Python 3 is one-sided. Yes, Python 3 is very idealistic when it comes to unicode, but I don't see any major flaws in the language. While there are issues when it comes to console interaction etc, we have to admit that Python 2 is not much better here (/mitmproxy#L62). Maybe I'm a bit naïve, but the proposal to add .decode to the unicode type actually sounds deeply wrong. To be fair, he wrote his reviews before the improvements in 3.4 and 3.5 were out.

To see how bad it actually is, it ported netlib.socks and netlib.certutils over to Python 3 - turns out it's actually not that bad. http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html turns out to be an excellent guide pointing to the right resources.

netlib.socks (mitmproxy/netlib@c357f9f):

  • Working with raw bytes is actually a little bit nicer in Python 3
  • I discovered that the socks module didn't handle unicode domain names properly before. Being explicit about the encoding forces this, which is probably not a bad thing.
  • Next to the unicode stuff, Python 3 actually comes with some nice stuff (ipaddress module, ...)

netlib.certutils (mitmproxy/netlib@9157581):

  • pyOpenSSL (or any other C library) obviously needs to be interfaced with bytes, but that's mostly a matter of running the testsuite and appending b whereever neccessary.
  • ported in about 20 minutes, where most of the time has been spent on researching file system encodings - something that shouldn't really affect us that much.

TL;DR: From my little experience I feel that mitmproxy would actually benefit from moving to Python 3 (code-wise). Nevertheless, this is not pressing and I'm happy to continue on 2.7 for a while. If we decide to port, I'm happy to help.

Cheers,
Max

Member

mhils commented Aug 17, 2014

This leaves us with two issues. First, there's a legacy concern - quite a few people use libmproxy to build things, and I know that I personally would probably have a few weeks of work ahead of me just in converting existing related projects over.

I can't add much to this point - a few weeks sounds fairly bad though.

The second thing has to do with Python 3 and its treatment of Unicode/strings. Mitmproxy is "special", in the sense that it deals with loosely typed, untrusted and possibly intentionally corrupted data flows. It's a boundary layer between a weird world where string encodings can't be assumed, and the higher levels like the UI and so forth, where Python 3 forces us to have unicode. In this sense, it's similar to other programs that deal with on-the-wire data, like web frameworks (though our problem is even worse). Have a look at Armin Ronacher's experience with Python 3 Unicode and Flask/Jinja2/etc.:

As much as I respect Armin Ronacher for Flask, jinja2 and werkzeug, I feel that his rant on Python 3 is one-sided. Yes, Python 3 is very idealistic when it comes to unicode, but I don't see any major flaws in the language. While there are issues when it comes to console interaction etc, we have to admit that Python 2 is not much better here (/mitmproxy#L62). Maybe I'm a bit naïve, but the proposal to add .decode to the unicode type actually sounds deeply wrong. To be fair, he wrote his reviews before the improvements in 3.4 and 3.5 were out.

To see how bad it actually is, it ported netlib.socks and netlib.certutils over to Python 3 - turns out it's actually not that bad. http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html turns out to be an excellent guide pointing to the right resources.

netlib.socks (mitmproxy/netlib@c357f9f):

  • Working with raw bytes is actually a little bit nicer in Python 3
  • I discovered that the socks module didn't handle unicode domain names properly before. Being explicit about the encoding forces this, which is probably not a bad thing.
  • Next to the unicode stuff, Python 3 actually comes with some nice stuff (ipaddress module, ...)

netlib.certutils (mitmproxy/netlib@9157581):

  • pyOpenSSL (or any other C library) obviously needs to be interfaced with bytes, but that's mostly a matter of running the testsuite and appending b whereever neccessary.
  • ported in about 20 minutes, where most of the time has been spent on researching file system encodings - something that shouldn't really affect us that much.

TL;DR: From my little experience I feel that mitmproxy would actually benefit from moving to Python 3 (code-wise). Nevertheless, this is not pressing and I'm happy to continue on 2.7 for a while. If we decide to port, I'm happy to help.

Cheers,
Max

@nanonyme

This comment has been minimized.

Show comment
Hide comment
@nanonyme

nanonyme Aug 29, 2014

Contributor

How about trying to support both? It may not be an unsurmountable task especially with good test coverage. Python2 will likely be here for a long time still.

Contributor

nanonyme commented Aug 29, 2014

How about trying to support both? It may not be an unsurmountable task especially with good test coverage. Python2 will likely be here for a long time still.

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Aug 29, 2014

Member

While it's certainly possible to support both, I don't see how this would move the project forward. Why should we take the additional burden of supporting Python 2.x if we migrate our codebase? The only convincing argument for me would be a large external codebase that relies on mitmproxy/libmproxy.

Member

mhils commented Aug 29, 2014

While it's certainly possible to support both, I don't see how this would move the project forward. Why should we take the additional burden of supporting Python 2.x if we migrate our codebase? The only convincing argument for me would be a large external codebase that relies on mitmproxy/libmproxy.

@cortesi

This comment has been minimized.

Show comment
Hide comment
@cortesi

cortesi Oct 24, 2014

Member

The time has come to move to Python3, possibly for 0.12. I'm adding this ticket to the milestone.

I will say that I don't expect to have any issues porting things to Python3 below the UI layer. Netlib, pathod, and libmproxy should all be reasonably easy. Where I expect to have issues is where we're forced to display data that may not have a fixed encoding - for instance, mitmproxy console.

Be that as it may, the scales have, I feel, tipped in favor of Python3 in the last year or so.

Member

cortesi commented Oct 24, 2014

The time has come to move to Python3, possibly for 0.12. I'm adding this ticket to the milestone.

I will say that I don't expect to have any issues porting things to Python3 below the UI layer. Netlib, pathod, and libmproxy should all be reasonably easy. Where I expect to have issues is where we're forced to display data that may not have a fixed encoding - for instance, mitmproxy console.

Be that as it may, the scales have, I feel, tipped in favor of Python3 in the last year or so.

@cortesi cortesi added this to the 0.12 milestone Oct 24, 2014

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Oct 25, 2014

Member

👍

Member

mhils commented Oct 25, 2014

👍

@bradleypeabody

This comment has been minimized.

Show comment
Hide comment
@bradleypeabody

bradleypeabody Feb 23, 2015

Contributor

fwiw - I'm also a proponent of moving to Python 3 and leaving 2 in the dust. I did a couple of mitmproxy installs a few months ago on CentOS 6.x and had the interesting problem of them only having Python 2.6 available by default. Manually upgrading to 2.7 was a pain and it would have been just as easy to move directly to 3 (probably easier actually). Legacy support will always be an important issue, but moving forward is just as important (e.g. http://fedoraproject.org/wiki/Changes/Python_3_as_Default https://wiki.ubuntu.com/Python/3 )

P.S. Docker's popularity is also incidentally helping solve this problem - at least for Linux servers. Supporting a newer Python version for an individual application is a piece of cake using containers/Docker.

Contributor

bradleypeabody commented Feb 23, 2015

fwiw - I'm also a proponent of moving to Python 3 and leaving 2 in the dust. I did a couple of mitmproxy installs a few months ago on CentOS 6.x and had the interesting problem of them only having Python 2.6 available by default. Manually upgrading to 2.7 was a pain and it would have been just as easy to move directly to 3 (probably easier actually). Legacy support will always be an important issue, but moving forward is just as important (e.g. http://fedoraproject.org/wiki/Changes/Python_3_as_Default https://wiki.ubuntu.com/Python/3 )

P.S. Docker's popularity is also incidentally helping solve this problem - at least for Linux servers. Supporting a newer Python version for an individual application is a piece of cake using containers/Docker.

@cortesi

This comment has been minimized.

Show comment
Hide comment
@cortesi

cortesi Mar 19, 2015

Member

A note for readers of this ticket - we're now waiting on PyInstaller to have a mature Python3 branch before we move. It's either that, or begin to roll our own distribution packages for major platforms, which would be a major timesink.

Member

cortesi commented Mar 19, 2015

A note for readers of this ticket - we're now waiting on PyInstaller to have a mature Python3 branch before we move. It's either that, or begin to roll our own distribution packages for major platforms, which would be a major timesink.

@bazabaza

This comment has been minimized.

Show comment
Hide comment
@bazabaza

bazabaza Mar 20, 2015

@cortesi thanks for info. Please keep us up :)

bazabaza commented Mar 20, 2015

@cortesi thanks for info. Please keep us up :)

@elitest

This comment has been minimized.

Show comment
Hide comment
@elitest

elitest Apr 7, 2015

Contributor

I looked into python 2->3 porting over the weekend, and also watched some of the pycon 2014 videos on the topic.

If we were just to move 2.x in the dust, how would it get done? Code freeze while it gets ported? Just use python’s 2to3 and fix stuff that breaks? Or fork and then backport?

I think I’d like to try to make the case for python 2 & 3 compatible code in the same codebase. The feeling I got was that so much work has gone into python 2.6 and 2.7 around backporting Python 3 stuff that it is often easier to clean up your code than to do a hard break and that python 3 seem to be at a point where there are decent tools that help you write agnostic code. I had no idea for example that if you do a “python -3 mitmproxy” under recent-ish versions of 2.x, it gives you debug output, about code that it comes across that needs to be addressed for python 3.x.

The other things that seemed to be major roadblocks to porting, were things like CPython extentions, which we don’t do, and not having good testing which we I think we have. Can Travis help us out in this manner? I agree that the strings handling vs byte/unicode handling, is probably the biggest problem, but that isn’t going to be solved by dropping 2.x support is it?

references:
http://python3porting.com/
https://www.youtube.com/watch?v=nx0sXSeJbmI
https://www.youtube.com/watch?v=f_6vDi7ywuA

Contributor

elitest commented Apr 7, 2015

I looked into python 2->3 porting over the weekend, and also watched some of the pycon 2014 videos on the topic.

If we were just to move 2.x in the dust, how would it get done? Code freeze while it gets ported? Just use python’s 2to3 and fix stuff that breaks? Or fork and then backport?

I think I’d like to try to make the case for python 2 & 3 compatible code in the same codebase. The feeling I got was that so much work has gone into python 2.6 and 2.7 around backporting Python 3 stuff that it is often easier to clean up your code than to do a hard break and that python 3 seem to be at a point where there are decent tools that help you write agnostic code. I had no idea for example that if you do a “python -3 mitmproxy” under recent-ish versions of 2.x, it gives you debug output, about code that it comes across that needs to be addressed for python 3.x.

The other things that seemed to be major roadblocks to porting, were things like CPython extentions, which we don’t do, and not having good testing which we I think we have. Can Travis help us out in this manner? I agree that the strings handling vs byte/unicode handling, is probably the biggest problem, but that isn’t going to be solved by dropping 2.x support is it?

references:
http://python3porting.com/
https://www.youtube.com/watch?v=nx0sXSeJbmI
https://www.youtube.com/watch?v=f_6vDi7ywuA

@cortesi

This comment has been minimized.

Show comment
Hide comment
@cortesi

cortesi Apr 7, 2015

Member

So, there would be a preparatory stage where we get ready for a Python 3 migration by getting rid of Python 2-isms, migrating to backported features, and bring our unit test suite back up to 100%, and so forth. Then there would be a flagfall with a code freeze, and after a few days of effort we'd emerge as a Python 3 only project. I don't think maintaining a 2&3 compatible codebase is worth the effort, unless you're a library has a user constituency in both camps. We're in the relatively luxurious position where we can actually make a complete shift without annoying too many people, and in fact, without most of our users noticing.

I did a survey recently, and ALL our dependencies seem to be Python 3-ready. The one exception is PyInstaller, a perennial bugbear. Its Python3 branch (https://github.com/pyinstaller/pyinstaller/tree/python3) doesn't seem mature, and is getting less development attention than the main branch. I'm watching to see how that shakes out.

Member

cortesi commented Apr 7, 2015

So, there would be a preparatory stage where we get ready for a Python 3 migration by getting rid of Python 2-isms, migrating to backported features, and bring our unit test suite back up to 100%, and so forth. Then there would be a flagfall with a code freeze, and after a few days of effort we'd emerge as a Python 3 only project. I don't think maintaining a 2&3 compatible codebase is worth the effort, unless you're a library has a user constituency in both camps. We're in the relatively luxurious position where we can actually make a complete shift without annoying too many people, and in fact, without most of our users noticing.

I did a survey recently, and ALL our dependencies seem to be Python 3-ready. The one exception is PyInstaller, a perennial bugbear. Its Python3 branch (https://github.com/pyinstaller/pyinstaller/tree/python3) doesn't seem mature, and is getting less development attention than the main branch. I'm watching to see how that shakes out.

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Apr 8, 2015

Member

I'd propose something along these lines:

  1. Make netlib/pathod cross-compatible with Python 2 and 3, mark all workarounds for Python 2. The expected outcome is that the netlib/pathod tests run fine for Python 2 and 3 while mitmproxy is still Python 2 only. This could be done iteratively without freezing anything - the required workarounds shouldn't be too severe.
  2. Move mitmproxy to Python 3 - we should see whether we can retain Python 2 compatibility for the first adjustments, but we should not be resistant to do a code freeze here.
  3. Ditch Python 2 for everything, remove workarounds.

If you want to help, I'd be happy to see some initial work on netlib/pathod that makes it cross-compatible and could be immediately merged into master. My start with the python3 branch was nice to explore the situation, but keeping Pyhton 2 compatibility here for a bit might be useful to make the transition smooth.

Member

mhils commented Apr 8, 2015

I'd propose something along these lines:

  1. Make netlib/pathod cross-compatible with Python 2 and 3, mark all workarounds for Python 2. The expected outcome is that the netlib/pathod tests run fine for Python 2 and 3 while mitmproxy is still Python 2 only. This could be done iteratively without freezing anything - the required workarounds shouldn't be too severe.
  2. Move mitmproxy to Python 3 - we should see whether we can retain Python 2 compatibility for the first adjustments, but we should not be resistant to do a code freeze here.
  3. Ditch Python 2 for everything, remove workarounds.

If you want to help, I'd be happy to see some initial work on netlib/pathod that makes it cross-compatible and could be immediately merged into master. My start with the python3 branch was nice to explore the situation, but keeping Pyhton 2 compatibility here for a bit might be useful to make the transition smooth.

@cortesi

This comment has been minimized.

Show comment
Hide comment
@cortesi

cortesi Apr 8, 2015

Member

That might be worth the effort as an intermediate step. We should definitely wait until we have a plausible path forward with PyInstaller before we move on this - a large fraction of our users rely on this at the moment.

Member

cortesi commented Apr 8, 2015

That might be worth the effort as an intermediate step. We should definitely wait until we have a plausible path forward with PyInstaller before we move on this - a large fraction of our users rely on this at the moment.

@JPS46225

This comment has been minimized.

Show comment
Hide comment
@JPS46225

JPS46225 Jun 6, 2015

Hi Max (mhils)

I was just reading your comment about mitmproxy only being supported up to Python 2.7. I was wondering whether this was still the case, or if it had been updated to support the latest Python release?

JPS46225 commented Jun 6, 2015

Hi Max (mhils)

I was just reading your comment about mitmproxy only being supported up to Python 2.7. I was wondering whether this was still the case, or if it had been updated to support the latest Python release?

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Jun 6, 2015

Member

No news yet - I'd be happy to see someone start with netlib. 😉

Member

mhils commented Jun 6, 2015

No news yet - I'd be happy to see someone start with netlib. 😉

@mhils mhils added the enhancement label Jun 26, 2015

@JPS46225

This comment has been minimized.

Show comment
Hide comment
@JPS46225

JPS46225 Jul 16, 2015

Hi Guys

I was just wondering if there has been any recent progress in regards to the move to Python 3?

JPS46225 commented Jul 16, 2015

Hi Guys

I was just wondering if there has been any recent progress in regards to the move to Python 3?

@anatasiajp

This comment has been minimized.

Show comment
Hide comment
@anatasiajp

anatasiajp Jul 23, 2015

I think I will try to make mitmproxy work on both Python 2 and 3, well but I need time because mitmproxy is too big, and Twisted have no Python 3 which is the hardest math. I also want to suggest you use as less as possible number of thirdparty library, that make mitmproxy too depentdent, for example we can risk Twisted with Python native.

anatasiajp commented Jul 23, 2015

I think I will try to make mitmproxy work on both Python 2 and 3, well but I need time because mitmproxy is too big, and Twisted have no Python 3 which is the hardest math. I also want to suggest you use as less as possible number of thirdparty library, that make mitmproxy too depentdent, for example we can risk Twisted with Python native.

@JPS46225

This comment has been minimized.

Show comment
Hide comment
@JPS46225

JPS46225 Jul 23, 2015

Hi

Just wondering what you mean by 'twisted with Python native' and also 'less as possible number of third party library', I've only installed what's necessary to get Mitmproxy functioning. Also, is there a way to assign a proxy to a device that doesn't have internal settings to configure proxies, perhaps, through a gateway/router?

JPS46225 commented Jul 23, 2015

Hi

Just wondering what you mean by 'twisted with Python native' and also 'less as possible number of third party library', I've only installed what's necessary to get Mitmproxy functioning. Also, is there a way to assign a proxy to a device that doesn't have internal settings to configure proxies, perhaps, through a gateway/router?

@JPS46225

This comment has been minimized.

Show comment
Hide comment
@JPS46225

JPS46225 Jul 23, 2015

Also, if there is anything else that I can do to improve overall experience, please let me know

JPS46225 commented Jul 23, 2015

Also, if there is anything else that I can do to improve overall experience, please let me know

@JPS46225

This comment has been minimized.

Show comment
Hide comment
@JPS46225

JPS46225 Jul 23, 2015

Oh, yes, I see what you mean now by twisted, also I'd love to know how I can learn to do stuff like proxy configuration and other Python modules, if you know of any resources, please share them?

JPS46225 commented Jul 23, 2015

Oh, yes, I see what you mean now by twisted, also I'd love to know how I can learn to do stuff like proxy configuration and other Python modules, if you know of any resources, please share them?

@mhils mhils removed this from the 0.12 milestone Jul 24, 2015

@anatasiajp

This comment has been minimized.

Show comment
Hide comment
@anatasiajp

anatasiajp Jul 25, 2015

Hi @JPS46225, to answer your question:

Just wondering what you mean by 'twisted with Python native' and also 'less as possible number of third party library', I've only installed what's necessary to get Mitmproxy functioning.

Well, we can use socketserver instead Twisted, they are almost the same, but Twisted support something that socketserver don't have, but we can modify socketserver to have those.

Also, is there a way to assign a proxy to a device that doesn't have internal settings to configure proxies, perhaps, through a gateway/router?

Like above, socketserver (lower level), HTTPServer or TCPServer (higher).

With socketserver/HTTPServer/TCPServer we can create a proxy server that listen at whatever port we want.

I love socks5 mode for mitmproxy, it can filter/change HTTPS sites, that is a rare feature that almost no socks5 proxy do.

anatasiajp commented Jul 25, 2015

Hi @JPS46225, to answer your question:

Just wondering what you mean by 'twisted with Python native' and also 'less as possible number of third party library', I've only installed what's necessary to get Mitmproxy functioning.

Well, we can use socketserver instead Twisted, they are almost the same, but Twisted support something that socketserver don't have, but we can modify socketserver to have those.

Also, is there a way to assign a proxy to a device that doesn't have internal settings to configure proxies, perhaps, through a gateway/router?

Like above, socketserver (lower level), HTTPServer or TCPServer (higher).

With socketserver/HTTPServer/TCPServer we can create a proxy server that listen at whatever port we want.

I love socks5 mode for mitmproxy, it can filter/change HTTPS sites, that is a rare feature that almost no socks5 proxy do.

@JPS46225

This comment has been minimized.

Show comment
Hide comment
@JPS46225

JPS46225 Jul 25, 2015

Hi cattleyavns

You're going to have to assume that I know absolutely nothing about this, so every little detail helps...

I have heard of socket server, but don't get it, would you mind elaborating more on the subject?

JPS46225 commented Jul 25, 2015

Hi cattleyavns

You're going to have to assume that I know absolutely nothing about this, so every little detail helps...

I have heard of socket server, but don't get it, would you mind elaborating more on the subject?

@JPS46225

This comment has been minimized.

Show comment
Hide comment
@JPS46225

JPS46225 Jul 25, 2015

Any tutorials related to the subject, would also help immensely...

JPS46225 commented Jul 25, 2015

Any tutorials related to the subject, would also help immensely...

@JPS46225

This comment has been minimized.

Show comment
Hide comment
@JPS46225

JPS46225 Jul 25, 2015

Ok, so, basically what I've known about Mitmproxy is so far, is that you have to tell client device to use the server that's running Mitmproxy, which would be the computer and then tell it the port to use. What I mean by internal settings, is that unlike my iPad, where I can configure a proxy for it to use, the other device which I'm trying this on, doesn't have the option to configure a proxy

JPS46225 commented Jul 25, 2015

Ok, so, basically what I've known about Mitmproxy is so far, is that you have to tell client device to use the server that's running Mitmproxy, which would be the computer and then tell it the port to use. What I mean by internal settings, is that unlike my iPad, where I can configure a proxy for it to use, the other device which I'm trying this on, doesn't have the option to configure a proxy

@PidgeyL

This comment has been minimized.

Show comment
Hide comment
@PidgeyL

PidgeyL Aug 4, 2015

Is there any estimate (date-wise) of when MitMProxy will be supported in Python 3.x?

PidgeyL commented Aug 4, 2015

Is there any estimate (date-wise) of when MitMProxy will be supported in Python 3.x?

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Aug 4, 2015

Member

No - the ROI for porting is still fairly low for us and we currently focus on other features. I'm happy to help anyone who wants to contribute this feature though.

Member

mhils commented Aug 4, 2015

No - the ROI for porting is still fairly low for us and we currently focus on other features. I'm happy to help anyone who wants to contribute this feature though.

@riX2000

This comment has been minimized.

Show comment
Hide comment
@riX2000

riX2000 Aug 4, 2015

ROI for me was higher than trying to get the python 2 version working. Spent 2 days on that and only like 2 hours porting it to python 3...

riX2000 commented Aug 4, 2015

ROI for me was higher than trying to get the python 2 version working. Spent 2 days on that and only like 2 hours porting it to python 3...

@PidgeyL

This comment has been minimized.

Show comment
Hide comment
@PidgeyL

PidgeyL Aug 4, 2015

I also haven't gotten the 2.x version working yet. Do you have the ported version somewhere available?

PidgeyL commented Aug 4, 2015

I also haven't gotten the 2.x version working yet. Do you have the ported version somewhere available?

@riX2000

This comment has been minimized.

Show comment
Hide comment
@riX2000

riX2000 Aug 4, 2015

Sorry, I just edited the installed package, and I'm sure there is several errors and bugs left.
I just ran it with python3 and fixed any errors using 2to3 and some manual fixing until it worked.

The best would be if someone could make a tutorial on how to get it working on a system using python3 as default (Arch in my case). I tried several methods, including AUR, git, pip2 etc.. none worked.

riX2000 commented Aug 4, 2015

Sorry, I just edited the installed package, and I'm sure there is several errors and bugs left.
I just ran it with python3 and fixed any errors using 2to3 and some manual fixing until it worked.

The best would be if someone could make a tutorial on how to get it working on a system using python3 as default (Arch in my case). I tried several methods, including AUR, git, pip2 etc.. none worked.

@PidgeyL

This comment has been minimized.

Show comment
Hide comment
@PidgeyL

PidgeyL Aug 4, 2015

So I assume you just downloaded the archive, and imported from its subfolders?

PidgeyL commented Aug 4, 2015

So I assume you just downloaded the archive, and imported from its subfolders?

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Aug 4, 2015

Member

Porting mitmproxy to Python 3 unfortunately takes way more time than two
hours. We're really happy for (partial) contributions here, this project is
completely volunteer-driven and this is a good chance to engage yourself.
:-)

On Tue, Aug 4, 2015 at 3:43 PM Pieter-Jan notifications@github.com wrote:

I also haven't gotten the 2.x version working yet. Do you have the ported
version somewhere available?


Reply to this email directly or view it on GitHub
#326 (comment)
.

Member

mhils commented Aug 4, 2015

Porting mitmproxy to Python 3 unfortunately takes way more time than two
hours. We're really happy for (partial) contributions here, this project is
completely volunteer-driven and this is a good chance to engage yourself.
:-)

On Tue, Aug 4, 2015 at 3:43 PM Pieter-Jan notifications@github.com wrote:

I also haven't gotten the 2.x version working yet. Do you have the ported
version somewhere available?


Reply to this email directly or view it on GitHub
#326 (comment)
.

@riX2000

This comment has been minimized.

Show comment
Hide comment
@riX2000

riX2000 Aug 4, 2015

@PidgeyL No, I edited the files installed with pip in /usr/lib/site-packages/. Not the recommeneded way of course, just clone the git and go from there, my way was a dirty hack.

@mhils I'm sure, but to just get it working to capture data it took two hours :)

riX2000 commented Aug 4, 2015

@PidgeyL No, I edited the files installed with pip in /usr/lib/site-packages/. Not the recommeneded way of course, just clone the git and go from there, my way was a dirty hack.

@mhils I'm sure, but to just get it working to capture data it took two hours :)

@PidgeyL

This comment has been minimized.

Show comment
Hide comment
@PidgeyL

PidgeyL Aug 4, 2015

@mhils: Would it take long to update just the libmproxy? (which I assume is what @riX2000 did?)

PidgeyL commented Aug 4, 2015

@mhils: Would it take long to update just the libmproxy? (which I assume is what @riX2000 did?)

@riX2000

This comment has been minimized.

Show comment
Hide comment
@riX2000

riX2000 Aug 4, 2015

yes, libmproxy (and netlib..)

riX2000 commented Aug 4, 2015

yes, libmproxy (and netlib..)

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Aug 4, 2015

Member

This is still the way to go: #326 (comment)

Member

mhils commented Aug 4, 2015

This is still the way to go: #326 (comment)

@riX2000

This comment has been minimized.

Show comment
Hide comment
@riX2000

riX2000 Aug 4, 2015

What about the python3 branch on mitmproxy/netlib? Looks to be about the same fixes I did, is it working?

riX2000 commented Aug 4, 2015

What about the python3 branch on mitmproxy/netlib? Looks to be about the same fixes I did, is it working?

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Aug 4, 2015

Member

This was a quick attempt to see how far we get with little effort. Directly
porting everything to py3 isn't viable though - py2/3 cross compatibility
is the first goal for netlib.

riX2000 notifications@github.com schrieb am Di., 4. Aug. 2015 16:18:

What about the python3 branch on mitmproxy/netlib? Looks to be about the
same fixes I did, is it working?


Reply to this email directly or view it on GitHub
#326 (comment)
.

Member

mhils commented Aug 4, 2015

This was a quick attempt to see how far we get with little effort. Directly
porting everything to py3 isn't viable though - py2/3 cross compatibility
is the first goal for netlib.

riX2000 notifications@github.com schrieb am Di., 4. Aug. 2015 16:18:

What about the python3 branch on mitmproxy/netlib? Looks to be about the
same fixes I did, is it working?


Reply to this email directly or view it on GitHub
#326 (comment)
.

@riX2000

This comment has been minimized.

Show comment
Hide comment
@riX2000

riX2000 Aug 4, 2015

Hmm I was going to start porting netlib, but then I noticed one of the reasons the python2 version did not work.. the mitmproxy/dev assumes the python 2 executable is 'python' and equivalently 'pip', which is usually not the case if python 3 is installed. Editing this file to use python2 and pip2 did the trick..
@mhils Maybe add a note about this in the readme or even better a python and pip version check in the dev file.

riX2000 commented Aug 4, 2015

Hmm I was going to start porting netlib, but then I noticed one of the reasons the python2 version did not work.. the mitmproxy/dev assumes the python 2 executable is 'python' and equivalently 'pip', which is usually not the case if python 3 is installed. Editing this file to use python2 and pip2 did the trick..
@mhils Maybe add a note about this in the readme or even better a python and pip version check in the dev file.

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Oct 12, 2015

Contributor

The only convincing argument for me would be a large external codebase that relies on mitmproxy/libmproxy.

Scrapy tests use libmproxy; it'd be nice to have Python 2/3 compatible codebase, not just Python 3 rewrite.

the mitmproxy/dev assumes the python 2 executable is 'python' and equivalently 'pip', which is usually not the case if python 3 is installed.

According to https://www.python.org/dev/peps/pep-0394/ it shouldn't be the case. python should be Python 3 only in a Python 3 virtualenv or in distributions which go against this PEP (like Arch Linux).

Contributor

kmike commented Oct 12, 2015

The only convincing argument for me would be a large external codebase that relies on mitmproxy/libmproxy.

Scrapy tests use libmproxy; it'd be nice to have Python 2/3 compatible codebase, not just Python 3 rewrite.

the mitmproxy/dev assumes the python 2 executable is 'python' and equivalently 'pip', which is usually not the case if python 3 is installed.

According to https://www.python.org/dev/peps/pep-0394/ it shouldn't be the case. python should be Python 3 only in a Python 3 virtualenv or in distributions which go against this PEP (like Arch Linux).

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Oct 12, 2015

Member

Thanks for the feedback, @kmike. Is it just https://github.com/scrapy/scrapy/blob/master/tests/test_proxy_connect.py or did I miss some parts?

FWIW, I ported netlib a few weeks ago to be cross-comptabile with 2.7 and 3.5. We're not sure what to do with mitmproxy, but we'll take that into consideration.

Member

mhils commented Oct 12, 2015

Thanks for the feedback, @kmike. Is it just https://github.com/scrapy/scrapy/blob/master/tests/test_proxy_connect.py or did I miss some parts?

FWIW, I ported netlib a few weeks ago to be cross-comptabile with 2.7 and 3.5. We're not sure what to do with mitmproxy, but we'll take that into consideration.

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Oct 12, 2015

Contributor

@mhils great to hear netib is ported, thanks!

Thanks for the feedback, @kmike. Is it just https://github.com/scrapy/scrapy/blob/master/tests/test_proxy_connect.py or did I miss some parts?

Yes, it is just that. Currently for HTTP proxy tests we're using Twisted.

Contributor

kmike commented Oct 12, 2015

@mhils great to hear netib is ported, thanks!

Thanks for the feedback, @kmike. Is it just https://github.com/scrapy/scrapy/blob/master/tests/test_proxy_connect.py or did I miss some parts?

Yes, it is just that. Currently for HTTP proxy tests we're using Twisted.

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Oct 12, 2015

Contributor

Do you target 3.5 only, not 3.4? Python 3.4 is the default Python 3 in latest Ubuntu LTS, so it'd be nice to support it as well. Are there new features you're using?

Contributor

kmike commented Oct 12, 2015

Do you target 3.5 only, not 3.4? Python 3.4 is the default Python 3 in latest Ubuntu LTS, so it'd be nice to support it as well. Are there new features you're using?

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Oct 12, 2015

Member

We're using a few 3.5 features, but nothing that couldn't be fixed easily. I'm personally fine with supporting 3.5 only (which should be the default for Ubuntu 15.10 and then 16.04), but I'd be happy to accept patches that add 3.4 compatibility. 😉

Member

mhils commented Oct 12, 2015

We're using a few 3.5 features, but nothing that couldn't be fixed easily. I'm personally fine with supporting 3.5 only (which should be the default for Ubuntu 15.10 and then 16.04), but I'd be happy to accept patches that add 3.4 compatibility. 😉

@kmike

This comment has been minimized.

Show comment
Hide comment
@kmike

kmike Oct 12, 2015

Contributor

Ok, sounds good :)

Contributor

kmike commented Oct 12, 2015

Ok, sounds good :)

@scone

This comment has been minimized.

Show comment
Hide comment
@scone

scone Nov 11, 2015

Has anyone begun the porting process? I'm game for working on the port, but don't want to duplicate efforts.

scone commented Nov 11, 2015

Has anyone begun the porting process? I'm game for working on the port, but don't want to duplicate efforts.

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Nov 11, 2015

Member

I don't think so. If you go for it, please make it as incremental as you
can (e.g. make the tests for one class or one *.py file pass) and send
partial pull requests as early as possible. :-)

scone notifications@github.com schrieb am Mi., 11. Nov. 2015 08:55:

Has anyone begun the porting process? I'm game for working on the port,
but don't want to duplicate efforts.


Reply to this email directly or view it on GitHub
#326 (comment)
.

Member

mhils commented Nov 11, 2015

I don't think so. If you go for it, please make it as incremental as you
can (e.g. make the tests for one class or one *.py file pass) and send
partial pull requests as early as possible. :-)

scone notifications@github.com schrieb am Mi., 11. Nov. 2015 08:55:

Has anyone begun the porting process? I'm game for working on the port,
but don't want to duplicate efforts.


Reply to this email directly or view it on GitHub
#326 (comment)
.

@ianozsvald

This comment has been minimized.

Show comment
Hide comment
@ianozsvald

ianozsvald Feb 28, 2016

+1 for Python 3.5 support, I just tried to install it and was surprised to find mitmdump fail with import thread. Given we have <4 years to the 2020 sunset date for Python 2.7, I'm surprised to find tools that aren't Python 3.4+ ready (I understand the issues, I talk about this on stage at my monthly PyDataLondon [2,800+ members] meetup to get folk to make the jump). For now I'll downgrade to an old 2.7 install.

ianozsvald commented Feb 28, 2016

+1 for Python 3.5 support, I just tried to install it and was surprised to find mitmdump fail with import thread. Given we have <4 years to the 2020 sunset date for Python 2.7, I'm surprised to find tools that aren't Python 3.4+ ready (I understand the issues, I talk about this on stage at my monthly PyDataLondon [2,800+ members] meetup to get folk to make the jump). For now I'll downgrade to an old 2.7 install.

@cortesi

This comment has been minimized.

Show comment
Hide comment
@cortesi

cortesi Feb 28, 2016

Member

@ianozsvald Mitmproxy is a good example of a difficult Python 2-3 transition. If you're interested in the issue - and especially if you feel qualified to speak publicly about it - it would be a good idea to look closely and sympathetically at why we're not there yet.

  • Mitmproxy has lots of dependencies, some of which only recently became Python 3 ready themselves.
  • We're not interested in inelegantly straddling Python 2 and 3 - we'll be moving across to Python3 wholesale. A lot of projects integrate mitmproxy as a library, which means that a move to Python3 will force integrators to either stay with old versions of mitmproxy or to move to Python3 themselves.
  • Mitmproxy works at a slightly unusual boundary where Python3's unicode and byte handling becomes a pain point. Armin Ronacher has written eloquently about this (though his post is now somewhat outdated): http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/.
  • Mitmproxy is distributed as a compiled binary for all major platforms. We rely on pyinstaller for this, but pyinstaller has only supported Python3 in a stable release since last October. Pyinstaller has been a huge pain-point for us, and I'm waiting for the Python3 support to mature a bit before the transition.

I hope you will be less surprised next time you find a large, complicated tool still stuck on 2.7. The bug tracker is not the place to discuss this (feel free to drop me an email), but I've come to consider the way the Python2-3 transition was handled to be one of the greatest technical blunders of the last decade. Long after mitmproxy itself has shifted to Python3, the vast bulk of production Python code will still be on 2.7. We'd better make peace with that.

If you'd like to help things along for mitmproxy itself, please join us on the Slack channel to discuss, and then to roll up your sleeves and pitch in. ;)

Member

cortesi commented Feb 28, 2016

@ianozsvald Mitmproxy is a good example of a difficult Python 2-3 transition. If you're interested in the issue - and especially if you feel qualified to speak publicly about it - it would be a good idea to look closely and sympathetically at why we're not there yet.

  • Mitmproxy has lots of dependencies, some of which only recently became Python 3 ready themselves.
  • We're not interested in inelegantly straddling Python 2 and 3 - we'll be moving across to Python3 wholesale. A lot of projects integrate mitmproxy as a library, which means that a move to Python3 will force integrators to either stay with old versions of mitmproxy or to move to Python3 themselves.
  • Mitmproxy works at a slightly unusual boundary where Python3's unicode and byte handling becomes a pain point. Armin Ronacher has written eloquently about this (though his post is now somewhat outdated): http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/.
  • Mitmproxy is distributed as a compiled binary for all major platforms. We rely on pyinstaller for this, but pyinstaller has only supported Python3 in a stable release since last October. Pyinstaller has been a huge pain-point for us, and I'm waiting for the Python3 support to mature a bit before the transition.

I hope you will be less surprised next time you find a large, complicated tool still stuck on 2.7. The bug tracker is not the place to discuss this (feel free to drop me an email), but I've come to consider the way the Python2-3 transition was handled to be one of the greatest technical blunders of the last decade. Long after mitmproxy itself has shifted to Python3, the vast bulk of production Python code will still be on 2.7. We'd better make peace with that.

If you'd like to help things along for mitmproxy itself, please join us on the Slack channel to discuss, and then to roll up your sleeves and pitch in. ;)

@ianozsvald

This comment has been minimized.

Show comment
Hide comment
@ianozsvald

ianozsvald Feb 29, 2016

Hi Aldo. Sorry, I wasn't trying to be unsympathetic. I had seen the discussion around a wholesale shift from Py2 to Py3 (this seems like a sensible approach if feasible), I understand the legacy difficulties with longer running users. I've followed and have been involved in the discussions around the move of projects like numpy, matplotlib and django (years back), scrapy (just now feasible via Twisted's updates), PySpark and lots of smaller projects. I've ported a few very small projects (several hours each to successful PR, a few involving web scraping/byte issues), nothing on the scale of mitmproxy.

I'm very aware of the str/unicode/byte pain point and I've followed Armin's posts over the years. I spend almost all of my time in the data science world where almost all the major ports were completed years ago (with dual version support). A part of my outspoken-ness at public meetups is to try to get the slower moving users to upgrade (which is totally feasible in the data science world now) such that the developers have an easier time supporting just the one Python 3 platform.

Re. your request that I pitch in - you're totally right that I should if I wanted the port to move faster, however I won't. mitmproxy was a test for a side project and the Py2.7 version did what I needed. Elsewhere I do put plenty of time into the Py community (right now I'm co-chair for our 3rd annual PyDataLondon O/S conference, I'm also co-chair for our monthly 200+ user meetup), so I hope you'll understand. Sorry to cause noise.

ianozsvald commented Feb 29, 2016

Hi Aldo. Sorry, I wasn't trying to be unsympathetic. I had seen the discussion around a wholesale shift from Py2 to Py3 (this seems like a sensible approach if feasible), I understand the legacy difficulties with longer running users. I've followed and have been involved in the discussions around the move of projects like numpy, matplotlib and django (years back), scrapy (just now feasible via Twisted's updates), PySpark and lots of smaller projects. I've ported a few very small projects (several hours each to successful PR, a few involving web scraping/byte issues), nothing on the scale of mitmproxy.

I'm very aware of the str/unicode/byte pain point and I've followed Armin's posts over the years. I spend almost all of my time in the data science world where almost all the major ports were completed years ago (with dual version support). A part of my outspoken-ness at public meetups is to try to get the slower moving users to upgrade (which is totally feasible in the data science world now) such that the developers have an easier time supporting just the one Python 3 platform.

Re. your request that I pitch in - you're totally right that I should if I wanted the port to move faster, however I won't. mitmproxy was a test for a side project and the Py2.7 version did what I needed. Elsewhere I do put plenty of time into the Py community (right now I'm co-chair for our 3rd annual PyDataLondon O/S conference, I'm also co-chair for our monthly 200+ user meetup), so I hope you'll understand. Sorry to cause noise.

@Kriechi

This comment has been minimized.

Show comment
Hide comment
@Kriechi

Kriechi Jun 19, 2016

Member

Just a quick update:

Thanks to the great work of @dufferzafar as part of this years GSoC, we are getting close to running mitmproxy in Python 3.5. As of a few days ago, pathod is now fully compatible with Python 2.7 and 3.5. netlib was already py3-compatible, thanks to the great efforts by @mhils a couple of months ago.
This means all dependencies for mitmproxy are able to run in Python 2 and 3.

We expect to migrate the remaining mitmproxy code to Python 3 within the next weeks - so stay tuned!

Member

Kriechi commented Jun 19, 2016

Just a quick update:

Thanks to the great work of @dufferzafar as part of this years GSoC, we are getting close to running mitmproxy in Python 3.5. As of a few days ago, pathod is now fully compatible with Python 2.7 and 3.5. netlib was already py3-compatible, thanks to the great efforts by @mhils a couple of months ago.
This means all dependencies for mitmproxy are able to run in Python 2 and 3.

We expect to migrate the remaining mitmproxy code to Python 3 within the next weeks - so stay tuned!

@ianozsvald

This comment has been minimized.

Show comment
Hide comment
@ianozsvald

ianozsvald Jun 20, 2016

Hello again, it is great to hear about the Python 3 porting progress. A while back I wondered if I could use my PyDataLondon audience to gather some useful data for a couple of projects (notably yours, h5py, dask and a couple of others). Last week I queried my London audience, 13% of my 3,400+ members responded to 4 emailed survey prompts. Here's a summary of the results:
http://ianozsvald.com/2016/06/20/results-for-which-version-of-python-2vs3-do-london-data-scientists-use/

In short - my London Data Science audience predominantly use Python 2.7 at work, Python 3.4+ is dominant outside of work and I hypothesise that Python 3.x to be dominant roughly this time next year. Python <=2.6 and <=3.3 are hardly used by any of this audience. Possibly these figures are useful in future discussions you might have about which Python versions need support into the future. Cheers, Ian.

ianozsvald commented Jun 20, 2016

Hello again, it is great to hear about the Python 3 porting progress. A while back I wondered if I could use my PyDataLondon audience to gather some useful data for a couple of projects (notably yours, h5py, dask and a couple of others). Last week I queried my London audience, 13% of my 3,400+ members responded to 4 emailed survey prompts. Here's a summary of the results:
http://ianozsvald.com/2016/06/20/results-for-which-version-of-python-2vs3-do-london-data-scientists-use/

In short - my London Data Science audience predominantly use Python 2.7 at work, Python 3.4+ is dominant outside of work and I hypothesise that Python 3.x to be dominant roughly this time next year. Python <=2.6 and <=3.3 are hardly used by any of this audience. Possibly these figures are useful in future discussions you might have about which Python versions need support into the future. Cheers, Ian.

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Jul 6, 2016

Member

Quick update: We managed to build Python 3.5 binaries for pathod and we only have a couple of modules left for mitmproxy:

  • test/mitmproxy/test_web_master.py
  • test/mitmproxy/test_flow_format_compat.py (#1318)
  • test/mitmproxy/test_examples.py (#1321)
  • test/mitmproxy/test_protocol_http2.py (#1325)
  • test/mitmproxy/test_flow.py
  • test/mitmproxy/test_dump.py (#1299)
Member

mhils commented Jul 6, 2016

Quick update: We managed to build Python 3.5 binaries for pathod and we only have a couple of modules left for mitmproxy:

  • test/mitmproxy/test_web_master.py
  • test/mitmproxy/test_flow_format_compat.py (#1318)
  • test/mitmproxy/test_examples.py (#1321)
  • test/mitmproxy/test_protocol_http2.py (#1325)
  • test/mitmproxy/test_flow.py
  • test/mitmproxy/test_dump.py (#1299)

@dufferzafar dufferzafar added this to the python-3 milestone Jul 7, 2016

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Jul 9, 2016

Member

While there is still a bit to do, thanks to @dufferzafar's great GSoC work all tests pass now on Python 3! 🎉 🐍

Member

mhils commented Jul 9, 2016

While there is still a bit to do, thanks to @dufferzafar's great GSoC work all tests pass now on Python 3! 🎉 🐍

@cortesi

This comment has been minimized.

Show comment
Hide comment
@cortesi

cortesi Sep 4, 2016

Member

The Python2/3 straddle is now complete. Whatever remains should be split up, and attached to the milestone for 1.0, which will be our first Python3-only release.

Member

cortesi commented Sep 4, 2016

The Python2/3 straddle is now complete. Whatever remains should be split up, and attached to the milestone for 1.0, which will be our first Python3-only release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment