Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Python 3.3 or newer #160

Open
steinn opened this issue Sep 5, 2013 · 39 comments
Open

Upgrade to Python 3.3 or newer #160

steinn opened this issue Sep 5, 2013 · 39 comments

Comments

@steinn
Copy link
Contributor

@steinn steinn commented Sep 5, 2013

Mailpile is currently written in Python 2.7, which is no longer being actively developed and improved by the Python community. In order to stay current with the state of the art and reap the benefits of ongoing developments, Mailpile should upgrade to Python 3.3 (or newer).

(Rephrased by BRE for voting. Original issue text follows.)

I'm curious what Python version(s) Mailpile is targeting. It would be interesting to see Mailpile supporting Python 2.7 and 3.3+

The current code runs on Python 2.7 and it can probably run on Python 2.6 with minor changes (if any). By only supporting 2.7 and 3.3+ it should hopefully be fairly easy to use the same code base.

Python 2.6 end of life is in October, PEP 361, and therefor I think a good reason to not supporting it.
Python 2.7 end of life is in May 2015, PEP 373, which isn't far away!

Ubuntu is also planing on moving to Python 3 in the 14.04 Release, Ubuntu Python 3, which should be released according to schedule in April 2014. That alone is quite interesting and will hopefully get more people using Python 3.

An issue with updating the current code base to Python 3.3+ support is the Python GnuPGInterface library. It does not support Python 3.3 and probably never will (at least by upstream) since it has not been updated since 2002. Haven't looked into alternatives but there must be something.

If there is interest for his I'm willing to help.

@Ivoz
Copy link
Contributor

@Ivoz Ivoz commented Sep 6, 2013

Regarding the Python GnuPGInterface library, it seems there are newer versions (0.3.5) on both pypi and googlecode.

https://pypi.python.org/pypi/python-gnupg/
https://code.google.com/p/python-gnupg/

The newer PyPI tags claim compatibility all the way up to python 3.3, so it would be interesting to hear why the older sourceforge source is being used and not pypi.

@danx0r
Copy link
Contributor

@danx0r danx0r commented Sep 7, 2013

we may want to look at http://pythonhosted.org/six/

seems to facilitate writing 2.7-compatible code that is as future-proof against py3 changes as possible

@illume
Copy link

@illume illume commented Mar 18, 2014

For best compatibility, single source is the way forward for python projects. Since the stated reason for staying with the legacy python 2.x was for best compatibility, it seems time to change the stance of the project to move to a single source.

It is almost April 2014 so more distros are moving to python 3. The python community is also moving in a big way to python 3.3/3.4. As previously said, python 2.7 is nearing end of life, meaning no security fixes, and no updates for platforms (new windows/osx etc). Writing a single source code base that works on 3.3+ and 2.6/2.7 isn't too much different, and lots of projects have documented their experiences with this method now.

As well as six, there is http://python-future.org/ and some of the projects with small compatibility modules include twisted, numpy, pygame, coverage, etc.

Here is an old blog post from 2009 showing how it is done: http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html

The jinja2 package has a _compat module which is about 100 lines long: https://github.com/mitsuhiko/jinja2/blob/master/jinja2/_compat.py

Since you are already using jinja2, I don't see why you couldn't vendorise its compatibility module :)

@BjarniRunar BjarniRunar added the Back End label Apr 4, 2014
@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Apr 4, 2014

I agree that this is becoming more compelling all the time. We have already started to hit what at first glance looks like limitations in the python base libraries (e.g. #558 ) and it's unlikely that those bugs will ever get fixed on the 2.x branch.

I also did some very basic tests which implied that Python 3 is much better at releasing unused memory back to the operating system, and considering that RAM usage is one of our main bottlenecks that is a strong argument in favour of making the switch.

The core team (myself, @smari and @brennannovak) is discussing this.

@nifgraup
Copy link
Contributor

@nifgraup nifgraup commented Apr 16, 2014

Python 2.7 EOL has been moved 5 years into the future, see http://hg.python.org/peps/file/76d43e52d978/pep-0373.txt

@Ivoz
Copy link
Contributor

@Ivoz Ivoz commented Apr 17, 2014

Mailpile is still a relatively new application, so it shouldn't be aiming to stay behind.

@steinn steinn closed this Apr 17, 2014
@steinn steinn reopened this Apr 17, 2014
@steinn
Copy link
Contributor Author

@steinn steinn commented Apr 17, 2014

@nifgraup yes but after 2015 there isn't a guarantee of regular releases, only if the core devs have interest in it. I think they are hoping that vendors (ubuntu, red hat, etc.) will handle releasing patches (this is my understanding from the python dev mailing list).

@Ivoz
Copy link
Contributor

@Ivoz Ivoz commented Apr 17, 2014

The releases will essentially just be bugfix releases, when python devs feel one is warranted for whatever comes up in the future. The promise is essentially saying they'll keep on officially paying attention to 2.7 and releasing bugfixes as needed until 2020.

@BjarniRunar BjarniRunar removed their assignment Apr 26, 2014
@nidico
Copy link

@nidico nidico commented Oct 22, 2014

This is also relevant due to STARTTLS support in imaplib starting in Python 3.2 (see #868).

@fawkesley
Copy link
Contributor

@fawkesley fawkesley commented Nov 19, 2014

+1 for Python3 support, especially as you're a "greenfield" project.

I'm trying to run on Python 3 - blocker so far is the python spambayes library.

@BjarniRunar BjarniRunar added the Roadmap label Nov 28, 2014
@BjarniRunar BjarniRunar added this to the Post 1.0 Roadmap milestone Nov 28, 2014
@BjarniRunar BjarniRunar changed the title Python 3.3+ support Mailpile should upgrade to Python 3.3 or newer Nov 28, 2014
@BjarniRunar BjarniRunar changed the title Mailpile should upgrade to Python 3.3 or newer Upgrade to Python 3.3 or newer Nov 28, 2014
@Ivoz
Copy link
Contributor

@Ivoz Ivoz commented Dec 10, 2014

For python 3 libraries, there is a port of PyDNS called Py3DNS. I'm unsure about its cross compatibility with Python 2. PGPDump is 2/3 compatible, same with MarkupSafe. Spambayes hasn't been updated since 2011 and I assume is python 2 compatible only, so you'd need to work out what to do about that. Is it the best python library for filtering spam?

@fawkesley
Copy link
Contributor

@fawkesley fawkesley commented Dec 10, 2014

I looked into porting spambayes a few weeks back but fell at the first hurdle - sourceforge!! Might be worth forking and bringing over to github, wonder how the code looks... :S

@konklone
Copy link

@konklone konklone commented May 10, 2015

Note that since this ticket was filed, the communty is up to Python 3.4 (3.4.3, as of this comment).

@nklever
Copy link

@nklever nklever commented Jun 28, 2015

One more argument to upgrade to Python 3.x is due to STARTTLS support in pop3lib starting in Python 3.4 (see POP3lib Doc)

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Aug 16, 2015

Just wanted to chime in again; thanks for all the comments guys. This remains on our long term roadmap, but won't be addressed in the near term because our priority at the moment is to just ship something that works. :-)

@da2x
Copy link
Contributor

@da2x da2x commented Feb 2, 2016

The Nikola project went through a Python version re-prioritization recently. Long story short, the project will drop Python 2.7 because it’s a hassle to maintain and most users run Python 3 or had it setup.

An environment with only Python 2.7 is not receiving updates and should be considered insecure. It’s five and a half year old and only receiving a trickle of maintenance updates. Given the Mailpile project’s focus is on security, it should only rely on recent software that receives updates. I see the major roadbloacks for running on Python 3 now are dependencies on libraries like Spambayes that aren’t receiving updates either.

Mailpile is now one of only five software on my machine that runs Py2 and not Py3. Please bring that list down to four.

@aviau
Copy link
Contributor

@aviau aviau commented Feb 2, 2016

👍

AFAIK @BjarniRunar does not have the time for that now. It would be great if one of you guys could start looking into dependencies like Spambayes, maybe contribute some Python3 support patches to them! I'm sure Python3 support patches wouldn't be refused in Mailpile either, but we have to find a way to do it incrementally and without affecting the upcoming release.

It could be a great way to start contirbuting to Mailpile:

  • write tests for python2
  • gradually enable tests for python3, as they work
  • this way we detect regressions and we can move incrementally

If there is interest in helping the Mailpile Dev Team (the one and only @BjarniRunar) with this, I would definitely put some time in it.

Its relatively easy to support both Python3 and Python2. I think that this is what we should do for now.

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Feb 3, 2016

@aviau That sounds like a very reasonable plan. Thank you for the suggestion!

@tilboerner
Copy link

@tilboerner tilboerner commented Feb 21, 2017

What's the current state of this issue? I think it might be helpful to have a roadmap or checklist for the migration to Python 3. It would make it easy to tackle the project one step at a time, make progress visible, and newcomers might find it easier to find a starting point.

Which dependencies are there, which have completed the move, which ones haven't? Assuming the application will remain Python2 compatible for a while, which parts of the application can be migrated independently? Which are the necessary steps for each part? Is 3.3.* still the minimum target, or can it be moved to 3.4.*?

I'm thinking of something like the following (feel free to edit it out of this comment):

Dependencies

  • pyfoomatic (v3.1.4)
  • dingbats (not as of v0.9)

Mailpile

  • package
    • package.subpackage
      Here is a relevant note.
      • test coverage
      • migrated code
@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Feb 21, 2017

Hi @tilboerner!

Current state: this is a "community only" issue at the moment, and I'm not aware of anyone from the community actively working on it. I'm not willing to spend any effort on it myself because I am already overworked and my efforts are more needed elsewhere (making a usable release).

So if you're interested, just dive in... whoever does the work becomes the boss. If you want to create a TODO list, that sounds super reasonable. If you want to just start randomly patching files to make them 3.x compatible, that's also perfectly fine. :-)

However, to prevent you from wasting your time: please keep pull requests very small in scope so I can easily review them. Many small pull requests will get merged much more quickly than a few large ones.

@koleesch1
Copy link

@koleesch1 koleesch1 commented Aug 10, 2017

Hello, here is my requirements-dev.txt for pip to run virtualenv...

SpamBayes doesn't exists for python 3...
Pydns moved to Py3dns

I get an error during start of mailpile
"from jinja2 import Environment, BaseLoader, TemplateNotFound
ImportError: No module named jinja2"
jinja2 is installed...

requirements_dev.txt

appdirs
setuptools>=11.3
cryptography>=1.3.4
lxml>=2.3.2
Jinja2
#spambayes>=1.1b1
selenium>=2.40.0
markupsafe
nose
mock>=1.0.1
pycrypto
py3dns
pgpdump
pillow
pbr
pycrypto

@jonathan-s
Copy link

@jonathan-s jonathan-s commented Jan 27, 2018

@BjarniRunar It seems like spambayes is the main blocker for python3, would something like https://github.com/dinever/antispam be acceptable?

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Jan 27, 2018

@jonathan-s: Mailpile's antispam and statistical classifiers are written in a modular way, so dropping in a replacement for spambayes should be relatively straightforward. So yes, if it works well then that's a roadblock cleared! The bar isn't low - spambayes works really well in my experience - but replacing it with something slightly worse but actively being developed/improved is still an option, especially if it lets us move towards Python 3.x down the line.

Looking at the antispam package you linked, the API it currently offers appears too limited (in particular, I saw no way to ensure the classifier database was saved encrypted), but if it's under development it could certainly be fixed.

@jonathan-s
Copy link

@jonathan-s jonathan-s commented Jan 27, 2018

@BjarniRunar I've been trying to find other packages related to bayesian spam. Sadly it seems like there are not many packages out there. And for the linked package it looks fairly simple and I'm unsure whether it'll add many more features (unless someone from mailpile community decides that these features are important).

For mailpile what features are important for spam classification? You mentioned that the classifier database needs to be saved encrypted.

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Jan 27, 2018

@jonathan-s, in practice load/save to/from an in-memory buffer is required for the encryption. But if there is a classifier class that cooperates with Python's pickle, then we get this free - so on second thought there may not be a need for an explicit API.

Our spambayes code uses a lower-level spambayes API to bypass the spambayes built-in tokenizer and instead use the Mailpile search-engine tokenizer - the idea being that consistency between those two subsystems would be useful for exploring/explaining spam classifier decisions at some point, but also to ensure that metadata like key IDs and message structure were used during classification. Basically, if it's useful to be able to search for an attribute, it should be a useful signal for classification.

Our entire interface with Spambayes is tiny: https://github.com/mailpile/Mailpile/blob/master/mailpile/plugins/autotag_sb.py

@jonathan-s
Copy link

@jonathan-s jonathan-s commented Jan 27, 2018

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Jan 27, 2018

@jonathan-s - I haven't looked into it, but that may also be a feasible way to resolve this. Part of me feels it would be better for the wider community if we made an effort to take over spambayes and port it to Python 3, but I have no idea how difficult that will be. And perfect is of course the enemy of good...

@jonathan-s
Copy link

@jonathan-s jonathan-s commented Jan 27, 2018

BjarniRunar added a commit that referenced this issue Jul 19, 2019
@intika
Copy link

@intika intika commented Jan 7, 2020

💀 RIP Python 2 ^^

@darkuranium
Copy link

@darkuranium darkuranium commented Nov 11, 2020

Any news on this? Python2's been EOL for nearly a year by now.

@bjmgeek
Copy link

@bjmgeek bjmgeek commented Nov 12, 2020

It works for me on PyPy 3
Python 3.6.9 (7.3.2+dfsg-2, Sep 26 2020, 23:01:23)
[PyPy 7.3.2 with GCC 10.2.0]

The only thing was that requirements.txt included pydns but for python3 it needs to be py3dns

@BjarniRunar
Copy link
Member

@BjarniRunar BjarniRunar commented Nov 12, 2020

Thank you for reporting that, @bjmgeek ! We had done a bunch of prep work, but actually testing it never happened. It's really heartening to hear this, makes the whole thing seem much less daunting.

@bjmgeek
Copy link

@bjmgeek bjmgeek commented Nov 16, 2020

It doesn't pass unit tests though. pydns being one sticking point.

@raboof
Copy link

@raboof raboof commented Dec 13, 2020

It works for me on PyPy 3

on master? I tried running master with python3 but that immediately produced errors - I tried to stumble through fixing some of those in master...raboof:python3 (but be aware I don't actually know python ;) ) - is there some other trick I'm missing?

@reyman
Copy link

@reyman reyman commented Jan 12, 2021

I'm also interested by an answer on this @BjarniRunar @bjmgeek ? There is only py3dns that blocks ?

@reyman reyman mentioned this issue Jan 12, 2021
5 of 10 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet