New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Python 3.3 or newer #160

Open
steinn opened this Issue Sep 5, 2013 · 32 comments

Comments

Projects
None yet
@steinn
Contributor

steinn commented Sep 5, 2013

Mailpile is currently written in Python 2.7, which is no longer being actively developed and improved by the Python community. In order to stay current with the state of the art and reap the benefits of ongoing developments, Mailpile should upgrade to Python 3.3 (or newer).

(Rephrased by BRE for voting. Original issue text follows.)

I'm curious what Python version(s) Mailpile is targeting. It would be interesting to see Mailpile supporting Python 2.7 and 3.3+

The current code runs on Python 2.7 and it can probably run on Python 2.6 with minor changes (if any). By only supporting 2.7 and 3.3+ it should hopefully be fairly easy to use the same code base.

Python 2.6 end of life is in October, PEP 361, and therefor I think a good reason to not supporting it.
Python 2.7 end of life is in May 2015, PEP 373, which isn't far away!

Ubuntu is also planing on moving to Python 3 in the 14.04 Release, Ubuntu Python 3, which should be released according to schedule in April 2014. That alone is quite interesting and will hopefully get more people using Python 3.

An issue with updating the current code base to Python 3.3+ support is the Python GnuPGInterface library. It does not support Python 3.3 and probably never will (at least by upstream) since it has not been updated since 2002. Haven't looked into alternatives but there must be something.

If there is interest for his I'm willing to help.

@Ivoz

This comment has been minimized.

Contributor

Ivoz commented Sep 6, 2013

Regarding the Python GnuPGInterface library, it seems there are newer versions (0.3.5) on both pypi and googlecode.

https://pypi.python.org/pypi/python-gnupg/
https://code.google.com/p/python-gnupg/

The newer PyPI tags claim compatibility all the way up to python 3.3, so it would be interesting to hear why the older sourceforge source is being used and not pypi.

@danx0r

This comment has been minimized.

Contributor

danx0r commented Sep 7, 2013

we may want to look at http://pythonhosted.org/six/

seems to facilitate writing 2.7-compatible code that is as future-proof against py3 changes as possible

@illume

This comment has been minimized.

illume commented Mar 18, 2014

For best compatibility, single source is the way forward for python projects. Since the stated reason for staying with the legacy python 2.x was for best compatibility, it seems time to change the stance of the project to move to a single source.

It is almost April 2014 so more distros are moving to python 3. The python community is also moving in a big way to python 3.3/3.4. As previously said, python 2.7 is nearing end of life, meaning no security fixes, and no updates for platforms (new windows/osx etc). Writing a single source code base that works on 3.3+ and 2.6/2.7 isn't too much different, and lots of projects have documented their experiences with this method now.

As well as six, there is http://python-future.org/ and some of the projects with small compatibility modules include twisted, numpy, pygame, coverage, etc.

Here is an old blog post from 2009 showing how it is done: http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html

The jinja2 package has a _compat module which is about 100 lines long: https://github.com/mitsuhiko/jinja2/blob/master/jinja2/_compat.py

Since you are already using jinja2, I don't see why you couldn't vendorise its compatibility module :)

@BjarniRunar BjarniRunar added the Back End label Apr 4, 2014

@BjarniRunar

This comment has been minimized.

Member

BjarniRunar commented Apr 4, 2014

I agree that this is becoming more compelling all the time. We have already started to hit what at first glance looks like limitations in the python base libraries (e.g. #558 ) and it's unlikely that those bugs will ever get fixed on the 2.x branch.

I also did some very basic tests which implied that Python 3 is much better at releasing unused memory back to the operating system, and considering that RAM usage is one of our main bottlenecks that is a strong argument in favour of making the switch.

The core team (myself, @smari and @brennannovak) is discussing this.

@nifgraup

This comment has been minimized.

Contributor

nifgraup commented Apr 16, 2014

Python 2.7 EOL has been moved 5 years into the future, see http://hg.python.org/peps/file/76d43e52d978/pep-0373.txt

@Ivoz

This comment has been minimized.

Contributor

Ivoz commented Apr 17, 2014

Mailpile is still a relatively new application, so it shouldn't be aiming to stay behind.

@steinn steinn closed this Apr 17, 2014

@steinn steinn reopened this Apr 17, 2014

@steinn

This comment has been minimized.

Contributor

steinn commented Apr 17, 2014

@nifgraup yes but after 2015 there isn't a guarantee of regular releases, only if the core devs have interest in it. I think they are hoping that vendors (ubuntu, red hat, etc.) will handle releasing patches (this is my understanding from the python dev mailing list).

@Ivoz

This comment has been minimized.

Contributor

Ivoz commented Apr 17, 2014

The releases will essentially just be bugfix releases, when python devs feel one is warranted for whatever comes up in the future. The promise is essentially saying they'll keep on officially paying attention to 2.7 and releasing bugfixes as needed until 2020.

@BjarniRunar BjarniRunar removed their assignment Apr 26, 2014

@nidico

This comment has been minimized.

nidico commented Oct 22, 2014

This is also relevant due to STARTTLS support in imaplib starting in Python 3.2 (see #868).

@paulfurley

This comment has been minimized.

Contributor

paulfurley commented Nov 19, 2014

+1 for Python3 support, especially as you're a "greenfield" project.

I'm trying to run on Python 3 - blocker so far is the python spambayes library.

@BjarniRunar BjarniRunar added the Roadmap label Nov 28, 2014

@BjarniRunar BjarniRunar added this to the Post 1.0 Roadmap milestone Nov 28, 2014

@BjarniRunar BjarniRunar changed the title from Python 3.3+ support to Mailpile should upgrade to Python 3.3 or newer Nov 28, 2014

@BjarniRunar BjarniRunar changed the title from Mailpile should upgrade to Python 3.3 or newer to Upgrade to Python 3.3 or newer Nov 28, 2014

@Ivoz

This comment has been minimized.

Contributor

Ivoz commented Dec 10, 2014

For python 3 libraries, there is a port of PyDNS called Py3DNS. I'm unsure about its cross compatibility with Python 2. PGPDump is 2/3 compatible, same with MarkupSafe. Spambayes hasn't been updated since 2011 and I assume is python 2 compatible only, so you'd need to work out what to do about that. Is it the best python library for filtering spam?

@paulfurley

This comment has been minimized.

Contributor

paulfurley commented Dec 10, 2014

I looked into porting spambayes a few weeks back but fell at the first hurdle - sourceforge!! Might be worth forking and bringing over to github, wonder how the code looks... :S

@konklone

This comment has been minimized.

konklone commented May 10, 2015

Note that since this ticket was filed, the communty is up to Python 3.4 (3.4.3, as of this comment).

@nklever

This comment has been minimized.

nklever commented Jun 28, 2015

One more argument to upgrade to Python 3.x is due to STARTTLS support in pop3lib starting in Python 3.4 (see POP3lib Doc)

@joar

This comment has been minimized.

joar commented Jul 3, 2015

I'd put a vote on this, however I do this.

I don't have any experience making single-source Python 2/3 applications beyond from __future__ import print_function, but as far as I know, six is the way to do it, and it's what Djjango uses.

I don't know how much 2to3 will do for you, but it might be an idea to run it against mailpile and see how much you get.

@Ivoz

This comment has been minimized.

Contributor

Ivoz commented Jul 28, 2015

@joar the biggest hurdle is usually getting your dependencies 2/3 compatible first.

@lukebranch

This comment has been minimized.

lukebranch commented Jul 28, 2015

I recommend keeping Python 2.7 compatibility. There are still a huge number of projects and libraries that will stay with 2.7 for the foreseeable future.

@FirefighterBlu3

This comment has been minimized.

FirefighterBlu3 commented Aug 12, 2015

I'm a huge user of py3, my entire set of servers in numerous datacenters is nearly Python 3.4 pure. It's pretty easy to support py2.7 with native py3 written code which is probably the best design since it only means a few tweak to support "old" python while the rest is designed for modern python.

I've just started in on mailpile and I'm happy to help with a bit of time here and there.

@BjarniRunar

This comment has been minimized.

Member

BjarniRunar commented Aug 16, 2015

Just wanted to chime in again; thanks for all the comments guys. This remains on our long term roadmap, but won't be addressed in the near term because our priority at the moment is to just ship something that works. :-)

@da2x

This comment has been minimized.

Contributor

da2x commented Feb 2, 2016

The Nikola project went through a Python version re-prioritization recently. Long story short, the project will drop Python 2.7 because it’s a hassle to maintain and most users run Python 3 or had it setup.

An environment with only Python 2.7 is not receiving updates and should be considered insecure. It’s five and a half year old and only receiving a trickle of maintenance updates. Given the Mailpile project’s focus is on security, it should only rely on recent software that receives updates. I see the major roadbloacks for running on Python 3 now are dependencies on libraries like Spambayes that aren’t receiving updates either.

Mailpile is now one of only five software on my machine that runs Py2 and not Py3. Please bring that list down to four.

@aviau

This comment has been minimized.

Contributor

aviau commented Feb 2, 2016

👍

AFAIK @BjarniRunar does not have the time for that now. It would be great if one of you guys could start looking into dependencies like Spambayes, maybe contribute some Python3 support patches to them! I'm sure Python3 support patches wouldn't be refused in Mailpile either, but we have to find a way to do it incrementally and without affecting the upcoming release.

It could be a great way to start contirbuting to Mailpile:

  • write tests for python2
  • gradually enable tests for python3, as they work
  • this way we detect regressions and we can move incrementally

If there is interest in helping the Mailpile Dev Team (the one and only @BjarniRunar) with this, I would definitely put some time in it.

Its relatively easy to support both Python3 and Python2. I think that this is what we should do for now.

@BjarniRunar

This comment has been minimized.

Member

BjarniRunar commented Feb 3, 2016

@aviau That sounds like a very reasonable plan. Thank you for the suggestion!

@tilboerner

This comment has been minimized.

tilboerner commented Feb 21, 2017

What's the current state of this issue? I think it might be helpful to have a roadmap or checklist for the migration to Python 3. It would make it easy to tackle the project one step at a time, make progress visible, and newcomers might find it easier to find a starting point.

Which dependencies are there, which have completed the move, which ones haven't? Assuming the application will remain Python2 compatible for a while, which parts of the application can be migrated independently? Which are the necessary steps for each part? Is 3.3.* still the minimum target, or can it be moved to 3.4.*?

I'm thinking of something like the following (feel free to edit it out of this comment):

Dependencies

  • pyfoomatic (v3.1.4)
  • dingbats (not as of v0.9)

Mailpile

  • package
    • package.subpackage
      Here is a relevant note.
      • test coverage
      • migrated code
@BjarniRunar

This comment has been minimized.

Member

BjarniRunar commented Feb 21, 2017

Hi @tilboerner!

Current state: this is a "community only" issue at the moment, and I'm not aware of anyone from the community actively working on it. I'm not willing to spend any effort on it myself because I am already overworked and my efforts are more needed elsewhere (making a usable release).

So if you're interested, just dive in... whoever does the work becomes the boss. If you want to create a TODO list, that sounds super reasonable. If you want to just start randomly patching files to make them 3.x compatible, that's also perfectly fine. :-)

However, to prevent you from wasting your time: please keep pull requests very small in scope so I can easily review them. Many small pull requests will get merged much more quickly than a few large ones.

@koleesch1

This comment has been minimized.

koleesch1 commented Aug 10, 2017

Hello, here is my requirements-dev.txt for pip to run virtualenv...

SpamBayes doesn't exists for python 3...
Pydns moved to Py3dns

I get an error during start of mailpile
"from jinja2 import Environment, BaseLoader, TemplateNotFound
ImportError: No module named jinja2"
jinja2 is installed...

requirements_dev.txt

appdirs
setuptools>=11.3
cryptography>=1.3.4
lxml>=2.3.2
Jinja2
#spambayes>=1.1b1
selenium>=2.40.0
markupsafe
nose
mock>=1.0.1
pycrypto
py3dns
pgpdump
pillow
pbr
pycrypto

@jonathan-s

This comment has been minimized.

jonathan-s commented Jan 27, 2018

@BjarniRunar It seems like spambayes is the main blocker for python3, would something like https://github.com/dinever/antispam be acceptable?

@BjarniRunar

This comment has been minimized.

Member

BjarniRunar commented Jan 27, 2018

@jonathan-s: Mailpile's antispam and statistical classifiers are written in a modular way, so dropping in a replacement for spambayes should be relatively straightforward. So yes, if it works well then that's a roadblock cleared! The bar isn't low - spambayes works really well in my experience - but replacing it with something slightly worse but actively being developed/improved is still an option, especially if it lets us move towards Python 3.x down the line.

Looking at the antispam package you linked, the API it currently offers appears too limited (in particular, I saw no way to ensure the classifier database was saved encrypted), but if it's under development it could certainly be fixed.

@jonathan-s

This comment has been minimized.

jonathan-s commented Jan 27, 2018

@BjarniRunar I've been trying to find other packages related to bayesian spam. Sadly it seems like there are not many packages out there. And for the linked package it looks fairly simple and I'm unsure whether it'll add many more features (unless someone from mailpile community decides that these features are important).

For mailpile what features are important for spam classification? You mentioned that the classifier database needs to be saved encrypted.

@BjarniRunar

This comment has been minimized.

Member

BjarniRunar commented Jan 27, 2018

@jonathan-s, in practice load/save to/from an in-memory buffer is required for the encryption. But if there is a classifier class that cooperates with Python's pickle, then we get this free - so on second thought there may not be a need for an explicit API.

Our spambayes code uses a lower-level spambayes API to bypass the spambayes built-in tokenizer and instead use the Mailpile search-engine tokenizer - the idea being that consistency between those two subsystems would be useful for exploring/explaining spam classifier decisions at some point, but also to ensure that metadata like key IDs and message structure were used during classification. Basically, if it's useful to be able to search for an attribute, it should be a useful signal for classification.

Our entire interface with Spambayes is tiny: https://github.com/mailpile/Mailpile/blob/master/mailpile/plugins/autotag_sb.py

@jonathan-s

This comment has been minimized.

jonathan-s commented Jan 27, 2018

@BjarniRunar

This comment has been minimized.

Member

BjarniRunar commented Jan 27, 2018

@jonathan-s - I haven't looked into it, but that may also be a feasible way to resolve this. Part of me feels it would be better for the wider community if we made an effort to take over spambayes and port it to Python 3, but I have no idea how difficult that will be. And perfect is of course the enemy of good...

@jonathan-s

This comment has been minimized.

jonathan-s commented Jan 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment