Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
PLIP: Port Plone to Python 3 #2368
Proposer: Philip Bauer
Seconder: Alexander Loechel
Support Python 3 and 2.7.
Plone 5.2 should run on Python 2.7, 3.6 and 3.7
For Plone 6 Python 3 should be the default but it should still run on 2.7.
On 2020-01-01 support for Python 2.7 ends. Migrating to Python 3 is a requirement to ensure the continued success of Plone.
Most of the dependencies of Plone (Zope, ZODB, CMFCore, GenericSetup, DCWorkflow, RestrictedPython...) already run on Python 3 or are actively being ported.
Proposal & Implementation
Keep the functionality the same but allow it to run in Python 3 as well as in Python 2.7.
To achieve compatibility with both versions use six, e.g. for changed imports.
In most cases incompatible code can be replaced by code that works in both versions.
Whenever a change is needed the code should be changed in such a way that Python 3 is the default and 2.7 is the fallback that can be removed by deleting complete lines. This should make it easier to drop support for 2.7 in the future.
Approaches to porting Plone to Python 3
There are five strategies to porting Plone to Python 3. We need to follow all of them to make it in reasonable time:
All changes must result in pull-requests that are tested with Python 2.7 and eventually in Python 3.
Plone will continue to support Python 2.7 until a PLIP to drop support for Python 2.7 is written and implemented. Continued support for Python 2.7 beyond 2020-01-01 will ensure that everyone can upgrade choosing their own speed but are not necessarily stuck on a old version of Plone.
Plone will support Python 3.7 and 3.6. Python 3.7 will be released on 2018-06-15 and will be supported until 2023. 3.6 is supported until 2021-12-23 and 3.5 is supported until 2020-09-13.
Python 3.4 has its end of life (EOL) on 2019-03-16 and it is not worth the effort to support it for the limited time it still has. Python 3.5 EOL is 2020-09-13.
See the Python release- and support-roadmap at https://devguide.python.org/#status-of-python-branches
ZServer, WSGI, FTP and WebDAV
There are plans to reimplement ZServer for Python 3 on top of aiohttp at the Sauna Sprint in July 2018. ZServer is not yet ported to Python 3. Since ZServer holds support for FTP and WebDAV these features will need to also be implemented there or replaced.
Currently plone.recipe.zope2instance already supports waitress and uwsgi. The WSGIPublisher in Zope will probably be the default publisher.
When running in Python 2 the old ZServer will still work but is not available in Python 3.
(see discussion below)
The ZODB is compatible with Python 3 but a DB created in Python 2.7 cannot be used in Python 3 without being modified before.
Archetypes deprecation and removal
Plone running on Python 3 will not support Archetypes because it makes no sense to maintain two frameworks and probably nobody will invest the time to migrate AT to Python 3.
The following packages should be deprecated:
As far as I can see the only code from them we need to move is GopipIndex (the index for sorting against
On Python 2 Archetypes will still work but needs to be added as a dependency (maybe as a extra).
The following is half-educated guesswork:
The real timeline depends a lot on other PLIPs and the momentum that the community can muster. I nevertheless suggest to aspire to follow a timeline since 2020-01-01 is a hard deadline that is beyond our influence. Having no stable release on Python 3 some time before that will seriously impact Plone.
This will allow people to test and migrate some addons before 2020.
I do not think that waiting for Plone 6 with this PLIP is an viable option since Plone 6 will most likely depend on the state of pastanaga/plone-react and having two hard dependencies for a mayor release is a great recipe for delays.
Leads: Philip Bauer (@pbauer), ...
We need separate leads for many of the deliverables (e.g. plone.testing, ZODB-Upgrades, FTP/WebDAV etc...). Please consider championing one of the deliverables and add your name here.
As of now (March 30th) Plone already starts up with Python 3 and you can create a instance and play around with it. Many features still do not work, most noticeable static resources. But this foundations allows parallelizing work.
How can I help?
@pbauer Just FYI that we begin reimplementing ZServer for Python 3 on top of aiohttp (unless we find better base) at Sauna Sprint in July 2018. We are serious, because we rely heavily on current asyncore base of ZServer and async patches for wsgi don’t look interesting when asyncio could be an option.
Just pointing out that the deliverable *Document best-practices for porting Addons* in turn depends on several of the other dependencies, because if you want to port an Add-on you need a 5.2 build that can run on Python 3, and you need all the testing infrastructure and plone.testing and plone.app.testing to work on Python 3.…
On Fri, Mar 30, 2018 at 10:43 AM Asko Soukka ***@***.***> wrote: @pbauer <https://github.com/pbauer> Just FYI that we begin reimplementing ZServer for Python 3 on top of aiohttp (unless we find better base) at Sauna Sprint in July 2018. We are serious, because we rely heavily on current asyncore base of ZServer and async patches for wsgi don’t look interesting when asyncio could be an option. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2368 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcHxr5QETTBzryM1rQkEje_cU20lfIRks5tjm6-gaJpZM4TB4fz> .
Also: I'm looking for a partner in crime for this. See
IMHO as we are doing the porting now I would stick with 3.6 for now, there's no much gain to keep our CI matrix more complex for the sake of it... As we have a steady working version on 3.6, we can add the +3.6 python versions on the CI matrix.
I can try to work on the CI bits... one thing that I would like to work on would be on catching regressions, with all the six porting for example, if new pull requests are not tested for regressions, we can endlessly play hide and seek.
@gforcada can you please create a PLIP-job for this in jenkins? The job should at first only follow the steps in #2041 (comment). Running tests is not yet possible. That would help us catch errors from merged or broken branches.
@gforcada thanks for the jenkins-job. The buildout works fine now. But the test itself makes no sense:
timeout 10 ./bin/wsgi.py etc/waitress.ini
It should pass if the last line of output is something like
For me the output looks like this:
Once the testlayers in plone.app.testing are ported to python3 we can start proper testing for real.
added a commit
May 5, 2018
Here comes a quick status-update before the sprint in Halle next week: We've made great progress and most features of Plone work in python 3.
A couple of core-features still need some love (e.g. login in Plone, adding files and images, schema-editor etc.). I'm optimistic that we can make good progress on many of these during the sprint.
We even started testing in jenkins (https://jenkins.plone.org/view/PLIPs/job/plip-py3/). For now we only run the tests of plone.app.contenttypes but for those we already get:
There is still a lot of work to be done but thanks to everyone involved the progress is encouraging.
We have migrated our 120M Python2 objects Relstorage database to Python3 using mdtools.relstorage. We've made the repository public for other ZODB users to use as inspiration:
At the sprint in Halle we made very good progress. Plone is running nicely on Python 3 but now we have to fix over 500 failing tests. I wrote a blogpost at https://www.starzel.de/blog/porting-plone-to-python-3 with some more details.
Here is report of the current state of Plone on Python 3. It covers what was done in the last couple of weeks since the Buschenschank Sprint (https://community.plone.org/t/this-was-the-buschenschanksprint-2018/6711) and lists some things that still need to be done.
&tldr;~Things are looking good but we still need to do a lot of work to migrate some remaining packages, fix all tests and provide a solid database migration-story. If you can, please come to the sprint in Halle in October (https://www.meetup.com/de-DE/Zope-Sprint/events/252468356/)!
On June 25-27 there was a Mini-Sprint in Munich where Jens Klein, David Glick and Philip Bauer worked mainly on fixing the tests with Python 3.
For that sprint we started out with 7459 Tests with 138 failures and 40 errors and ended with 8147 Tests with 113 failures and 56 errors.
We worked on a bunch of tasks:
There were a lot of tests failing where the issue was not obvious. We fixed all of the the following:
Fail on Jenkins only
Some tests still fail when running in Jenkins but pass when running locally. Weird.
After the sprint Jens tackled the package
David ported plone.app.robotframework to Python 3. For this we had to re-add XMLRPC support to Zope (it was removed for Zope 4) and ported it to Python 3. We also had to work around a pretty tough bug in Python itself. See plone/plone.app.robotframework#80 for details.
Locally the robot-test are now running and some are passing. We still need to setup Jenkins properly to run them there and then fix the failing robot-tests.
Migrate tests away from PloneTestCase and PloneTestCaseFixture
We migrated a ton of tests from PloneTestCase to plone.app.testing. I also changed PloneTestCase to use Dexterity instead of Archetypes to make porting some tests easier that rely heavily on self.publish (see plone/plone.app.testing#50). A lot of additional tests are now running because of this change but I also found a couple of issues where dexterity is failing (e.g. testClearFindAndRebuildKeepsModificationDate).
To run doctests in Python 3 and Python 2 we use a pattern
Strategy to deal with random ordering
In Python 3 (as actually in Py2) keys in a dict do not have a deterministic order. There are many tests which implicitly test the order of dicts. They had to be fixed to work in Py2 and Py3).
We applied a couple of different patterns for different tests. The best is to simply sort the keys.
Where are some of the test that used to fail:
Cannot adapt lists
A issue with
Fix issues in GenericSetup
David fixed some more issues in the import/export code of GenericSetup. The main issue there was (as usual) serialization of lists of bytes and text.
Issue with IObjectInitializedEvent
When creating Dexterity-content with invokeFactory there is a issue when a new item is moved before it's completely added. Using createContentInContainer instead fixes this. See plone/plone.app.contentrules#37
Plone runs fine with the current configuration (https://github.com/plone/buildout.coredev/blob/5.2/py3.cfg). But there are a lot of open tasks that need to be done before we can make a release.
Packages that need to be migrated
Fix all tests
We are currently at 9408 tests with 170 failures and 331 errors. The added tests and failures are mostly because I changed PloneTestCase to use Dexterity and also included plone.rest and plone.restapi in the Python 3 build.
Merge all open pull-requests
We currently have 88 open pull-requests that need to be reviewed and merged. They also need change notes and updated trove classifiers to specify support for python3:
(updated by @jensens at 2018-09-27)
We need to start testing the migration-script
Document Support for Windows
So far no one tested Plone with Python 3 on Windows.
We need to add the necessary scripts to the WSGI start script:
Improve Jenkins setup
The Jenkins setup for the PLIP job is basically OK but we soon need to start testing all pull-requests and changes in Python 2 and Python 3. For this we need some serious Tox know how.
Migrate mtest to python 3
@Rotonen did a lot of work on test parallelization with serious speed-improvements (see https://community.plone.org/t/ci-run-speedups-for-buildout-coredev/6225/14). As soon as we test everything with Py2 and Py3 tests take even longer. We are looking forward for the tests speedup be in use.
Adapt Plone's ZMI pages
There is a lot of work being done to port the ZMI to Bootstrap (see zopefoundation/Zope#249). We will have to make a list all ZMI templates that Plone adds and test/fix them the new ZMI theme.
Port development plugins
The most important developer tools are probably
Port other plugins
We would love to see that some frequently used plugins are already compatible with Python 3 when when the first alpha of 5.2 is released.
Test against Python 3.7
Python 3.7 is out. We need to update the jenkins-jobs to test against it. One of the changes that might be relevant is that dicts preserve the order of items by insertion.
Deal with Deprecation- and ResourceWarnings
Running Plone on Python 3 yields many deprecation warnings. We should fix these. Also there are a lot of ResourceWarnings for unclosed files, we want to get rid of these by using
Please pitch in to bring Plone to Python 3:
referenced this issue
Jul 14, 2018
The Saltlabs-Sprint in Halle (https://www.meetup.com/de-DE/Zope-Sprint/events/252468356) is approaching. More than 30 people are coming to the sprint (!!!) and I hope we will get a lot done. In preparation for the sprint there is some work under way:
Merge branches: @jensens, @thet and I are currently trying to merge as many python3-branches as possible. We're currently down to 44 unmerged braches. Many of the remaining branches work fine in python 3 but depend on changes in other packages (mosty plone.app.testing, plone.testing, Products.CMFPlone or plone.rfc822), so they are not easily merged.
Fix failing tests: We currently have 216 failing tests in Python 3, most of those are in CMFPlone (because I changed PloneTestCase to use Dexterity) and in plone.restapi.
With https://jenkins.plone.org/view/PLIPs/job/py3_on_py2/ I created a jenkins-job that runs the branches of this PLIP in Python 2. That should make it easier to find issues that need to be fixed before merging.
I would have loved to have a alpha-release of 5.2 out before the sprint but fixing all remaining tests and merging all remaining branches seems more and more unlikely (even I have to sleep sometimes and need to earn some money...).
But: Things are looking good:
If you want to help:
The tasks for the sprint are basically still the same as mentioned above. We'll publish a more detailed list next week.
Regarding WebDAV and WSGI - there is a WSGI middleware for WebDAV (which also includes a standalone WebDAV server):
This project looks quite promising:
This could help us a lot providing WebDAV on WSGI for Plone/Zope.
There are some test failures in CMFCore due to Plone monkey patches that change the behavior that CMFCore tests for (at least patches/dateIndexPatch and patches/speed). Ideally each patch should come with a function to apply it and a function to unapply it and we could apply them as part of the Plone test fixture and during Zope startup instead of at import time. But that's probably quite some work so the practical thing to do for now is to trust that CMFCore is fully tested prior to release and stop running its tests as part of Plone's test suite.
This was referenced
Oct 2, 2018
A super-short update: