PLIP: Update Zope Dependencies

[thet]

Proposer : Johannes Raggam

Seconder : Philip Bauer

Abstract

This PLIP is about updating Plone's framework dependencies (ZCA, ZTK and Zope) to the latest possible versions.

Motivation

Plone 5 still uses ages old ZTK dependencies and Zope2. While the ZTK dependencies are still actively developed and most of them already Python 3 compatible, the development for Zope 2 has nearly stopped. The successor of Zope2 is Zope 4. The major release version indicates that backwards incompatible refactoring has happened: long deprecated stuff was removed, some performance improvements and bugfixes were done, a few features were added.
To be able to move forward with Plone, we have to modernize our framework dependencies. Also, this PLIP is a precondition for Python 3 compatibility.

Assumptions

• We are not able to switch to Pyramid / SubstanceD soon.
• Modernizing our framework dependencies will help with such a possible switch in the future.
• A recent set of ZTK packages is a precondition for Python 3 compatibility.
• Patching and hacking the ZPublisher is better done with a cleaned up version.

Proposal & Implementation

PLIP configuration: https://github.com/plone/buildout.coredev/blob/5.0/plips/plip-zope4.cfg
Jenkins PLIP job: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/

The PLIP buildout configuration extends versions.cfg and sources.cfg from the Zope project. We added the option auto-checkout = * to check out all sources with mr.developer to get the very latest versions for each dependency. Only some test dependencies are installed as egg. For easier tracking, the eggs are installed in the plips/eggs folder.

We started to fix test failures and created branches and pull requests for all of our changes. Not everything is pull-requested yet. Many of the failures came from moved imports, removed functionality (e.g. no doctest suite any more in zope.testing, as we can use the stdlib for that) but also bugfixes in production code. E.g:

• Compatibility with PythonScripts zopefoundation/zope.publisher#10
• zopefoundation/Zope@3320082
• Interface comparison with emtpy __name__ reports incorrect equality zopefoundation/zope.interface#31

All fixes are made backwards compatible for our current set of dependencies.

At the Alpine City Sprint most of the failures could be fixed. There are only a few test failures left. Also, there are a lot of test isolation problems, but that's another story.

Two packages are not yet updated: Products.ZCatalog and ZODB 4. We still have to decide, if these packages are part of this PLIP. Updating them involves finding a solution for the "ZCatalog Regression: indexing None Values" problem: zopefoundation/Products.ZCatalog#5

We propose this PLIP for inclusion in Plone 6.

Deliverables

• Test fixes.
• Bug fixes.
• Green Jenkins build.
• Upgrade guide based on changes on packages within this PLIP.
• Check monkey patches for their compatibility with Zope 4.

- [ ] Replace dependency on "Zope2" with "Zope". Eventually provide a Zope2 shim. - not necessary - Zope version 4.0 is still named Zope2.

Risks

Zope 4 is currently in alpha state, there is no release yet and development activity is - as with Zope2 - not very high.
We need to show more responsibility for Zope 2 and related packages.

Participants

• Johannes Raggam ( @thet / PLIP proposer / Alpine City Sprint Zope 4 Team)
• Philip Baiuer ( @pbauer / PLIP seconder / Alpine City Sprint Zope 4 Team)
• Gil Forcada ( @gforcada / Jenkins Zope 4 PLIP Job)
• Maurits van Rees ( @mauritsvanrees / Tracking down and fixing Compatibility with PythonScripts zopefoundation/zope.publisher#10 )
• Jens Klein ( @jensens / Tracking down Zope 2 product initialization bug: zopefoundation/Zope@3320082 )
• Alexander Loechel ( @loechel / Began working on Products.RestrictedPython bringing to Python 3. Not part of this PLIP)

Python 3

Python 3 compatibility is not part of the PLIP. A major showstopper for Python 3 compatibility is Products.RestrictedPython. @loechel worked on that during the Alpine City Sprint 2016, which resulted in this document: zopefoundation/RestrictedPython@45c50cb
Also, ZODB 4 is a precondition for Python 3 compatibility.

Notes

Some experiments to install Plone 5 on Zope 4 started back in 2013: https://dev.plone.org/ticket/13822
At the Plone conference 2015 I demoed a running installation of Plone 5 on Zope 4 with the package: https://github.com/collective/experimental.zope4 and started to fix some failing tests.
A major effort on working on this PLIP was done at the Alpine City Strategic Sprint 2016 in Innsbruck: http://www.coactivate.org/projects/alpinecitysprint-2016
Related to this PLIP: "WIP try ZODB3" plone/buildout.coredev#137

[pbauer]

\o/ tests are finally green: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/16/console

There are some pull-request to be made for plone and zope-packages and at least one issue needs to be looked at (plone/plone.supermodel@8a4d202).

The plip-job uses source-checkouts for everything but the following:

BTrees = 4.2.0
Babel = 1.3
Chameleon = 2.24
FormEncode = 1.3.0
Markdown = 2.6.5
Pillow = 3.1.0
Products.PDBDebugMode = 1.3.1
Products.PloneTestCase = 0.9.18
Products.SecureMailHost = 1.1.2
Products.SiteErrorLog = 2.13.2
Products.ZSQLMethods = 2.13.4
PyYAML = 3.11
Pygments = 2.0.2
RestrictedPython = 3.6.0
Unidecode = 0.4.1
WebOb = 1.4
ZConfig = 3.0.4
ZODB = 4.2.0
argh = 0.26.1
argparse = 1.3.0
coverage = 3.7.1
cssmin = 0.2.0
cssselect = 0.9.1
decorator = 4.0.6
docutils = 0.12
feedparser = 5.2.1
five.globalrequest = 1.0
future = 0.15.2
interlude = 1.3.1
lxml = 3.5.0
mailinglogger = 3.8.0
manuel = 1.8.0
mechanize = 0.2.5
mock = 1.0.1
mocker = 1.1.1
pathtools = 0.1.2
persistent = 4.1.1
plone.mocktestcase = 1.0b3
ply = 3.4
python_dateutil = 1.5
python_gettext = 3.0
python_openid = 2.2.5
pytz = 2015.7
robotframework = 2.8.7
robotframework_debuglibrary = 0.3
robotframework_ride = 1.3
robotframework_selenium2library = 1.7.4
robotsuite = 1.7.0
roman = 1.4.0
selenium = 2.46.0
setuptools = 19.6.2
simplejson = 3.8.1
six = 1.9.0
slimit = 0.8.1
sourcecodegen = 0.6.14
transaction = 1.4.4
unittest2 = 0.8.0
watchdog = 0.8.3
wsgi_intercept = 0.4
z3c.caching = 2.0a1
z3c.coverage = 2.0.3
z3c.objpath = 1.1
z3c.pt = 3.0.0a1
z3c.template = 2.0.0
z3c.zcmlhook = 1.0b1
zc.buildout = 2.5.0
zc.lockfile = 1.1.0
zc.recipe.egg = 2.0.3
zc.relation = 1.0
zc.sourcefactory = 0.7.0
zdaemon = 4.1.0
zodbpickle = 0.6.0
zope.app.applicationcontrol = 3.5.10
zope.app.appsetup = 3.14.0
zope.app.authentication = 3.8.0
zope.app.basicskin = 3.5.1
zope.app.broken = 3.6.0
zope.app.component = 3.9.3
zope.app.container = 3.9.2
zope.app.content = 3.5.1
zope.app.debug = 3.4.1
zope.app.dependable = 3.5.1
zope.app.error = 3.5.3
zope.app.exception = 3.6.3
zope.app.file = 3.6.1
zope.app.folder = 3.5.2
zope.app.form = 4.0.2
zope.app.generations = 3.6.1
zope.app.http = 3.9.0
zope.app.i18n = 3.6.4
zope.app.intid = 3.7.1
zope.app.locales = 3.6.2
zope.app.localpermission = 3.7.2
zope.app.pagetemplate = 3.11.2
zope.app.principalannotation = 3.7.0
zope.app.publication = 3.12.0
zope.app.publisher = 3.10.2
zope.app.renderer = 3.5.1
zope.app.rotterdam = 3.5.3
zope.app.schema = 3.5.0
zope.app.security = 3.7.5
zope.app.testing = 3.7.8
zope.app.wsgi = 3.9.3
zope.app.zcmlfiles = 3.7.1
zope.app.zopeappgenerations = 3.5.1
zope.broken = 3.6.0
zope.sequencesort = 4.0.1
zope.testbrowser = 3.11.1
zope.untrustedpython = 4.0.0

[pbauer]

FWIW: I just tested a current Plone5-based project of ours with the versions/checkouts of this PLIP and found no issue. Needs more review though.

[mcdonc]

FWIW, if you're going to refactor ZPublisher a bit, you might take a look at the (very aged, but does it matter?) repoze.zope2, which attempted to do just that:

http://svn.repoze.org/repoze.zope2/trunk/

I started to write Pyramid shortly after this, as I felt that I was never going to be able to factor the publisher sufficiently while maintaining bw compat, so I figured "nuke it from orbit and start over" ;)

[thet]

@pbauer 👍 👍
most of the eggs listed above are only test dependencies.

@mcdonc tnx for the pointer! we will look at it. does repoze.zope2 use WebOb instead of Zope's Request/Response objects? That would be interesting too...

[mcdonc]

No it uses Zope request/response.

[thet]

Unmerged Pull Requests

#1370
zopefoundation/zope.publisher#10
zopefoundation/five.customerize#1

Merged Pull Requests

plone/plone.api#301
plone/plone.app.caching#21
plone/plone.app.contenttypes#317
plone/plone.app.customerize#8
plone/plone.app.dexterity#205
plone/plone.app.i18n#4
plone/plone.app.imaging#25
plone/plone.app.layout#77
plone/plone.app.lockingbehavior#5
plone/plone.app.testing#20
plone/plone.app.upgrade#66
plone/plone.app.users#53
plone/plone.app.z3cform#38
plone/plone.autoform#10
plone/plone.behavior#7
plone/plone.dexterity#50
plone/plone.formwidget.namedfile#17
plone/plone.indexer#3
plone/plone.memoize#3
plone/plone.registry#7
plone/plone.recipe.zope2instance#20
plone/plone.resource#9
plone/plone.rfc822#1
plone/plone.supermodel#13
plone/plone.testing#16
plone/plone.testing#18
plone/Products.Archetypes#51
plone/Products.ATContentTypes#30
plone/Products.PortalTransforms#13
zopefoundation/Zope#45
https://github.com/zopefoundation/Products.ZCatalog/tree/thet-zope4
zopefoundation/five.pt#6

[pbauer]

For now we need to skip the following tests since they fail using Zope4:

• plone.app.lockingbehavior.tests.test_locking.test_lockablebehavior
• plone.dexterity.tests.test_schema.test_portalTypeToSchemaName_looks_up_portal_for_prefix
• plone.dexterity.tests.test_utils.test_getAdditionalSchemata
• Products.CMFPlone.controlpanel.tests.test_controlpanel_browser_maintenance.test_maintenance_pack_database

Also this pull adds some Zope 4 specific case handling, which should be reviewed again:
https://github.com/plone/plone.supermodel/pull/13/files

[ebrehault]

This PLIP is approved for Plone 6, see https://community.plone.org/t/framework-team-meeting-minutes-2016-03-01/1663

[hvelarde]

awesome!