Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PLIP: Update Zope Dependencies #1351

4 of 5 tasks
thet opened this issue Jan 30, 2016 · 38 comments
4 of 5 tasks

PLIP: Update Zope Dependencies #1351

thet opened this issue Jan 30, 2016 · 38 comments


Copy link

thet commented Jan 30, 2016

Proposer : Johannes Raggam

Seconder : Philip Bauer


This PLIP is about updating Plone's framework dependencies (ZCA, ZTK and Zope) to the latest possible versions.


Plone 5 still uses ages old ZTK dependencies and Zope2. While the ZTK dependencies are still actively developed and most of them already Python 3 compatible, the development for Zope 2 has nearly stopped. The successor of Zope2 is Zope 4. The major release version indicates that backwards incompatible refactoring has happened: long deprecated stuff was removed, some performance improvements and bugfixes were done, a few features were added.
To be able to move forward with Plone, we have to modernize our framework dependencies. Also, this PLIP is a precondition for Python 3 compatibility.


  • We are not able to switch to Pyramid / SubstanceD soon.
  • Modernizing our framework dependencies will help with such a possible switch in the future.
  • A recent set of ZTK packages is a precondition for Python 3 compatibility.
  • Patching and hacking the ZPublisher is better done with a cleaned up version.

Proposal & Implementation

PLIP configuration:
Jenkins PLIP job:

The PLIP buildout configuration extends versions.cfg and sources.cfg from the Zope project. We added the option auto-checkout = * to check out all sources with mr.developer to get the very latest versions for each dependency. Only some test dependencies are installed as egg. For easier tracking, the eggs are installed in the plips/eggs folder.

We started to fix test failures and created branches and pull requests for all of our changes. Not everything is pull-requested yet. Many of the failures came from moved imports, removed functionality (e.g. no doctest suite any more in zope.testing, as we can use the stdlib for that) but also bugfixes in production code. E.g:

All fixes are made backwards compatible for our current set of dependencies.

At the Alpine City Sprint most of the failures could be fixed. There are only a few test failures left. Also, there are a lot of test isolation problems, but that's another story.

Two packages are not yet updated: Products.ZCatalog and ZODB 4. We still have to decide, if these packages are part of this PLIP. Updating them involves finding a solution for the "ZCatalog Regression: indexing None Values" problem: zopefoundation/Products.ZCatalog#5

We propose this PLIP for inclusion in Plone 6.


  • Test fixes.
  • Bug fixes.
  • Green Jenkins build.
  • Upgrade guide based on changes on packages within this PLIP.
  • Check monkey patches for their compatibility with Zope 4.

- [ ] Replace dependency on "Zope2" with "Zope". Eventually provide a Zope2 shim. - not necessary - Zope version 4.0 is still named Zope2.


Zope 4 is currently in alpha state, there is no release yet and development activity is - as with Zope2 - not very high.
We need to show more responsibility for Zope 2 and related packages.


Python 3

Python 3 compatibility is not part of the PLIP. A major showstopper for Python 3 compatibility is Products.RestrictedPython. @loechel worked on that during the Alpine City Sprint 2016, which resulted in this document: zopefoundation/RestrictedPython@45c50cb
Also, ZODB 4 is a precondition for Python 3 compatibility.


Some experiments to install Plone 5 on Zope 4 started back in 2013:
At the Plone conference 2015 I demoed a running installation of Plone 5 on Zope 4 with the package: and started to fix some failing tests.
A major effort on working on this PLIP was done at the Alpine City Strategic Sprint 2016 in Innsbruck:
Related to this PLIP: "WIP try ZODB3" plone/buildout.coredev#137

Copy link

pbauer commented Feb 1, 2016

\o/ tests are finally green:

There are some pull-request to be made for plone and zope-packages and at least one issue needs to be looked at (plone/plone.supermodel@8a4d202).

The plip-job uses source-checkouts for everything but the following:

BTrees = 4.2.0
Babel = 1.3
Chameleon = 2.24
FormEncode = 1.3.0
Markdown = 2.6.5
Pillow = 3.1.0
Products.PDBDebugMode = 1.3.1
Products.PloneTestCase = 0.9.18
Products.SecureMailHost = 1.1.2
Products.SiteErrorLog = 2.13.2
Products.ZSQLMethods = 2.13.4
PyYAML = 3.11
Pygments = 2.0.2
RestrictedPython = 3.6.0
Unidecode = 0.4.1
WebOb = 1.4
ZConfig = 3.0.4
ZODB = 4.2.0
argh = 0.26.1
argparse = 1.3.0
coverage = 3.7.1
cssmin = 0.2.0
cssselect = 0.9.1
decorator = 4.0.6
docutils = 0.12
feedparser = 5.2.1
five.globalrequest = 1.0
future = 0.15.2
interlude = 1.3.1
lxml = 3.5.0
mailinglogger = 3.8.0
manuel = 1.8.0
mechanize = 0.2.5
mock = 1.0.1
mocker = 1.1.1
pathtools = 0.1.2
persistent = 4.1.1
plone.mocktestcase = 1.0b3
ply = 3.4
python_dateutil = 1.5
python_gettext = 3.0
python_openid = 2.2.5
pytz = 2015.7
robotframework = 2.8.7
robotframework_debuglibrary = 0.3
robotframework_ride = 1.3
robotframework_selenium2library = 1.7.4
robotsuite = 1.7.0
roman = 1.4.0
selenium = 2.46.0
setuptools = 19.6.2
simplejson = 3.8.1
six = 1.9.0
slimit = 0.8.1
sourcecodegen = 0.6.14
transaction = 1.4.4
unittest2 = 0.8.0
watchdog = 0.8.3
wsgi_intercept = 0.4
z3c.caching = 2.0a1
z3c.coverage = 2.0.3
z3c.objpath = 1.1 = 3.0.0a1
z3c.template = 2.0.0
z3c.zcmlhook = 1.0b1
zc.buildout = 2.5.0
zc.lockfile = 1.1.0
zc.recipe.egg = 2.0.3
zc.relation = 1.0
zc.sourcefactory = 0.7.0
zdaemon = 4.1.0
zodbpickle = 0.6.0 = 3.5.10 = 3.14.0 = 3.8.0 = 3.5.1 = 3.6.0 = 3.9.3 = 3.9.2 = 3.5.1 = 3.4.1 = 3.5.1 = 3.5.3 = 3.6.3 = 3.6.1 = 3.5.2 = 4.0.2 = 3.6.1 = 3.9.0 = 3.6.4 = 3.7.1 = 3.6.2 = 3.7.2 = 3.11.2 = 3.7.0 = 3.12.0 = 3.10.2 = 3.5.1 = 3.5.3 = 3.5.0 = 3.7.5 = 3.7.8 = 3.9.3 = 3.7.1 = 3.5.1
zope.broken = 3.6.0
zope.sequencesort = 4.0.1
zope.testbrowser = 3.11.1
zope.untrustedpython = 4.0.0

Copy link

pbauer commented Feb 1, 2016

FWIW: I just tested a current Plone5-based project of ours with the versions/checkouts of this PLIP and found no issue. Needs more review though.

Copy link

mcdonc commented Feb 1, 2016

FWIW, if you're going to refactor ZPublisher a bit, you might take a look at the (very aged, but does it matter?) repoze.zope2, which attempted to do just that:

I started to write Pyramid shortly after this, as I felt that I was never going to be able to factor the publisher sufficiently while maintaining bw compat, so I figured "nuke it from orbit and start over" ;)

Copy link
Member Author

thet commented Feb 1, 2016

@pbauer 👍 👍
most of the eggs listed above are only test dependencies.

@mcdonc tnx for the pointer! we will look at it. does repoze.zope2 use WebOb instead of Zope's Request/Response objects? That would be interesting too...

Copy link

mcdonc commented Feb 1, 2016

No it uses Zope request/response.

Copy link

pbauer commented Feb 8, 2016

For now we need to skip the following tests since they fail using Zope4:

  • plone.dexterity.tests.test_schema.test_portalTypeToSchemaName_looks_up_portal_for_prefix
  • plone.dexterity.tests.test_utils.test_getAdditionalSchemata
  • Products.CMFPlone.controlpanel.tests.test_controlpanel_browser_maintenance.test_maintenance_pack_database

Also this pull adds some Zope 4 specific case handling, which should be reviewed again:

Copy link

This PLIP is approved for Plone 6, see

Copy link

hvelarde commented Mar 1, 2016


@gforcada gforcada modified the milestones: Plone 6.0, Future Mar 1, 2016
Copy link
Member Author

thet commented Mar 2, 2016



@pbauer pbauer changed the title PLIP: Zope 4 / Update Dependencies PLIP: Update Zope Dependencies Mar 31, 2016
Copy link

pbauer commented Mar 31, 2016

I removed the Zope4-reference from the name of the PLIP since the version-bump will only add another fun layer of confusion that we struggled with since the release of the ill-named Zope3.

Copy link
Member Author

thet commented Mar 31, 2016

The plip configuration file is still named plip-zope4.cfg. Maybe it's better to keep it as it is, since we would need to update the jenkins PLIP job too.

Copy link

pbauer commented Apr 1, 2016

Sure. That is way less visible than a PLIP-title.

Copy link

I just spend a bit of time on ZCatalog. I merged the commit lingering on the branch.

I also fixed the problem with zopefoundation/Products.ZCatalog#5 - by ignoring None values being indexed, rather than raising TypeErrors.

Both of these changes are in a ZCatalog 3.1.1 release. That release does depend on ZODB 4, so only use it if you want to do the ZODB 4 migration at the same time.

Copy link
Member Author

thet commented Jul 17, 2016

Thanks @hannosch ! That clears the way for the important ZODB updates...
Are there incompatibilities with ZODB 4? AFAIK the pickle support in Python 3 has changed, but ZODB still works transparent regarding to the pickle format and still supports Python 2 and 3.

Copy link

@thet sorry, got confused with Python 2 to 3 for a moment. IIRC ZODB 4 will need a manual migration when moving from an existing database created under Python 2 to code running under Python 3. But that should be the only one, and it's not relevant here, as Python 3 support is still a ways off.

Copy link

@thet @hannosch there is to convert ZODB databases to python 3

Copy link
Member Author

thet commented Jul 17, 2016

@hannosch fine! although, hopefully Python 3 will get relevant for Plone soon :)

Copy link

pbauer commented Jul 17, 2016

What about ZODB 5? Did anyone review the changes that @jimfulton did and what they mean in the context of Plone? See Same goes for ZEO 5.

Copy link

Note that ZODB 5 and ZEO 5 are still in alpha. ZODB 5 changes are mainly internal.

ZEO 5 is still evolving, but will provide some significant advantages, including SSL and prefetch.

I think it's a little early to worry about ZODB 5 and ZEO 5.

Also note that RelStorage is being actively developed again.

Copy link

hannosch commented Sep 9, 2016

I'm confused by the seeming assertion that zdaemon and ZServer are somehow
linked. (Maybe I'm misunderstanding you.)

My bad, what I'm calling ZServer here is not just "ZServer the package", but the new ZServer project I made, which includes the old ZServer package and a whole bunch of other stuff, including Zope instance creation logic, zope.conf parsing, zopectl support incl. it's zdaemon based start/stop/restart handling, the non-WSGI ZPublisher.Publish module, testing support for all of those and other packages like webdav, Lifetime and Signals.

It's maybe a bad name, since it ended up being a dumping ground for anything that wasn't necessary to run a WSGI based Zope instance - so it's more like OldZope (tm) :)

Especially the startup, configuration and testing code tie these pieces together, so I went with a single project for all of it, rather than trying to divide it up further.

WebDAV and XMLRPC should be easy enough to provide in an WSGI context,
although I think they've both been overtaken by Rest (which has the
distinct benefit of not having an actual protocol that has to followed. ;)

Indeed. My main goal was to make it possible to run Zope without all of these extra protocols, so applications like Plone can decide what they want to use. And hopefully porting to Python 3 becomes easier with fewer I/O boundaries to take care of.

Copy link

Rudd-O commented Sep 30, 2016

Breaks my heart to see Zope abandon WebDAV. Yes, I know it can still be gotten with ZServer... but, as time passes by and modules don't feel the need to support WebDAV anymore, less and less of what's Zope and Plone will work correctly via WebDAV, even with ZServer.

Copy link

pbauer commented Feb 2, 2017

The start of a upgrade-guide is here:

In there we collect everything that need to be documented and move it to their respective places after it looks complete.

@ladplone ladplone modified the milestones: Plone 5.1, Plone 6.0 Feb 15, 2017
Copy link

pbauer commented Feb 19, 2017

After much work was done during and after the Apline City Sprint ( the tests for this PLIP are now green ( Remaining tasks will be defined and worked on at PLOG and at the the Zope 2 Resurrection Sprint (

Copy link

pbauer commented Sep 18, 2017

Zope 4.0b1 has been released after the great work everyone did at the sprint in Halle last week! 🎉

I would love to get this PLIP into a state where we can be productively work on it during the conference-sprint.

One main issue at the moment is that our branch plonezope4 of Zope (zopefoundation/Zope#174) needs to be separated into pull-requests. @MrTango started working on this at the last sprint but it would be more productive if the authors of the relevant changes @thet, @davisagli, @mauritsvanrees, @MatthewWilkes, @MrTango had a look at their changes and created pull requests if they are still valid. There are only 13 commits of which some are already obsolete and I guess it is doable.

I'll buy nice and cold 🍺 for each of you once these commits are resolved.

Today I did some merging, cleanup and rebased most branches related to this PLIP. I also changed the PLIP-config to no longer checkout all packages but use released packages wherever possible. That saves a lot of time when running the buildout.

And now we can also look at the many new and exciting test-failures in 😉

Copy link

pbauer commented Sep 19, 2017

I put in some more work and fixed a good couple of things. Among the exciting new issues: Zope now uses new style classes, most tools from CMFCore are now utilities and brains cannot acquire the REQUEST.

Many of the remaining issues are related to plone/Products.PlonePAS#24 (anyone???) and zopefoundation/Products.ZCatalog#27

Copy link

pbauer commented Sep 28, 2017

I fixed the issues with PlonePAS (ugly but working for now: plone/Products.PlonePAS#25).

Now I see a lot of testbrowser-failures, many of which come from CSRF-protection where we get the confirm-action view instead of the expected URL. One rather simple such test is test_rename_form_cancel from It passes for AT but fails for DX.

Can anyone give me hint about how to prevent that (except adding _authenticator to all the old tests)?

Copy link

plone.protect has a suggestion, see the paragraph 'Fixing CSRF Protection failures in tests' on PyPI:

from plone.protect.authenticator import createToken
self.request = self.layer['request']
self.request.form['_authenticator'] = createToken()
# and maybe this:
self.request.environ['REQUEST_METHOD'] = 'POST'

That would require a change for each failing test, though hopefully in a shared setUp method this might solve several failures at once.

An alternative might be to set PLONE_CSRF_DISABLED=true in the environment of the OS. You could try that out first: export PLONE_CSRF_DISABLED=true; bin/test -s some.package and see if that helps. If it does, then you can unset the variable again, and temporarily set it in some tests with a contextmanager.

Possibly, such a contextmanager could be added to plone.protect or plone.testing or so the code is shared.

Copy link

pbauer commented Oct 3, 2017

@mauritsvanrees I'm aware of these options but to lazy to to that. What I wonder is which change might be responsible for these issues appearing now and if we can work around it there. It did not happen 5 month ago when we had all test green (with source-checouts of zope.testbrowser and plone.testing).

Copy link

hannosch commented Oct 3, 2017

@pbauer I'd check if the code in is still working as expected. Publication events are now emitted at a different place in the WSGI publisher. It should be more correct and fixes other issues. But maybe that's caused an issue for the transformchain code or something specific for testbrowser tests.

Copy link

pbauer commented Oct 22, 2017

All branches in here are merged. Thanks so much to everyone helped working on it!. We have still 5 failing tests left in the new 5.2-branch of the coredev, that we will fix in the next couple of days. What remains to be done is to:

@jensens jensens moved this from In Process (approved) to Stalled (approved with no activity) in PLIPs Nov 14, 2017
@jensens jensens moved this from Stalled (approved with no activity) to Under Review (implemented) in PLIPs Nov 14, 2017
@jensens jensens moved this from Under Review (implemented) to Merged in PLIPs Nov 14, 2017
Copy link

jensens commented Sep 27, 2018

I think except documentation this can be seen as done. Documentation has its own ticket #1940
So, I close this!

@jensens jensens closed this as completed Sep 27, 2018
@jensens jensens moved this from In progress to Done in Zope 4 Tasks Sep 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

No branches or pull requests