Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PLIP: Update Zope Dependencies #1351

Closed
thet opened this issue Jan 30, 2016 · 38 comments
Closed

PLIP: Update Zope Dependencies #1351

thet opened this issue Jan 30, 2016 · 38 comments

Comments

@thet
Copy link
Member

@thet thet commented Jan 30, 2016

Proposer : Johannes Raggam

Seconder : Philip Bauer

Abstract

This PLIP is about updating Plone's framework dependencies (ZCA, ZTK and Zope) to the latest possible versions.

Motivation

Plone 5 still uses ages old ZTK dependencies and Zope2. While the ZTK dependencies are still actively developed and most of them already Python 3 compatible, the development for Zope 2 has nearly stopped. The successor of Zope2 is Zope 4. The major release version indicates that backwards incompatible refactoring has happened: long deprecated stuff was removed, some performance improvements and bugfixes were done, a few features were added.
To be able to move forward with Plone, we have to modernize our framework dependencies. Also, this PLIP is a precondition for Python 3 compatibility.

Assumptions

  • We are not able to switch to Pyramid / SubstanceD soon.
  • Modernizing our framework dependencies will help with such a possible switch in the future.
  • A recent set of ZTK packages is a precondition for Python 3 compatibility.
  • Patching and hacking the ZPublisher is better done with a cleaned up version.

Proposal & Implementation

PLIP configuration: https://github.com/plone/buildout.coredev/blob/5.0/plips/plip-zope4.cfg
Jenkins PLIP job: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/

The PLIP buildout configuration extends versions.cfg and sources.cfg from the Zope project. We added the option auto-checkout = * to check out all sources with mr.developer to get the very latest versions for each dependency. Only some test dependencies are installed as egg. For easier tracking, the eggs are installed in the plips/eggs folder.

We started to fix test failures and created branches and pull requests for all of our changes. Not everything is pull-requested yet. Many of the failures came from moved imports, removed functionality (e.g. no doctest suite any more in zope.testing, as we can use the stdlib for that) but also bugfixes in production code. E.g:

All fixes are made backwards compatible for our current set of dependencies.

At the Alpine City Sprint most of the failures could be fixed. There are only a few test failures left. Also, there are a lot of test isolation problems, but that's another story.

Two packages are not yet updated: Products.ZCatalog and ZODB 4. We still have to decide, if these packages are part of this PLIP. Updating them involves finding a solution for the "ZCatalog Regression: indexing None Values" problem: zopefoundation/Products.ZCatalog#5

We propose this PLIP for inclusion in Plone 6.

Deliverables

  • Test fixes.
  • Bug fixes.
  • Green Jenkins build.
  • Upgrade guide based on changes on packages within this PLIP.
  • Check monkey patches for their compatibility with Zope 4.

- [ ] Replace dependency on "Zope2" with "Zope". Eventually provide a Zope2 shim. - not necessary - Zope version 4.0 is still named Zope2.

Risks

Zope 4 is currently in alpha state, there is no release yet and development activity is - as with Zope2 - not very high.
We need to show more responsibility for Zope 2 and related packages.

Participants

Python 3

Python 3 compatibility is not part of the PLIP. A major showstopper for Python 3 compatibility is Products.RestrictedPython. @loechel worked on that during the Alpine City Sprint 2016, which resulted in this document: zopefoundation/RestrictedPython@45c50cb
Also, ZODB 4 is a precondition for Python 3 compatibility.

Notes

Some experiments to install Plone 5 on Zope 4 started back in 2013: https://dev.plone.org/ticket/13822
At the Plone conference 2015 I demoed a running installation of Plone 5 on Zope 4 with the package: https://github.com/collective/experimental.zope4 and started to fix some failing tests.
A major effort on working on this PLIP was done at the Alpine City Strategic Sprint 2016 in Innsbruck: http://www.coactivate.org/projects/alpinecitysprint-2016
Related to this PLIP: "WIP try ZODB3" plone/buildout.coredev#137

@pbauer
Copy link
Member

@pbauer pbauer commented Feb 1, 2016

\o/ tests are finally green: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/16/console

There are some pull-request to be made for plone and zope-packages and at least one issue needs to be looked at (plone/plone.supermodel@8a4d202).

The plip-job uses source-checkouts for everything but the following:

BTrees = 4.2.0
Babel = 1.3
Chameleon = 2.24
FormEncode = 1.3.0
Markdown = 2.6.5
Pillow = 3.1.0
Products.PDBDebugMode = 1.3.1
Products.PloneTestCase = 0.9.18
Products.SecureMailHost = 1.1.2
Products.SiteErrorLog = 2.13.2
Products.ZSQLMethods = 2.13.4
PyYAML = 3.11
Pygments = 2.0.2
RestrictedPython = 3.6.0
Unidecode = 0.4.1
WebOb = 1.4
ZConfig = 3.0.4
ZODB = 4.2.0
argh = 0.26.1
argparse = 1.3.0
coverage = 3.7.1
cssmin = 0.2.0
cssselect = 0.9.1
decorator = 4.0.6
docutils = 0.12
feedparser = 5.2.1
five.globalrequest = 1.0
future = 0.15.2
interlude = 1.3.1
lxml = 3.5.0
mailinglogger = 3.8.0
manuel = 1.8.0
mechanize = 0.2.5
mock = 1.0.1
mocker = 1.1.1
pathtools = 0.1.2
persistent = 4.1.1
plone.mocktestcase = 1.0b3
ply = 3.4
python_dateutil = 1.5
python_gettext = 3.0
python_openid = 2.2.5
pytz = 2015.7
robotframework = 2.8.7
robotframework_debuglibrary = 0.3
robotframework_ride = 1.3
robotframework_selenium2library = 1.7.4
robotsuite = 1.7.0
roman = 1.4.0
selenium = 2.46.0
setuptools = 19.6.2
simplejson = 3.8.1
six = 1.9.0
slimit = 0.8.1
sourcecodegen = 0.6.14
transaction = 1.4.4
unittest2 = 0.8.0
watchdog = 0.8.3
wsgi_intercept = 0.4
z3c.caching = 2.0a1
z3c.coverage = 2.0.3
z3c.objpath = 1.1
z3c.pt = 3.0.0a1
z3c.template = 2.0.0
z3c.zcmlhook = 1.0b1
zc.buildout = 2.5.0
zc.lockfile = 1.1.0
zc.recipe.egg = 2.0.3
zc.relation = 1.0
zc.sourcefactory = 0.7.0
zdaemon = 4.1.0
zodbpickle = 0.6.0
zope.app.applicationcontrol = 3.5.10
zope.app.appsetup = 3.14.0
zope.app.authentication = 3.8.0
zope.app.basicskin = 3.5.1
zope.app.broken = 3.6.0
zope.app.component = 3.9.3
zope.app.container = 3.9.2
zope.app.content = 3.5.1
zope.app.debug = 3.4.1
zope.app.dependable = 3.5.1
zope.app.error = 3.5.3
zope.app.exception = 3.6.3
zope.app.file = 3.6.1
zope.app.folder = 3.5.2
zope.app.form = 4.0.2
zope.app.generations = 3.6.1
zope.app.http = 3.9.0
zope.app.i18n = 3.6.4
zope.app.intid = 3.7.1
zope.app.locales = 3.6.2
zope.app.localpermission = 3.7.2
zope.app.pagetemplate = 3.11.2
zope.app.principalannotation = 3.7.0
zope.app.publication = 3.12.0
zope.app.publisher = 3.10.2
zope.app.renderer = 3.5.1
zope.app.rotterdam = 3.5.3
zope.app.schema = 3.5.0
zope.app.security = 3.7.5
zope.app.testing = 3.7.8
zope.app.wsgi = 3.9.3
zope.app.zcmlfiles = 3.7.1
zope.app.zopeappgenerations = 3.5.1
zope.broken = 3.6.0
zope.sequencesort = 4.0.1
zope.testbrowser = 3.11.1
zope.untrustedpython = 4.0.0
@pbauer
Copy link
Member

@pbauer pbauer commented Feb 1, 2016

FWIW: I just tested a current Plone5-based project of ours with the versions/checkouts of this PLIP and found no issue. Needs more review though.

@mcdonc
Copy link

@mcdonc mcdonc commented Feb 1, 2016

FWIW, if you're going to refactor ZPublisher a bit, you might take a look at the (very aged, but does it matter?) repoze.zope2, which attempted to do just that:

http://svn.repoze.org/repoze.zope2/trunk/

I started to write Pyramid shortly after this, as I felt that I was never going to be able to factor the publisher sufficiently while maintaining bw compat, so I figured "nuke it from orbit and start over" ;)

@thet
Copy link
Member Author

@thet thet commented Feb 1, 2016

@pbauer 👍 👍
most of the eggs listed above are only test dependencies.

@mcdonc tnx for the pointer! we will look at it. does repoze.zope2 use WebOb instead of Zope's Request/Response objects? That would be interesting too...

@mcdonc
Copy link

@mcdonc mcdonc commented Feb 1, 2016

No it uses Zope request/response.

@pbauer
Copy link
Member

@pbauer pbauer commented Feb 8, 2016

For now we need to skip the following tests since they fail using Zope4:

  • plone.app.lockingbehavior.tests.test_locking.test_lockablebehavior
  • plone.dexterity.tests.test_schema.test_portalTypeToSchemaName_looks_up_portal_for_prefix
  • plone.dexterity.tests.test_utils.test_getAdditionalSchemata
  • Products.CMFPlone.controlpanel.tests.test_controlpanel_browser_maintenance.test_maintenance_pack_database

Also this pull adds some Zope 4 specific case handling, which should be reviewed again:
https://github.com/plone/plone.supermodel/pull/13/files

@ebrehault
Copy link
Member

@ebrehault ebrehault commented Mar 1, 2016

@hvelarde
Copy link
Member

@hvelarde hvelarde commented Mar 1, 2016

awesome!

@gforcada gforcada modified the milestones: Plone 6.0, Future Mar 1, 2016
@thet
Copy link
Member Author

@thet thet commented Mar 2, 2016

Related:

Also:

@pbauer pbauer changed the title PLIP: Zope 4 / Update Dependencies PLIP: Update Zope Dependencies Mar 31, 2016
@pbauer
Copy link
Member

@pbauer pbauer commented Mar 31, 2016

I removed the Zope4-reference from the name of the PLIP since the version-bump will only add another fun layer of confusion that we struggled with since the release of the ill-named Zope3.

@thet
Copy link
Member Author

@thet thet commented Mar 31, 2016

The plip configuration file is still named plip-zope4.cfg. Maybe it's better to keep it as it is, since we would need to update the jenkins PLIP job too.

@pbauer
Copy link
Member

@pbauer pbauer commented Apr 1, 2016

Sure. That is way less visible than a PLIP-title.

@hannosch
Copy link
Member

@hannosch hannosch commented Jul 17, 2016

I just spend a bit of time on ZCatalog. I merged the commit lingering on the https://github.com/zopefoundation/Products.ZCatalog/tree/thet-zope4 branch.

I also fixed the problem with zopefoundation/Products.ZCatalog#5 - by ignoring None values being indexed, rather than raising TypeErrors.

Both of these changes are in a ZCatalog 3.1.1 release. That release does depend on ZODB 4, so only use it if you want to do the ZODB 4 migration at the same time.

@thet
Copy link
Member Author

@thet thet commented Jul 17, 2016

Thanks @hannosch ! That clears the way for the important ZODB updates...
Are there incompatibilities with ZODB 4? AFAIK the pickle support in Python 3 has changed, but ZODB still works transparent regarding to the pickle format and still supports Python 2 and 3.

@hannosch
Copy link
Member

@hannosch hannosch commented Jul 17, 2016

@thet sorry, got confused with Python 2 to 3 for a moment. IIRC ZODB 4 will need a manual migration when moving from an existing database created under Python 2 to code running under Python 3. But that should be the only one, and it's not relevant here, as Python 3 support is still a ways off.

@gforcada
Copy link
Contributor

@gforcada gforcada commented Jul 17, 2016

@thet @hannosch there is https://pypi.io/project/zodb.py3migrate/ to convert ZODB databases to python 3

@thet
Copy link
Member Author

@thet thet commented Jul 17, 2016

@hannosch fine! although, hopefully Python 3 will get relevant for Plone soon :)

@pbauer
Copy link
Member

@pbauer pbauer commented Jul 17, 2016

What about ZODB 5? Did anyone review the changes that @jimfulton did and what they mean in the context of Plone? See https://github.com/zopefoundation/ZODB/blob/master/CHANGES.rst. Same goes for ZEO 5.

@jimfulton
Copy link

@jimfulton jimfulton commented Jul 17, 2016

Note that ZODB 5 and ZEO 5 are still in alpha. ZODB 5 changes are mainly internal.

ZEO 5 is still evolving, but will provide some significant advantages, including SSL and prefetch.

I think it's a little early to worry about ZODB 5 and ZEO 5.

Also note that RelStorage is being actively developed again.

@hannosch
Copy link
Member

@hannosch hannosch commented Sep 9, 2016

I'm confused by the seeming assertion that zdaemon and ZServer are somehow
linked. (Maybe I'm misunderstanding you.)

My bad, what I'm calling ZServer here is not just "ZServer the package", but the new ZServer project I made, which includes the old ZServer package and a whole bunch of other stuff, including Zope instance creation logic, zope.conf parsing, zopectl support incl. it's zdaemon based start/stop/restart handling, the non-WSGI ZPublisher.Publish module, testing support for all of those and other packages like webdav, Lifetime and Signals.

It's maybe a bad name, since it ended up being a dumping ground for anything that wasn't necessary to run a WSGI based Zope instance - so it's more like OldZope (tm) :)

Especially the startup, configuration and testing code tie these pieces together, so I went with a single project for all of it, rather than trying to divide it up further.

WebDAV and XMLRPC should be easy enough to provide in an WSGI context,
although I think they've both been overtaken by Rest (which has the
distinct benefit of not having an actual protocol that has to followed. ;)

Indeed. My main goal was to make it possible to run Zope without all of these extra protocols, so applications like Plone can decide what they want to use. And hopefully porting to Python 3 becomes easier with fewer I/O boundaries to take care of.

@Rudd-O
Copy link
Contributor

@Rudd-O Rudd-O commented Sep 30, 2016

Breaks my heart to see Zope abandon WebDAV. Yes, I know it can still be gotten with ZServer... but, as time passes by and modules don't feel the need to support WebDAV anymore, less and less of what's Zope and Plone will work correctly via WebDAV, even with ZServer.

@pbauer
Copy link
Member

@pbauer pbauer commented Feb 2, 2017

The start of a upgrade-guide is here: https://docs.google.com/document/d/1T-E6qD7BoTrebk9L6F1EC0qMvqSOAu0987av6wUZ7nY/edit#

In there we collect everything that need to be documented and move it to their respective places after it looks complete.

@ladplone ladplone modified the milestones: Plone 5.1, Plone 6.0 Feb 15, 2017
@pbauer
Copy link
Member

@pbauer pbauer commented Feb 19, 2017

After much work was done during and after the Apline City Sprint (https://www.starzel.de/blog/towards-plone-6) the tests for this PLIP are now green (http://jenkins.plone.org/view/PLIPs/job/plip-zope4/). Remaining tasks will be defined and worked on at PLOG and at the the Zope 2 Resurrection Sprint (https://www.meetup.com/de-DE/Zope-Sprint/events/237672812/).

@pbauer
Copy link
Member

@pbauer pbauer commented Sep 18, 2017

Zope 4.0b1 has been released after the great work everyone did at the sprint in Halle last week! 🎉

I would love to get this PLIP into a state where we can be productively work on it during the conference-sprint.

One main issue at the moment is that our branch plonezope4 of Zope (zopefoundation/Zope#174) needs to be separated into pull-requests. @MrTango started working on this at the last sprint but it would be more productive if the authors of the relevant changes @thet, @davisagli, @mauritsvanrees, @MatthewWilkes, @MrTango had a look at their changes and created pull requests if they are still valid. There are only 13 commits of which some are already obsolete and I guess it is doable.

I'll buy nice and cold 🍺 for each of you once these commits are resolved.

Today I did some merging, cleanup and rebased most branches related to this PLIP. I also changed the PLIP-config to no longer checkout all packages but use released packages wherever possible. That saves a lot of time when running the buildout.

And now we can also look at the many new and exciting test-failures in http://jenkins.plone.org/view/PLIPs/job/plip-zope4/ 😉

@pbauer
Copy link
Member

@pbauer pbauer commented Sep 19, 2017

I put in some more work and fixed a good couple of things. Among the exciting new issues: Zope now uses new style classes, most tools from CMFCore are now utilities and brains cannot acquire the REQUEST.

Many of the remaining issues are related to plone/Products.PlonePAS#24 (anyone???) and zopefoundation/Products.ZCatalog#27

@pbauer
Copy link
Member

@pbauer pbauer commented Sep 28, 2017

I fixed the issues with PlonePAS (ugly but working for now: plone/Products.PlonePAS#25).

Now I see a lot of testbrowser-failures, many of which come from CSRF-protection where we get the confirm-action view instead of the expected URL. One rather simple such test is test_rename_form_cancel from plone.app.content. It passes for AT but fails for DX.

Can anyone give me hint about how to prevent that (except adding _authenticator to all the old tests)?

@mauritsvanrees
Copy link
Member

@mauritsvanrees mauritsvanrees commented Oct 3, 2017

plone.protect has a suggestion, see the paragraph 'Fixing CSRF Protection failures in tests' on PyPI:

from plone.protect.authenticator import createToken
self.request = self.layer['request']
self.request.form['_authenticator'] = createToken()
# and maybe this:
self.request.environ['REQUEST_METHOD'] = 'POST'

That would require a change for each failing test, though hopefully in a shared setUp method this might solve several failures at once.

An alternative might be to set PLONE_CSRF_DISABLED=true in the environment of the OS. You could try that out first: export PLONE_CSRF_DISABLED=true; bin/test -s some.package and see if that helps. If it does, then you can unset the variable again, and temporarily set it in some tests with a contextmanager.

Possibly, such a contextmanager could be added to plone.protect or plone.testing or plone.app.testing so the code is shared.

@pbauer
Copy link
Member

@pbauer pbauer commented Oct 3, 2017

@mauritsvanrees I'm aware of these options but to lazy to to that. What I wonder is which change might be responsible for these issues appearing now and if we can work around it there. It did not happen 5 month ago when we had all test green (with source-checouts of zope.testbrowser and plone.testing).

@hannosch
Copy link
Member

@hannosch hannosch commented Oct 3, 2017

@pbauer I'd check if the code in https://github.com/plone/plone.transformchain/blob/master/plone/transformchain/zpublisher.py is still working as expected. Publication events are now emitted at a different place in the WSGI publisher. It should be more correct and fixes other issues. But maybe that's caused an issue for the transformchain code or something specific for testbrowser tests.

@pbauer
Copy link
Member

@pbauer pbauer commented Oct 22, 2017

All branches in here are merged. Thanks so much to everyone helped working on it!. We have still 5 failing tests left in the new 5.2-branch of the coredev, that we will fix in the next couple of days. What remains to be done is to:

@jensens jensens moved this from In Process (approved) to Stalled (approved with no activity) in PLIPs Nov 14, 2017
@jensens jensens moved this from Stalled (approved with no activity) to Under Review (implemented) in PLIPs Nov 14, 2017
@jensens jensens moved this from Under Review (implemented) to Merged in PLIPs Nov 14, 2017
@jensens
Copy link
Member

@jensens jensens commented Sep 27, 2018

I think except documentation this can be seen as done. Documentation has its own ticket #1940
So, I close this!

@jensens jensens closed this Sep 27, 2018
@jensens jensens moved this from In progress to Done in Zope 4 Tasks Sep 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
PLIPs
Merged
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.