PLIP: Update Zope Dependencies #1351

Open
thet opened this Issue Jan 30, 2016 · 37 comments

Comments

@thet
Member

thet commented Jan 30, 2016

Proposer : Johannes Raggam

Seconder : Philip Bauer

Abstract

This PLIP is about updating Plone's framework dependencies (ZCA, ZTK and Zope) to the latest possible versions.

Motivation

Plone 5 still uses ages old ZTK dependencies and Zope2. While the ZTK dependencies are still actively developed and most of them already Python 3 compatible, the development for Zope 2 has nearly stopped. The successor of Zope2 is Zope 4. The major release version indicates that backwards incompatible refactoring has happened: long deprecated stuff was removed, some performance improvements and bugfixes were done, a few features were added.
To be able to move forward with Plone, we have to modernize our framework dependencies. Also, this PLIP is a precondition for Python 3 compatibility.

Assumptions

  • We are not able to switch to Pyramid / SubstanceD soon.
  • Modernizing our framework dependencies will help with such a possible switch in the future.
  • A recent set of ZTK packages is a precondition for Python 3 compatibility.
  • Patching and hacking the ZPublisher is better done with a cleaned up version.

Proposal & Implementation

PLIP configuration: https://github.com/plone/buildout.coredev/blob/5.0/plips/plip-zope4.cfg
Jenkins PLIP job: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/

The PLIP buildout configuration extends versions.cfg and sources.cfg from the Zope project. We added the option auto-checkout = * to check out all sources with mr.developer to get the very latest versions for each dependency. Only some test dependencies are installed as egg. For easier tracking, the eggs are installed in the plips/eggs folder.

We started to fix test failures and created branches and pull requests for all of our changes. Not everything is pull-requested yet. Many of the failures came from moved imports, removed functionality (e.g. no doctest suite any more in zope.testing, as we can use the stdlib for that) but also bugfixes in production code. E.g:

All fixes are made backwards compatible for our current set of dependencies.

At the Alpine City Sprint most of the failures could be fixed. There are only a few test failures left. Also, there are a lot of test isolation problems, but that's another story.

Two packages are not yet updated: Products.ZCatalog and ZODB 4. We still have to decide, if these packages are part of this PLIP. Updating them involves finding a solution for the "ZCatalog Regression: indexing None Values" problem: zopefoundation/Products.ZCatalog#5

We propose this PLIP for inclusion in Plone 6.

Deliverables

  • Test fixes.
  • Bug fixes.
  • Green Jenkins build.
  • Upgrade guide based on changes on packages within this PLIP.
  • Check monkey patches for their compatibility with Zope 4.

- [ ] Replace dependency on "Zope2" with "Zope". Eventually provide a Zope2 shim. - not necessary - Zope version 4.0 is still named Zope2.

Risks

Zope 4 is currently in alpha state, there is no release yet and development activity is - as with Zope2 - not very high.
We need to show more responsibility for Zope 2 and related packages.

Participants

Python 3

Python 3 compatibility is not part of the PLIP. A major showstopper for Python 3 compatibility is Products.RestrictedPython. @loechel worked on that during the Alpine City Sprint 2016, which resulted in this document: zopefoundation/RestrictedPython@45c50cb
Also, ZODB 4 is a precondition for Python 3 compatibility.

Notes

Some experiments to install Plone 5 on Zope 4 started back in 2013: https://dev.plone.org/ticket/13822
At the Plone conference 2015 I demoed a running installation of Plone 5 on Zope 4 with the package: https://github.com/collective/experimental.zope4 and started to fix some failing tests.
A major effort on working on this PLIP was done at the Alpine City Strategic Sprint 2016 in Innsbruck: http://www.coactivate.org/projects/alpinecitysprint-2016
Related to this PLIP: "WIP try ZODB3" plone/buildout.coredev#137

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Feb 1, 2016

Member

\o/ tests are finally green: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/16/console

There are some pull-request to be made for plone and zope-packages and at least one issue needs to be looked at (plone/plone.supermodel@8a4d202).

The plip-job uses source-checkouts for everything but the following:

BTrees = 4.2.0
Babel = 1.3
Chameleon = 2.24
FormEncode = 1.3.0
Markdown = 2.6.5
Pillow = 3.1.0
Products.PDBDebugMode = 1.3.1
Products.PloneTestCase = 0.9.18
Products.SecureMailHost = 1.1.2
Products.SiteErrorLog = 2.13.2
Products.ZSQLMethods = 2.13.4
PyYAML = 3.11
Pygments = 2.0.2
RestrictedPython = 3.6.0
Unidecode = 0.4.1
WebOb = 1.4
ZConfig = 3.0.4
ZODB = 4.2.0
argh = 0.26.1
argparse = 1.3.0
coverage = 3.7.1
cssmin = 0.2.0
cssselect = 0.9.1
decorator = 4.0.6
docutils = 0.12
feedparser = 5.2.1
five.globalrequest = 1.0
future = 0.15.2
interlude = 1.3.1
lxml = 3.5.0
mailinglogger = 3.8.0
manuel = 1.8.0
mechanize = 0.2.5
mock = 1.0.1
mocker = 1.1.1
pathtools = 0.1.2
persistent = 4.1.1
plone.mocktestcase = 1.0b3
ply = 3.4
python_dateutil = 1.5
python_gettext = 3.0
python_openid = 2.2.5
pytz = 2015.7
robotframework = 2.8.7
robotframework_debuglibrary = 0.3
robotframework_ride = 1.3
robotframework_selenium2library = 1.7.4
robotsuite = 1.7.0
roman = 1.4.0
selenium = 2.46.0
setuptools = 19.6.2
simplejson = 3.8.1
six = 1.9.0
slimit = 0.8.1
sourcecodegen = 0.6.14
transaction = 1.4.4
unittest2 = 0.8.0
watchdog = 0.8.3
wsgi_intercept = 0.4
z3c.caching = 2.0a1
z3c.coverage = 2.0.3
z3c.objpath = 1.1
z3c.pt = 3.0.0a1
z3c.template = 2.0.0
z3c.zcmlhook = 1.0b1
zc.buildout = 2.5.0
zc.lockfile = 1.1.0
zc.recipe.egg = 2.0.3
zc.relation = 1.0
zc.sourcefactory = 0.7.0
zdaemon = 4.1.0
zodbpickle = 0.6.0
zope.app.applicationcontrol = 3.5.10
zope.app.appsetup = 3.14.0
zope.app.authentication = 3.8.0
zope.app.basicskin = 3.5.1
zope.app.broken = 3.6.0
zope.app.component = 3.9.3
zope.app.container = 3.9.2
zope.app.content = 3.5.1
zope.app.debug = 3.4.1
zope.app.dependable = 3.5.1
zope.app.error = 3.5.3
zope.app.exception = 3.6.3
zope.app.file = 3.6.1
zope.app.folder = 3.5.2
zope.app.form = 4.0.2
zope.app.generations = 3.6.1
zope.app.http = 3.9.0
zope.app.i18n = 3.6.4
zope.app.intid = 3.7.1
zope.app.locales = 3.6.2
zope.app.localpermission = 3.7.2
zope.app.pagetemplate = 3.11.2
zope.app.principalannotation = 3.7.0
zope.app.publication = 3.12.0
zope.app.publisher = 3.10.2
zope.app.renderer = 3.5.1
zope.app.rotterdam = 3.5.3
zope.app.schema = 3.5.0
zope.app.security = 3.7.5
zope.app.testing = 3.7.8
zope.app.wsgi = 3.9.3
zope.app.zcmlfiles = 3.7.1
zope.app.zopeappgenerations = 3.5.1
zope.broken = 3.6.0
zope.sequencesort = 4.0.1
zope.testbrowser = 3.11.1
zope.untrustedpython = 4.0.0
Member

pbauer commented Feb 1, 2016

\o/ tests are finally green: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/16/console

There are some pull-request to be made for plone and zope-packages and at least one issue needs to be looked at (plone/plone.supermodel@8a4d202).

The plip-job uses source-checkouts for everything but the following:

BTrees = 4.2.0
Babel = 1.3
Chameleon = 2.24
FormEncode = 1.3.0
Markdown = 2.6.5
Pillow = 3.1.0
Products.PDBDebugMode = 1.3.1
Products.PloneTestCase = 0.9.18
Products.SecureMailHost = 1.1.2
Products.SiteErrorLog = 2.13.2
Products.ZSQLMethods = 2.13.4
PyYAML = 3.11
Pygments = 2.0.2
RestrictedPython = 3.6.0
Unidecode = 0.4.1
WebOb = 1.4
ZConfig = 3.0.4
ZODB = 4.2.0
argh = 0.26.1
argparse = 1.3.0
coverage = 3.7.1
cssmin = 0.2.0
cssselect = 0.9.1
decorator = 4.0.6
docutils = 0.12
feedparser = 5.2.1
five.globalrequest = 1.0
future = 0.15.2
interlude = 1.3.1
lxml = 3.5.0
mailinglogger = 3.8.0
manuel = 1.8.0
mechanize = 0.2.5
mock = 1.0.1
mocker = 1.1.1
pathtools = 0.1.2
persistent = 4.1.1
plone.mocktestcase = 1.0b3
ply = 3.4
python_dateutil = 1.5
python_gettext = 3.0
python_openid = 2.2.5
pytz = 2015.7
robotframework = 2.8.7
robotframework_debuglibrary = 0.3
robotframework_ride = 1.3
robotframework_selenium2library = 1.7.4
robotsuite = 1.7.0
roman = 1.4.0
selenium = 2.46.0
setuptools = 19.6.2
simplejson = 3.8.1
six = 1.9.0
slimit = 0.8.1
sourcecodegen = 0.6.14
transaction = 1.4.4
unittest2 = 0.8.0
watchdog = 0.8.3
wsgi_intercept = 0.4
z3c.caching = 2.0a1
z3c.coverage = 2.0.3
z3c.objpath = 1.1
z3c.pt = 3.0.0a1
z3c.template = 2.0.0
z3c.zcmlhook = 1.0b1
zc.buildout = 2.5.0
zc.lockfile = 1.1.0
zc.recipe.egg = 2.0.3
zc.relation = 1.0
zc.sourcefactory = 0.7.0
zdaemon = 4.1.0
zodbpickle = 0.6.0
zope.app.applicationcontrol = 3.5.10
zope.app.appsetup = 3.14.0
zope.app.authentication = 3.8.0
zope.app.basicskin = 3.5.1
zope.app.broken = 3.6.0
zope.app.component = 3.9.3
zope.app.container = 3.9.2
zope.app.content = 3.5.1
zope.app.debug = 3.4.1
zope.app.dependable = 3.5.1
zope.app.error = 3.5.3
zope.app.exception = 3.6.3
zope.app.file = 3.6.1
zope.app.folder = 3.5.2
zope.app.form = 4.0.2
zope.app.generations = 3.6.1
zope.app.http = 3.9.0
zope.app.i18n = 3.6.4
zope.app.intid = 3.7.1
zope.app.locales = 3.6.2
zope.app.localpermission = 3.7.2
zope.app.pagetemplate = 3.11.2
zope.app.principalannotation = 3.7.0
zope.app.publication = 3.12.0
zope.app.publisher = 3.10.2
zope.app.renderer = 3.5.1
zope.app.rotterdam = 3.5.3
zope.app.schema = 3.5.0
zope.app.security = 3.7.5
zope.app.testing = 3.7.8
zope.app.wsgi = 3.9.3
zope.app.zcmlfiles = 3.7.1
zope.app.zopeappgenerations = 3.5.1
zope.broken = 3.6.0
zope.sequencesort = 4.0.1
zope.testbrowser = 3.11.1
zope.untrustedpython = 4.0.0

@pbauer pbauer referenced this issue in zopefoundation/Zope Feb 1, 2016

Merged

Fix two issues #45

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Feb 1, 2016

Member

FWIW: I just tested a current Plone5-based project of ours with the versions/checkouts of this PLIP and found no issue. Needs more review though.

Member

pbauer commented Feb 1, 2016

FWIW: I just tested a current Plone5-based project of ours with the versions/checkouts of this PLIP and found no issue. Needs more review though.

@mcdonc

This comment has been minimized.

Show comment
Hide comment
@mcdonc

mcdonc Feb 1, 2016

FWIW, if you're going to refactor ZPublisher a bit, you might take a look at the (very aged, but does it matter?) repoze.zope2, which attempted to do just that:

http://svn.repoze.org/repoze.zope2/trunk/

I started to write Pyramid shortly after this, as I felt that I was never going to be able to factor the publisher sufficiently while maintaining bw compat, so I figured "nuke it from orbit and start over" ;)

mcdonc commented Feb 1, 2016

FWIW, if you're going to refactor ZPublisher a bit, you might take a look at the (very aged, but does it matter?) repoze.zope2, which attempted to do just that:

http://svn.repoze.org/repoze.zope2/trunk/

I started to write Pyramid shortly after this, as I felt that I was never going to be able to factor the publisher sufficiently while maintaining bw compat, so I figured "nuke it from orbit and start over" ;)

@thet

This comment has been minimized.

Show comment
Hide comment
@thet

thet Feb 1, 2016

Member

@pbauer 👍 👍
most of the eggs listed above are only test dependencies.

@mcdonc tnx for the pointer! we will look at it. does repoze.zope2 use WebOb instead of Zope's Request/Response objects? That would be interesting too...

Member

thet commented Feb 1, 2016

@pbauer 👍 👍
most of the eggs listed above are only test dependencies.

@mcdonc tnx for the pointer! we will look at it. does repoze.zope2 use WebOb instead of Zope's Request/Response objects? That would be interesting too...

@mcdonc

This comment has been minimized.

Show comment
Hide comment
@mcdonc

mcdonc Feb 1, 2016

No it uses Zope request/response.

mcdonc commented Feb 1, 2016

No it uses Zope request/response.

@thet
@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Feb 8, 2016

Member

For now we need to skip the following tests since they fail using Zope4:

  • plone.app.lockingbehavior.tests.test_locking.test_lockablebehavior
  • plone.dexterity.tests.test_schema.test_portalTypeToSchemaName_looks_up_portal_for_prefix
  • plone.dexterity.tests.test_utils.test_getAdditionalSchemata
  • Products.CMFPlone.controlpanel.tests.test_controlpanel_browser_maintenance.test_maintenance_pack_database

Also this pull adds some Zope 4 specific case handling, which should be reviewed again:
https://github.com/plone/plone.supermodel/pull/13/files

Member

pbauer commented Feb 8, 2016

For now we need to skip the following tests since they fail using Zope4:

  • plone.app.lockingbehavior.tests.test_locking.test_lockablebehavior
  • plone.dexterity.tests.test_schema.test_portalTypeToSchemaName_looks_up_portal_for_prefix
  • plone.dexterity.tests.test_utils.test_getAdditionalSchemata
  • Products.CMFPlone.controlpanel.tests.test_controlpanel_browser_maintenance.test_maintenance_pack_database

Also this pull adds some Zope 4 specific case handling, which should be reviewed again:
https://github.com/plone/plone.supermodel/pull/13/files

@ebrehault

This comment has been minimized.

Show comment
Hide comment
Member

ebrehault commented Mar 1, 2016

@hvelarde

This comment has been minimized.

Show comment
Hide comment
@hvelarde

hvelarde Mar 1, 2016

Member

awesome!

Member

hvelarde commented Mar 1, 2016

awesome!

@gforcada gforcada modified the milestones: Plone 6.0, Future Mar 1, 2016

@thet

This comment has been minimized.

Show comment
Hide comment
@thet

thet Mar 2, 2016

Member

Related:

Also:

Member

thet commented Mar 2, 2016

Related:

Also:

@pbauer pbauer changed the title from PLIP: Zope 4 / Update Dependencies to PLIP: Update Zope Dependencies Mar 31, 2016

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Mar 31, 2016

Member

I removed the Zope4-reference from the name of the PLIP since the version-bump will only add another fun layer of confusion that we struggled with since the release of the ill-named Zope3.

Member

pbauer commented Mar 31, 2016

I removed the Zope4-reference from the name of the PLIP since the version-bump will only add another fun layer of confusion that we struggled with since the release of the ill-named Zope3.

@thet

This comment has been minimized.

Show comment
Hide comment
@thet

thet Mar 31, 2016

Member

The plip configuration file is still named plip-zope4.cfg. Maybe it's better to keep it as it is, since we would need to update the jenkins PLIP job too.

Member

thet commented Mar 31, 2016

The plip configuration file is still named plip-zope4.cfg. Maybe it's better to keep it as it is, since we would need to update the jenkins PLIP job too.

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Apr 1, 2016

Member

Sure. That is way less visible than a PLIP-title.

Member

pbauer commented Apr 1, 2016

Sure. That is way less visible than a PLIP-title.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Jul 17, 2016

Member

I just spend a bit of time on ZCatalog. I merged the commit lingering on the https://github.com/zopefoundation/Products.ZCatalog/tree/thet-zope4 branch.

I also fixed the problem with zopefoundation/Products.ZCatalog#5 - by ignoring None values being indexed, rather than raising TypeErrors.

Both of these changes are in a ZCatalog 3.1.1 release. That release does depend on ZODB 4, so only use it if you want to do the ZODB 4 migration at the same time.

Member

hannosch commented Jul 17, 2016

I just spend a bit of time on ZCatalog. I merged the commit lingering on the https://github.com/zopefoundation/Products.ZCatalog/tree/thet-zope4 branch.

I also fixed the problem with zopefoundation/Products.ZCatalog#5 - by ignoring None values being indexed, rather than raising TypeErrors.

Both of these changes are in a ZCatalog 3.1.1 release. That release does depend on ZODB 4, so only use it if you want to do the ZODB 4 migration at the same time.

@thet

This comment has been minimized.

Show comment
Hide comment
@thet

thet Jul 17, 2016

Member

Thanks @hannosch ! That clears the way for the important ZODB updates...
Are there incompatibilities with ZODB 4? AFAIK the pickle support in Python 3 has changed, but ZODB still works transparent regarding to the pickle format and still supports Python 2 and 3.

Member

thet commented Jul 17, 2016

Thanks @hannosch ! That clears the way for the important ZODB updates...
Are there incompatibilities with ZODB 4? AFAIK the pickle support in Python 3 has changed, but ZODB still works transparent regarding to the pickle format and still supports Python 2 and 3.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Jul 17, 2016

Member

@thet sorry, got confused with Python 2 to 3 for a moment. IIRC ZODB 4 will need a manual migration when moving from an existing database created under Python 2 to code running under Python 3. But that should be the only one, and it's not relevant here, as Python 3 support is still a ways off.

Member

hannosch commented Jul 17, 2016

@thet sorry, got confused with Python 2 to 3 for a moment. IIRC ZODB 4 will need a manual migration when moving from an existing database created under Python 2 to code running under Python 3. But that should be the only one, and it's not relevant here, as Python 3 support is still a ways off.

@gforcada

This comment has been minimized.

Show comment
Hide comment
@gforcada

gforcada Jul 17, 2016

Contributor

@thet @hannosch there is https://pypi.io/project/zodb.py3migrate/ to convert ZODB databases to python 3

Contributor

gforcada commented Jul 17, 2016

@thet @hannosch there is https://pypi.io/project/zodb.py3migrate/ to convert ZODB databases to python 3

@thet

This comment has been minimized.

Show comment
Hide comment
@thet

thet Jul 17, 2016

Member

@hannosch fine! although, hopefully Python 3 will get relevant for Plone soon :)

Member

thet commented Jul 17, 2016

@hannosch fine! although, hopefully Python 3 will get relevant for Plone soon :)

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Jul 17, 2016

Member

What about ZODB 5? Did anyone review the changes that @jimfulton did and what they mean in the context of Plone? See https://github.com/zopefoundation/ZODB/blob/master/CHANGES.rst. Same goes for ZEO 5.

Member

pbauer commented Jul 17, 2016

What about ZODB 5? Did anyone review the changes that @jimfulton did and what they mean in the context of Plone? See https://github.com/zopefoundation/ZODB/blob/master/CHANGES.rst. Same goes for ZEO 5.

@jimfulton

This comment has been minimized.

Show comment
Hide comment
@jimfulton

jimfulton Jul 17, 2016

Note that ZODB 5 and ZEO 5 are still in alpha. ZODB 5 changes are mainly internal.

ZEO 5 is still evolving, but will provide some significant advantages, including SSL and prefetch.

I think it's a little early to worry about ZODB 5 and ZEO 5.

Also note that RelStorage is being actively developed again.

Note that ZODB 5 and ZEO 5 are still in alpha. ZODB 5 changes are mainly internal.

ZEO 5 is still evolving, but will provide some significant advantages, including SSL and prefetch.

I think it's a little early to worry about ZODB 5 and ZEO 5.

Also note that RelStorage is being actively developed again.

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Jul 20, 2016

Member

@jimfulton thank for the info. It's great to see that there is so much active development going on!

Member

pbauer commented Jul 20, 2016

@jimfulton thank for the info. It's great to see that there is so much active development going on!

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Jul 22, 2016

Member

I fixed some circular dependencies between ZCatalog, SiteErrorLog and Zope, which required new coordinated releases of all of them to run their tests.

So you got a Zope2 4.0a1 release out on PyPi now. You can use the new minimal versions-prod.cfg file from the tag at https://github.com/zopefoundation/Zope/blob/4.0a1/versions-prod.cfg or the versions.cfg file including some development tools.

And if you feel like it, even a pip installs works now, without relying on download.zope.org:

virtualenv --python=python2.7 zope
cd zope
bin/pip install -r https://raw.githubusercontent.com/zopefoundation/Zope/4.0a1/requirements.txt
bin/mkzopeinstance -d .

Start with either:

bin/runzope -C etc/zope.conf -X "debug-mode=on"

or

bin/runwsgi -v etc/zope.ini

The WSGI support is still not great, as it doesn't do exception handling, but leaves that to a WSGI middleware. If someone has actually deployed Zope2 under WSGI, please help out at zopefoundation/Zope#64.

Member

hannosch commented Jul 22, 2016

I fixed some circular dependencies between ZCatalog, SiteErrorLog and Zope, which required new coordinated releases of all of them to run their tests.

So you got a Zope2 4.0a1 release out on PyPi now. You can use the new minimal versions-prod.cfg file from the tag at https://github.com/zopefoundation/Zope/blob/4.0a1/versions-prod.cfg or the versions.cfg file including some development tools.

And if you feel like it, even a pip installs works now, without relying on download.zope.org:

virtualenv --python=python2.7 zope
cd zope
bin/pip install -r https://raw.githubusercontent.com/zopefoundation/Zope/4.0a1/requirements.txt
bin/mkzopeinstance -d .

Start with either:

bin/runzope -C etc/zope.conf -X "debug-mode=on"

or

bin/runwsgi -v etc/zope.ini

The WSGI support is still not great, as it doesn't do exception handling, but leaves that to a WSGI middleware. If someone has actually deployed Zope2 under WSGI, please help out at zopefoundation/Zope#64.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Sep 9, 2016

Member

I've gotten most of the refactorings I wanted to do for Zope 4 done, most importantly the split between a WSGI-only Zope core and an optional new ZServer project, which has all the non-WSGI parts in it.

There's a new Zope2 4.0a2 release out, major changes are documented over at http://zope.readthedocs.io/en/latest/WHATSNEW.html

I've probably screwed something up in all my work, so bug reports are appreciated. There are also probably new changes that have to be made at the CMF/PAS level in addition to the Plone level. A lot of the formerly required products like Sessions, TemporaryFolder, SiteAccess, VirtualHostMonster are now optional and no longer installed by default or their persistent objects set up automatically. Default functional test helpers from Testing/ZopeTestCase use the WSGI publisher, so there can be minor differences in the generated responses (e.g. a Date header is always present).

One other larger point of discussion is, whether or not Plone wants to continue using the ZServer based publisher or make the switch to WSGI. Switching means the instance configuration changes and all instance startup scripts change, since there's no longer a zopectl or zdaemon for the WSGI part, but those are limited to ZServer.

You probably want to start out by adding a new install_requires for ZServer, but I'm guessing even than some buildout recipes like zope2instance might need to be adjusted or new ways of configuring and running Plone need to be discussed.

The Zope default configuration uses the waitress WSGI server (same as Pyramid) as it is basically a cleaned up rewrite of ZServer and still based on asyncore. But Plone could decide to use a different WSGI server like gunicorn by default.

Going the WSGI route also means there's no longer support for webdav, FTP and xmlrpc, but on the plus side views can actually implement their own logic to handle HEAD or PUT requests.

Another rough corner is exception handling, as standard_error_message and the exceptionhook are no longer there in WSGI. Instead the Zope 4 WSGI publisher brings back support for exception views to influence how error responses get constructed.

Member

hannosch commented Sep 9, 2016

I've gotten most of the refactorings I wanted to do for Zope 4 done, most importantly the split between a WSGI-only Zope core and an optional new ZServer project, which has all the non-WSGI parts in it.

There's a new Zope2 4.0a2 release out, major changes are documented over at http://zope.readthedocs.io/en/latest/WHATSNEW.html

I've probably screwed something up in all my work, so bug reports are appreciated. There are also probably new changes that have to be made at the CMF/PAS level in addition to the Plone level. A lot of the formerly required products like Sessions, TemporaryFolder, SiteAccess, VirtualHostMonster are now optional and no longer installed by default or their persistent objects set up automatically. Default functional test helpers from Testing/ZopeTestCase use the WSGI publisher, so there can be minor differences in the generated responses (e.g. a Date header is always present).

One other larger point of discussion is, whether or not Plone wants to continue using the ZServer based publisher or make the switch to WSGI. Switching means the instance configuration changes and all instance startup scripts change, since there's no longer a zopectl or zdaemon for the WSGI part, but those are limited to ZServer.

You probably want to start out by adding a new install_requires for ZServer, but I'm guessing even than some buildout recipes like zope2instance might need to be adjusted or new ways of configuring and running Plone need to be discussed.

The Zope default configuration uses the waitress WSGI server (same as Pyramid) as it is basically a cleaned up rewrite of ZServer and still based on asyncore. But Plone could decide to use a different WSGI server like gunicorn by default.

Going the WSGI route also means there's no longer support for webdav, FTP and xmlrpc, but on the plus side views can actually implement their own logic to handle HEAD or PUT requests.

Another rough corner is exception handling, as standard_error_message and the exceptionhook are no longer there in WSGI. Instead the Zope 4 WSGI publisher brings back support for exception views to influence how error responses get constructed.

@tomgross

This comment has been minimized.

Show comment
Hide comment
@tomgross

tomgross Sep 9, 2016

Member

At least for webdav there is a middleware: https://github.com/mar10/wsgidav

Member

tomgross commented Sep 9, 2016

At least for webdav there is a middleware: https://github.com/mar10/wsgidav

@jimfulton

This comment has been minimized.

Show comment
Hide comment
@jimfulton

jimfulton Sep 9, 2016

On Fri, Sep 9, 2016 at 6:46 AM, Hanno Schlichting notifications@github.com
wrote:
...

One other larger point of discussion is, whether or not Plone wants to
continue using the ZServer based publisher or make the switch to WSGI.
Switching means the instance configuration changes and all instance startup
scripts change, since there's no longer a zopectl or zdaemon for the WSGI
part, but those are limited to ZServer.

I'm confused by the seeming assertion that zdaemon and ZServer are somehow
linked. (Maybe I'm misunderstanding you.) zdaemon is completely
independent and easily used with a variety applications. It's my go-to
tool for deploying daemons. (I prefer it to supervisord, because having a
daemon per service makes componentization of the automation of service
deployment simpler.) I've even used it to run docker containers. :)

...

Going the WSGI route also means there's no longer support for webdav, FTP
and xmlrpc, but on the plus side views can actually implement their own
logic to handle HEAD or PUT requests.

FWIW, I view the inclusion of FTP support in ZServer as an over
generalization on my part. Sorry. If FTP support is desired (probably not
;) ), I would write a back end to a Python FTP server. (We did this many
years ago at ZC for SFTP support.)

WebDAV and XMLRPC should be easy enough to provide in an WSGI context,
although I think they've both been overtaken by Rest (which has the
distinct benefit of not having an actual protocol that has to followed. ;)
).

Jim

Jim Fulton
http://jimfulton.info

On Fri, Sep 9, 2016 at 6:46 AM, Hanno Schlichting notifications@github.com
wrote:
...

One other larger point of discussion is, whether or not Plone wants to
continue using the ZServer based publisher or make the switch to WSGI.
Switching means the instance configuration changes and all instance startup
scripts change, since there's no longer a zopectl or zdaemon for the WSGI
part, but those are limited to ZServer.

I'm confused by the seeming assertion that zdaemon and ZServer are somehow
linked. (Maybe I'm misunderstanding you.) zdaemon is completely
independent and easily used with a variety applications. It's my go-to
tool for deploying daemons. (I prefer it to supervisord, because having a
daemon per service makes componentization of the automation of service
deployment simpler.) I've even used it to run docker containers. :)

...

Going the WSGI route also means there's no longer support for webdav, FTP
and xmlrpc, but on the plus side views can actually implement their own
logic to handle HEAD or PUT requests.

FWIW, I view the inclusion of FTP support in ZServer as an over
generalization on my part. Sorry. If FTP support is desired (probably not
;) ), I would write a back end to a Python FTP server. (We did this many
years ago at ZC for SFTP support.)

WebDAV and XMLRPC should be easy enough to provide in an WSGI context,
although I think they've both been overtaken by Rest (which has the
distinct benefit of not having an actual protocol that has to followed. ;)
).

Jim

Jim Fulton
http://jimfulton.info

@jimfulton

This comment has been minimized.

Show comment
Hide comment
@jimfulton

jimfulton Sep 9, 2016

Thanks for all your work on this!!!

(I should have started my earlier response with that. :) )

Jim

On Fri, Sep 9, 2016 at 6:46 AM, Hanno Schlichting notifications@github.com
wrote:

I've gotten most of the refactorings I wanted to do for Zope 4 done, most
importantly the split between a WSGI-only Zope core and an optional new
ZServer project, which has all the non-WSGI parts in it.

There's a new Zope2 4.0a2 release out, major changes are documented over
at http://zope.readthedocs.io/en/latest/WHATSNEW.html

I've probably screwed something up in all my work, so bug reports are
appreciated. There are also probably new changes that have to be made at
the CMF/PAS level in addition to the Plone level. A lot of the formerly
required products like Sessions, TemporaryFolder, SiteAccess,
VirtualHostMonster are now optional and no longer installed by default or
their persistent objects set up automatically. Default functional test
helpers from Testing/ZopeTestCase use the WSGI publisher, so there can be
minor differences in the generated responses (e.g. a Date header is always
present).

One other larger point of discussion is, whether or not Plone wants to
continue using the ZServer based publisher or make the switch to WSGI.
Switching means the instance configuration changes and all instance startup
scripts change, since there's no longer a zopectl or zdaemon for the WSGI
part, but those are limited to ZServer. You probably want to start out by
adding a new install_requires for ZServer, but I'm guessing even than some
buildout recipes like zope2instance might need to be adjusted or new ways
of configuring and running Plone need to be discussed. The Zope default
configuration uses the waitress WSGI server (same as Pyramid) as it is
basically a cleaned up rewrite of ZServer and still based on asyncore. But
Plone could decide to use a different WSGI server like gunicorn by default.
Going the WSGI route also means there's no longer support for webdav, FTP
and xmlrpc, but on the plus side views can actually implement their own
logic to handle HEAD or PUT requests. Another rough corner is exception
handling, as standard_error_message and the exceptionhook are no longer
there in WSGI. Instead the Zope 4 WSGI publisher brings back support for
exception views to influence how error responses get constructed.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1351 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAvKd4nas5z27a20UQcqSRoVxW827JWtks5qoTkigaJpZM4HPxVc
.

Jim Fulton
http://jimfulton.info

Thanks for all your work on this!!!

(I should have started my earlier response with that. :) )

Jim

On Fri, Sep 9, 2016 at 6:46 AM, Hanno Schlichting notifications@github.com
wrote:

I've gotten most of the refactorings I wanted to do for Zope 4 done, most
importantly the split between a WSGI-only Zope core and an optional new
ZServer project, which has all the non-WSGI parts in it.

There's a new Zope2 4.0a2 release out, major changes are documented over
at http://zope.readthedocs.io/en/latest/WHATSNEW.html

I've probably screwed something up in all my work, so bug reports are
appreciated. There are also probably new changes that have to be made at
the CMF/PAS level in addition to the Plone level. A lot of the formerly
required products like Sessions, TemporaryFolder, SiteAccess,
VirtualHostMonster are now optional and no longer installed by default or
their persistent objects set up automatically. Default functional test
helpers from Testing/ZopeTestCase use the WSGI publisher, so there can be
minor differences in the generated responses (e.g. a Date header is always
present).

One other larger point of discussion is, whether or not Plone wants to
continue using the ZServer based publisher or make the switch to WSGI.
Switching means the instance configuration changes and all instance startup
scripts change, since there's no longer a zopectl or zdaemon for the WSGI
part, but those are limited to ZServer. You probably want to start out by
adding a new install_requires for ZServer, but I'm guessing even than some
buildout recipes like zope2instance might need to be adjusted or new ways
of configuring and running Plone need to be discussed. The Zope default
configuration uses the waitress WSGI server (same as Pyramid) as it is
basically a cleaned up rewrite of ZServer and still based on asyncore. But
Plone could decide to use a different WSGI server like gunicorn by default.
Going the WSGI route also means there's no longer support for webdav, FTP
and xmlrpc, but on the plus side views can actually implement their own
logic to handle HEAD or PUT requests. Another rough corner is exception
handling, as standard_error_message and the exceptionhook are no longer
there in WSGI. Instead the Zope 4 WSGI publisher brings back support for
exception views to influence how error responses get constructed.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1351 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAvKd4nas5z27a20UQcqSRoVxW827JWtks5qoTkigaJpZM4HPxVc
.

Jim Fulton
http://jimfulton.info

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Sep 9, 2016

Member

I'm confused by the seeming assertion that zdaemon and ZServer are somehow
linked. (Maybe I'm misunderstanding you.)

My bad, what I'm calling ZServer here is not just "ZServer the package", but the new ZServer project I made, which includes the old ZServer package and a whole bunch of other stuff, including Zope instance creation logic, zope.conf parsing, zopectl support incl. it's zdaemon based start/stop/restart handling, the non-WSGI ZPublisher.Publish module, testing support for all of those and other packages like webdav, Lifetime and Signals.

It's maybe a bad name, since it ended up being a dumping ground for anything that wasn't necessary to run a WSGI based Zope instance - so it's more like OldZope (tm) :)

Especially the startup, configuration and testing code tie these pieces together, so I went with a single project for all of it, rather than trying to divide it up further.

WebDAV and XMLRPC should be easy enough to provide in an WSGI context,
although I think they've both been overtaken by Rest (which has the
distinct benefit of not having an actual protocol that has to followed. ;)

Indeed. My main goal was to make it possible to run Zope without all of these extra protocols, so applications like Plone can decide what they want to use. And hopefully porting to Python 3 becomes easier with fewer I/O boundaries to take care of.

Member

hannosch commented Sep 9, 2016

I'm confused by the seeming assertion that zdaemon and ZServer are somehow
linked. (Maybe I'm misunderstanding you.)

My bad, what I'm calling ZServer here is not just "ZServer the package", but the new ZServer project I made, which includes the old ZServer package and a whole bunch of other stuff, including Zope instance creation logic, zope.conf parsing, zopectl support incl. it's zdaemon based start/stop/restart handling, the non-WSGI ZPublisher.Publish module, testing support for all of those and other packages like webdav, Lifetime and Signals.

It's maybe a bad name, since it ended up being a dumping ground for anything that wasn't necessary to run a WSGI based Zope instance - so it's more like OldZope (tm) :)

Especially the startup, configuration and testing code tie these pieces together, so I went with a single project for all of it, rather than trying to divide it up further.

WebDAV and XMLRPC should be easy enough to provide in an WSGI context,
although I think they've both been overtaken by Rest (which has the
distinct benefit of not having an actual protocol that has to followed. ;)

Indeed. My main goal was to make it possible to run Zope without all of these extra protocols, so applications like Plone can decide what they want to use. And hopefully porting to Python 3 becomes easier with fewer I/O boundaries to take care of.

@Rudd-O

This comment has been minimized.

Show comment
Hide comment
@Rudd-O

Rudd-O Sep 30, 2016

Contributor

Breaks my heart to see Zope abandon WebDAV. Yes, I know it can still be gotten with ZServer... but, as time passes by and modules don't feel the need to support WebDAV anymore, less and less of what's Zope and Plone will work correctly via WebDAV, even with ZServer.

Contributor

Rudd-O commented Sep 30, 2016

Breaks my heart to see Zope abandon WebDAV. Yes, I know it can still be gotten with ZServer... but, as time passes by and modules don't feel the need to support WebDAV anymore, less and less of what's Zope and Plone will work correctly via WebDAV, even with ZServer.

@gforcada gforcada referenced this issue in plone/buildout.coredev Oct 5, 2016

Merged

update zope.component and zope.interface #256

jensens added a commit that referenced this issue Feb 1, 2017

@jensens jensens referenced this issue in plone/buildout.coredev Feb 1, 2017

Closed

use newer Chameleon 3.0 #298

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Feb 2, 2017

Member

The start of a upgrade-guide is here: https://docs.google.com/document/d/1T-E6qD7BoTrebk9L6F1EC0qMvqSOAu0987av6wUZ7nY/edit#

In there we collect everything that need to be documented and move it to their respective places after it looks complete.

Member

pbauer commented Feb 2, 2017

The start of a upgrade-guide is here: https://docs.google.com/document/d/1T-E6qD7BoTrebk9L6F1EC0qMvqSOAu0987av6wUZ7nY/edit#

In there we collect everything that need to be documented and move it to their respective places after it looks complete.

@ladplone ladplone modified the milestones: Plone 5.1, Plone 6.0 Feb 15, 2017

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Feb 19, 2017

Member

After much work was done during and after the Apline City Sprint (https://www.starzel.de/blog/towards-plone-6) the tests for this PLIP are now green (http://jenkins.plone.org/view/PLIPs/job/plip-zope4/). Remaining tasks will be defined and worked on at PLOG and at the the Zope 2 Resurrection Sprint (https://www.meetup.com/de-DE/Zope-Sprint/events/237672812/).

Member

pbauer commented Feb 19, 2017

After much work was done during and after the Apline City Sprint (https://www.starzel.de/blog/towards-plone-6) the tests for this PLIP are now green (http://jenkins.plone.org/view/PLIPs/job/plip-zope4/). Remaining tasks will be defined and worked on at PLOG and at the the Zope 2 Resurrection Sprint (https://www.meetup.com/de-DE/Zope-Sprint/events/237672812/).

@pbauer pbauer referenced this issue Feb 19, 2017

Closed

Compatibility with ZODB 4 and 5 #1845

2 of 7 tasks complete

@jamadden jamadden referenced this issue in zopefoundation/zope.app.basicskin Apr 21, 2017

Merged

Python 3 support. #1

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Sep 18, 2017

Member

Zope 4.0b1 has been released after the great work everyone did at the sprint in Halle last week! 🎉

I would love to get this PLIP into a state where we can be productively work on it during the conference-sprint.

One main issue at the moment is that our branch plonezope4 of Zope (zopefoundation/Zope#174) needs to be separated into pull-requests. @MrTango started working on this at the last sprint but it would be more productive if the authors of the relevant changes @thet, @davisagli, @mauritsvanrees, @MatthewWilkes, @MrTango had a look at their changes and created pull requests if they are still valid. There are only 13 commits of which some are already obsolete and I guess it is doable.

I'll buy nice and cold 🍺 for each of you once these commits are resolved.

Today I did some merging, cleanup and rebased most branches related to this PLIP. I also changed the PLIP-config to no longer checkout all packages but use released packages wherever possible. That saves a lot of time when running the buildout.

And now we can also look at the many new and exciting test-failures in http://jenkins.plone.org/view/PLIPs/job/plip-zope4/ 😉

Member

pbauer commented Sep 18, 2017

Zope 4.0b1 has been released after the great work everyone did at the sprint in Halle last week! 🎉

I would love to get this PLIP into a state where we can be productively work on it during the conference-sprint.

One main issue at the moment is that our branch plonezope4 of Zope (zopefoundation/Zope#174) needs to be separated into pull-requests. @MrTango started working on this at the last sprint but it would be more productive if the authors of the relevant changes @thet, @davisagli, @mauritsvanrees, @MatthewWilkes, @MrTango had a look at their changes and created pull requests if they are still valid. There are only 13 commits of which some are already obsolete and I guess it is doable.

I'll buy nice and cold 🍺 for each of you once these commits are resolved.

Today I did some merging, cleanup and rebased most branches related to this PLIP. I also changed the PLIP-config to no longer checkout all packages but use released packages wherever possible. That saves a lot of time when running the buildout.

And now we can also look at the many new and exciting test-failures in http://jenkins.plone.org/view/PLIPs/job/plip-zope4/ 😉

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Sep 19, 2017

Member

I put in some more work and fixed a good couple of things. Among the exciting new issues: Zope now uses new style classes, most tools from CMFCore are now utilities and brains cannot acquire the REQUEST.

Many of the remaining issues are related to plone/Products.PlonePAS#24 (anyone???) and zopefoundation/Products.ZCatalog#27

Member

pbauer commented Sep 19, 2017

I put in some more work and fixed a good couple of things. Among the exciting new issues: Zope now uses new style classes, most tools from CMFCore are now utilities and brains cannot acquire the REQUEST.

Many of the remaining issues are related to plone/Products.PlonePAS#24 (anyone???) and zopefoundation/Products.ZCatalog#27

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Sep 28, 2017

Member

I fixed the issues with PlonePAS (ugly but working for now: plone/Products.PlonePAS#25).

Now I see a lot of testbrowser-failures, many of which come from CSRF-protection where we get the confirm-action view instead of the expected URL. One rather simple such test is test_rename_form_cancel from plone.app.content. It passes for AT but fails for DX.

Can anyone give me hint about how to prevent that (except adding _authenticator to all the old tests)?

Member

pbauer commented Sep 28, 2017

I fixed the issues with PlonePAS (ugly but working for now: plone/Products.PlonePAS#25).

Now I see a lot of testbrowser-failures, many of which come from CSRF-protection where we get the confirm-action view instead of the expected URL. One rather simple such test is test_rename_form_cancel from plone.app.content. It passes for AT but fails for DX.

Can anyone give me hint about how to prevent that (except adding _authenticator to all the old tests)?

@mauritsvanrees

This comment has been minimized.

Show comment
Hide comment
@mauritsvanrees

mauritsvanrees Oct 3, 2017

Member

plone.protect has a suggestion, see the paragraph 'Fixing CSRF Protection failures in tests' on PyPI:

from plone.protect.authenticator import createToken
self.request = self.layer['request']
self.request.form['_authenticator'] = createToken()
# and maybe this:
self.request.environ['REQUEST_METHOD'] = 'POST'

That would require a change for each failing test, though hopefully in a shared setUp method this might solve several failures at once.

An alternative might be to set PLONE_CSRF_DISABLED=true in the environment of the OS. You could try that out first: export PLONE_CSRF_DISABLED=true; bin/test -s some.package and see if that helps. If it does, then you can unset the variable again, and temporarily set it in some tests with a contextmanager.

Possibly, such a contextmanager could be added to plone.protect or plone.testing or plone.app.testing so the code is shared.

Member

mauritsvanrees commented Oct 3, 2017

plone.protect has a suggestion, see the paragraph 'Fixing CSRF Protection failures in tests' on PyPI:

from plone.protect.authenticator import createToken
self.request = self.layer['request']
self.request.form['_authenticator'] = createToken()
# and maybe this:
self.request.environ['REQUEST_METHOD'] = 'POST'

That would require a change for each failing test, though hopefully in a shared setUp method this might solve several failures at once.

An alternative might be to set PLONE_CSRF_DISABLED=true in the environment of the OS. You could try that out first: export PLONE_CSRF_DISABLED=true; bin/test -s some.package and see if that helps. If it does, then you can unset the variable again, and temporarily set it in some tests with a contextmanager.

Possibly, such a contextmanager could be added to plone.protect or plone.testing or plone.app.testing so the code is shared.

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Oct 3, 2017

Member

@mauritsvanrees I'm aware of these options but to lazy to to that. What I wonder is which change might be responsible for these issues appearing now and if we can work around it there. It did not happen 5 month ago when we had all test green (with source-checouts of zope.testbrowser and plone.testing).

Member

pbauer commented Oct 3, 2017

@mauritsvanrees I'm aware of these options but to lazy to to that. What I wonder is which change might be responsible for these issues appearing now and if we can work around it there. It did not happen 5 month ago when we had all test green (with source-checouts of zope.testbrowser and plone.testing).

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Oct 3, 2017

Member

@pbauer I'd check if the code in https://github.com/plone/plone.transformchain/blob/master/plone/transformchain/zpublisher.py is still working as expected. Publication events are now emitted at a different place in the WSGI publisher. It should be more correct and fixes other issues. But maybe that's caused an issue for the transformchain code or something specific for testbrowser tests.

Member

hannosch commented Oct 3, 2017

@pbauer I'd check if the code in https://github.com/plone/plone.transformchain/blob/master/plone/transformchain/zpublisher.py is still working as expected. Publication events are now emitted at a different place in the WSGI publisher. It should be more correct and fixes other issues. But maybe that's caused an issue for the transformchain code or something specific for testbrowser tests.

@tschorr tschorr referenced this issue in plone/buildout.coredev Oct 21, 2017

Closed

move wsgi parts to core.cfg #377

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Oct 22, 2017

Member

All branches in here are merged. Thanks so much to everyone helped working on it!. We have still 5 failing tests left in the new 5.2-branch of the coredev, that we will fix in the next couple of days. What remains to be done is to:

Member

pbauer commented Oct 22, 2017

All branches in here are merged. Thanks so much to everyone helped working on it!. We have still 5 failing tests left in the new 5.2-branch of the coredev, that we will fix in the next couple of days. What remains to be done is to:

@jensens jensens moved this from In Process (approved) to Stalled (approved with no activity) in PLIPs Nov 14, 2017

@jensens jensens moved this from Stalled (approved with no activity) to Under Review (implemented) in PLIPs Nov 14, 2017

@jensens jensens moved this from Under Review (implemented) to Merged in PLIPs Nov 14, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment