Python 3 compatibility

[tomgross]

According to caniusepython3 the following packages need some love to get Python 3 support:

• zExceptions
• initgroups -> only one C-File with no tests. Compiles using Python 3.5
• MultiMapping -> Python 3 support in master, needs a release
• tempstorage -> Python 3 support in master, needs a release
• zLOG
• Products.OFSP
• Products.ZCTextIndex
• nt_svcutils
• ExtensionClass (was missing Python 3 port of the C extension)
• Acquisition (missing port of the C extension to Python 3
• Persistence (which is blocking AccessControl, which is blocking DocumentTemplate)
• RestrictedPython (which is blocking AccessControl, which is blocking DocumentTemplate)
• AccessControl
• DocumentTemplate (Port to Python 3 DocumentTemplate#4)
• sourcecodegen (see Remove sourcecodegen dependency #71)
• zope.testbrowser
• five.globalrequest
• zope.globalrequest

Optional dependencies:

• Products.ZCatalog
• Products.BTreeFolder2
• Products.MailHost

Zope itself:

• Zope

[tseaver]

I updated the list with correct capitalizations and links.

[icemac]

Updated the list with four dependencies that were missing.

[hannosch]

initgroups turned out to be available in Python's os module in Python 2.7, so I dropped the dependency and made a new initgroups release that is merely a shim for os.initgroups.

For nt_svcutils, I removed the code from zopectl that used this. This was only needed to install and manage a Windows system service. For daemonizing someone can use supervisor or any other tool that provides that.

I also made some progress on ZCTextIndex, by removing its C code and replacing it with pure Python. The speed advantage shouldn't matter that much anymore, and if someone really cares about full text indexing, they'll likely use Solr or Elasticsearch these days.

Finally for DocumentTemplate, I have a branch to replace the C code with pure Python as well, but am not quite finished with that yet (zopefoundation/DocumentTemplate#2).

[hannosch]

I merged the code from Products.OFSP got back into Zope2, as there wasn't much left there. So it'll be done as part of Zope2 itself.

[hannosch]

I've also started to try and break out ZServer and make it an optional dependency. I think it should be possible to create a new ZServer distribution and move the ZServer, Lifetime and webdav packages into it. That new distribution should depend on Zope2, but not the other way around. There's still quite some work to do for that, but it should be possible. With that split, we could make Zope2 Python 3 compatible while using the WSGI publisher, without having to port ZServer at first or ever.

[leorochael]

It should be possible to cut out ZServer from Zope2 without making ZServer depend on Zope2 at all, by moving all the parts that import from ZPublisher into ZPublisher itself, turning ZPublisher into a pure WSGI app and turning ZServer into a pure WSGI HTTP Server.

I expand on this subject here.

[dwt]

I propose linking the respective tracking bugs for python3 compatibility in each project instead of just the projects. Anybody against this?

[icemac]

@dwt +1

[hannosch]

As a current status, we have a couple of issues / branches / PR's in progress and some of those can be worked on in parallel:

• Port to Python 3 RestrictedPython#12 (highest priority, as this is blocking/hindering progress on others)
• python3 compatibility AccessControl#21 (blocked in parts on RestrictedPython, but more progress might be possible, C extension port has started, but more progress can be made)
• Python 3 port DocumentTemplate#6 (porting started, next up are str/unicode issues, also blocked in parts on AccessControl/RestrictedPython)
• Port the C extension to Python 3? ExtensionClass#7 (C code was ported but some minor issues where found, needs more test cases in ExtensionClass itself)
• Py3 C extension port Persistence#3 (this might be done, but needs proper review of the C code)
• Port C extension to Python 3 Acquisition#12 (stefan-hof started work on porting the C extension, might be best to wait on him to push a first public version of that work)

Once all of these are done, work can begin on Zope2 itself. It might be possible to use all the WIP branches to start, but it might not. Once Zope2 is done, work can start on the various Products.* being dependent on Zope2.

[Daetalus]

Hi @hannosch

Just FYI, I try to let Pyston support Zope. But they didn't support the compiler package, which used by RestrictedPython. After some investigation. I try to use ast to replace compiler in RestrictedPython.

I start it by reading the source code of RestrictedPython. And then compare the ast before and after the MutatingWalker. Then use ast.NodeTransformer to do the same things. It basically a "reverse engineering". But the corner cases took me lots of time. More information is here: https://lab.nexedi.com/Daetalus/RestrictedPython-3.6.0/blob/c4652cb365edfe21e41c20bdf51c92dfab7a283f/src/RestrictedPython/notes.txt

It just a prototype, but in theory, it could work both Python 2/3 and Pyston. I think it's better to let you and other guys to know there already have some efforts to let RestrictedPython to support Python 3. Where to go on next is depend on you. (Just in case, I will resend this message to Zope dev mailist again).

Thanks all!

[hannosch]

@Daetalus Can you put this pointer into the RestrictedPython issue at zopefoundation/RestrictedPython#12. There's a group of people who have been working on a branch there for the past 12 months using the same approach you took.

[hannosch]

Most recent status and some tips for getting an instance / debug it over at #129.

[hannosch]

New Zope2 4.0a5 release is up on PyPi.

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

There's also three more errors related to ZTUtils.Tree, which needs both correct tests and an implementation. This is de/encoding a tree-state into/from a cookie value.

There are also one or two test failures related to HTTPRequest.processInput parsing.

And finally Products.MailHost isn't yet ported, as the only one of the automatically tested optional dependencies.

[tseaver]

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

I think I fixed the FileUpload tests today in #138.

There's also three more errors related to ZTUtils.Tree, which needs both correct tests and an implementation. This is de/encoding a tree-state into/from a cookie value.

I got those tests passing today in #137.

There are also one or two test failures related to HTTPRequest.processInput parsing.

One of those is fixed in #138.

And finally Products.MailHost isn't yet ported, as the only one of the automatically tested optional dependencies.

That one is ugly: the email package APIs want native str on both Python 2 and Python 3, which makes for a bunch of realy ugly branching.

[dataflake]

I've been working on MailHost, and like Tres says, it's another big string compatibility mess.

[hannosch]

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

I think I fixed the FileUpload tests today in #138.

There's still a test failure:

Error in test test_processInputs_with_file_upload_gets_iterator (ZPublisher.tests.testHTTPRequest.HTTPRequestTests)
Traceback (most recent call last):
  File "/opt/python/3.6.1/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/opt/python/3.6.1/lib/python3.6/unittest/case.py", line 601, in run
    testMethod()
  File "/home/travis/build/zopefoundation/Zope/src/ZPublisher/tests/testHTTPRequest.py", line 764, in test_processInputs_with_file_upload_gets_iterator
    self.assertEqual(list(f), ['test\n'])
TypeError: 'FileUpload' object is not iterable

I took that to mean the way FileUpload.__init__ puts methods from the underlying field storage into __dict__ doesn't work anymore. I thought new-style classes either need a __getattribute__ hook to intercept some of these, or in some cases like __iter__ even that isn't enough and it needs the method to be defined explicitly.

[hannosch]

Final test failure is fixed and the master branch is green on Py{27,34,35,36}.

Products.MailHost is the last optional but default dependency of Zope which isn't ported yet.

And of course lots of untested code in Zope is likely still wrong ;)

[gotcha]

untested code ? really ? :)

[tseaver]

#148 is the biggest unresolved blocker I know of for "core" Zope: I set up an instance yesterday on Python 3.6 and was able to exercise all the default ZMI options except those involving file uploads.

[hannosch]

All the dependencies are done (MailHost is now entirely optional). Closing this in favor of the b1 milestone, which tracks general issues - not only those about Python 3 compatibility.