Python 3 compatibility #39

Closed
tomgross opened this Issue Nov 8, 2015 · 20 comments

Comments

9 participants
@tomgross
Contributor

tomgross commented Nov 8, 2015

According to caniusepython3 the following packages need some love to get Python 3 support:

Optional dependencies:

Zope itself:

  • Zope
@tseaver

This comment has been minimized.

Show comment
Hide comment
@tseaver

tseaver Nov 9, 2015

Member

I updated the list with correct capitalizations and links.

Member

tseaver commented Nov 9, 2015

I updated the list with correct capitalizations and links.

@tomgross tomgross referenced this issue in plone/Products.CMFPlone Apr 4, 2016

Open

Improve Zope for a better Plone #1504

@icemac

This comment has been minimized.

Show comment
Hide comment
@icemac

icemac Jul 1, 2016

Contributor

Updated the list with four dependencies that were missing.

Contributor

icemac commented Jul 1, 2016

Updated the list with four dependencies that were missing.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Jul 21, 2016

Member

initgroups turned out to be available in Python's os module in Python 2.7, so I dropped the dependency and made a new initgroups release that is merely a shim for os.initgroups.

For nt_svcutils, I removed the code from zopectl that used this. This was only needed to install and manage a Windows system service. For daemonizing someone can use supervisor or any other tool that provides that.

I also made some progress on ZCTextIndex, by removing its C code and replacing it with pure Python. The speed advantage shouldn't matter that much anymore, and if someone really cares about full text indexing, they'll likely use Solr or Elasticsearch these days.

Finally for DocumentTemplate, I have a branch to replace the C code with pure Python as well, but am not quite finished with that yet (zopefoundation/DocumentTemplate#2).

Member

hannosch commented Jul 21, 2016

initgroups turned out to be available in Python's os module in Python 2.7, so I dropped the dependency and made a new initgroups release that is merely a shim for os.initgroups.

For nt_svcutils, I removed the code from zopectl that used this. This was only needed to install and manage a Windows system service. For daemonizing someone can use supervisor or any other tool that provides that.

I also made some progress on ZCTextIndex, by removing its C code and replacing it with pure Python. The speed advantage shouldn't matter that much anymore, and if someone really cares about full text indexing, they'll likely use Solr or Elasticsearch these days.

Finally for DocumentTemplate, I have a branch to replace the C code with pure Python as well, but am not quite finished with that yet (zopefoundation/DocumentTemplate#2).

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Jul 23, 2016

Member

I merged the code from Products.OFSP got back into Zope2, as there wasn't much left there. So it'll be done as part of Zope2 itself.

Member

hannosch commented Jul 23, 2016

I merged the code from Products.OFSP got back into Zope2, as there wasn't much left there. So it'll be done as part of Zope2 itself.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Jul 24, 2016

Member

I've also started to try and break out ZServer and make it an optional dependency. I think it should be possible to create a new ZServer distribution and move the ZServer, Lifetime and webdav packages into it. That new distribution should depend on Zope2, but not the other way around. There's still quite some work to do for that, but it should be possible. With that split, we could make Zope2 Python 3 compatible while using the WSGI publisher, without having to port ZServer at first or ever.

Member

hannosch commented Jul 24, 2016

I've also started to try and break out ZServer and make it an optional dependency. I think it should be possible to create a new ZServer distribution and move the ZServer, Lifetime and webdav packages into it. That new distribution should depend on Zope2, but not the other way around. There's still quite some work to do for that, but it should be possible. With that split, we could make Zope2 Python 3 compatible while using the WSGI publisher, without having to port ZServer at first or ever.

@leorochael

This comment has been minimized.

Show comment
Hide comment
@leorochael

leorochael Jul 25, 2016

Contributor

It should be possible to cut out ZServer from Zope2 without making ZServer depend on Zope2 at all, by moving all the parts that import from ZPublisher into ZPublisher itself, turning ZPublisher into a pure WSGI app and turning ZServer into a pure WSGI HTTP Server.

I expand on this subject here.

Contributor

leorochael commented Jul 25, 2016

It should be possible to cut out ZServer from Zope2 without making ZServer depend on Zope2 at all, by moving all the parts that import from ZPublisher into ZPublisher itself, turning ZPublisher into a pure WSGI app and turning ZServer into a pure WSGI HTTP Server.

I expand on this subject here.

@dwt

This comment has been minimized.

Show comment
Hide comment
@dwt

dwt Oct 4, 2016

Contributor

I propose linking the respective tracking bugs for python3 compatibility in each project instead of just the projects. Anybody against this?

Contributor

dwt commented Oct 4, 2016

I propose linking the respective tracking bugs for python3 compatibility in each project instead of just the projects. Anybody against this?

@icemac

This comment has been minimized.

Show comment
Hide comment
@icemac

icemac Oct 21, 2016

Contributor

@dwt +1

Contributor

icemac commented Oct 21, 2016

@dwt +1

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Jan 22, 2017

Member

As a current status, we have a couple of issues / branches / PR's in progress and some of those can be worked on in parallel:

Once all of these are done, work can begin on Zope2 itself. It might be possible to use all the WIP branches to start, but it might not. Once Zope2 is done, work can start on the various Products.* being dependent on Zope2.

Member

hannosch commented Jan 22, 2017

As a current status, we have a couple of issues / branches / PR's in progress and some of those can be worked on in parallel:

Once all of these are done, work can begin on Zope2 itself. It might be possible to use all the WIP branches to start, but it might not. Once Zope2 is done, work can start on the various Products.* being dependent on Zope2.

@Daetalus

This comment has been minimized.

Show comment
Hide comment
@Daetalus

Daetalus Jan 22, 2017

Hi @hannosch

Just FYI, I try to let Pyston support Zope. But they didn't support the compiler package, which used by RestrictedPython. After some investigation. I try to use ast to replace compiler in RestrictedPython.

I start it by reading the source code of RestrictedPython. And then compare the ast before and after the MutatingWalker. Then use ast.NodeTransformer to do the same things. It basically a "reverse engineering". But the corner cases took me lots of time. More information is here: https://lab.nexedi.com/Daetalus/RestrictedPython-3.6.0/blob/c4652cb365edfe21e41c20bdf51c92dfab7a283f/src/RestrictedPython/notes.txt

It just a prototype, but in theory, it could work both Python 2/3 and Pyston. I think it's better to let you and other guys to know there already have some efforts to let RestrictedPython to support Python 3. Where to go on next is depend on you. (Just in case, I will resend this message to Zope dev mailist again).

Thanks all!

Hi @hannosch

Just FYI, I try to let Pyston support Zope. But they didn't support the compiler package, which used by RestrictedPython. After some investigation. I try to use ast to replace compiler in RestrictedPython.

I start it by reading the source code of RestrictedPython. And then compare the ast before and after the MutatingWalker. Then use ast.NodeTransformer to do the same things. It basically a "reverse engineering". But the corner cases took me lots of time. More information is here: https://lab.nexedi.com/Daetalus/RestrictedPython-3.6.0/blob/c4652cb365edfe21e41c20bdf51c92dfab7a283f/src/RestrictedPython/notes.txt

It just a prototype, but in theory, it could work both Python 2/3 and Pyston. I think it's better to let you and other guys to know there already have some efforts to let RestrictedPython to support Python 3. Where to go on next is depend on you. (Just in case, I will resend this message to Zope dev mailist again).

Thanks all!

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Jan 22, 2017

Member

@Daetalus Can you put this pointer into the RestrictedPython issue at zopefoundation/RestrictedPython#12. There's a group of people who have been working on a branch there for the past 12 months using the same approach you took.

Member

hannosch commented Jan 22, 2017

@Daetalus Can you put this pointer into the RestrictedPython issue at zopefoundation/RestrictedPython#12. There's a group of people who have been working on a branch there for the past 12 months using the same approach you took.

@Daetalus Daetalus referenced this issue in zopefoundation/RestrictedPython Jan 22, 2017

Closed

Port to Python 3 #12

@icemac icemac referenced this issue May 5, 2017

Closed

Python 3 support #113

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch May 15, 2017

Member

Most recent status and some tips for getting an instance / debug it over at #129.

Member

hannosch commented May 15, 2017

Most recent status and some tips for getting an instance / debug it over at #129.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch May 22, 2017

Member

New Zope2 4.0a5 release is up on PyPi.

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

There's also three more errors related to ZTUtils.Tree, which needs both correct tests and an implementation. This is de/encoding a tree-state into/from a cookie value.

There are also one or two test failures related to HTTPRequest.processInput parsing.

And finally Products.MailHost isn't yet ported, as the only one of the automatically tested optional dependencies.

Member

hannosch commented May 22, 2017

New Zope2 4.0a5 release is up on PyPi.

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

There's also three more errors related to ZTUtils.Tree, which needs both correct tests and an implementation. This is de/encoding a tree-state into/from a cookie value.

There are also one or two test failures related to HTTPRequest.processInput parsing.

And finally Products.MailHost isn't yet ported, as the only one of the automatically tested optional dependencies.

@tseaver

This comment has been minimized.

Show comment
Hide comment
@tseaver

tseaver May 23, 2017

Member

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

I think I fixed the FileUpload tests today in #138.

There's also three more errors related to ZTUtils.Tree, which needs both correct tests and an implementation. This is de/encoding a tree-state into/from a cookie value.

I got those tests passing today in #137.

There are also one or two test failures related to HTTPRequest.processInput parsing.

One of those is fixed in #138.

And finally Products.MailHost isn't yet ported, as the only one of the automatically tested optional dependencies.

That one is ugly: the email package APIs want native str on both Python 2 and Python 3, which makes for a bunch of realy ugly branching.

Member

tseaver commented May 23, 2017

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

I think I fixed the FileUpload tests today in #138.

There's also three more errors related to ZTUtils.Tree, which needs both correct tests and an implementation. This is de/encoding a tree-state into/from a cookie value.

I got those tests passing today in #137.

There are also one or two test failures related to HTTPRequest.processInput parsing.

One of those is fixed in #138.

And finally Products.MailHost isn't yet ported, as the only one of the automatically tested optional dependencies.

That one is ugly: the email package APIs want native str on both Python 2 and Python 3, which makes for a bunch of realy ugly branching.

@dataflake

This comment has been minimized.

Show comment
Hide comment
@dataflake

dataflake May 24, 2017

Member

I've been working on MailHost, and like Tres says, it's another big string compatibility mess.

Member

dataflake commented May 24, 2017

I've been working on MailHost, and like Tres says, it's another big string compatibility mess.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch May 24, 2017

Member

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

I think I fixed the FileUpload tests today in #138.

There's still a test failure:

Error in test test_processInputs_with_file_upload_gets_iterator (ZPublisher.tests.testHTTPRequest.HTTPRequestTests)
Traceback (most recent call last):
  File "/opt/python/3.6.1/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/opt/python/3.6.1/lib/python3.6/unittest/case.py", line 601, in run
    testMethod()
  File "/home/travis/build/zopefoundation/Zope/src/ZPublisher/tests/testHTTPRequest.py", line 764, in test_processInputs_with_file_upload_gets_iterator
    self.assertEqual(list(f), ['test\n'])
TypeError: 'FileUpload' object is not iterable

I took that to mean the way FileUpload.__init__ puts methods from the underlying field storage into __dict__ doesn't work anymore. I thought new-style classes either need a __getattribute__ hook to intercept some of these, or in some cases like __iter__ even that isn't enough and it needs the method to be defined explicitly.

Member

hannosch commented May 24, 2017

This still has 2 test errors related to the HTTPRequest.FileUpload class. It's both an old-style class, using old-style class semantics and has to be adjusted to non-trivial changes in cgi.FieldStorage. FileUpload tries to be a half-transparent wrapper around FieldStorage.

I think I fixed the FileUpload tests today in #138.

There's still a test failure:

Error in test test_processInputs_with_file_upload_gets_iterator (ZPublisher.tests.testHTTPRequest.HTTPRequestTests)
Traceback (most recent call last):
  File "/opt/python/3.6.1/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/opt/python/3.6.1/lib/python3.6/unittest/case.py", line 601, in run
    testMethod()
  File "/home/travis/build/zopefoundation/Zope/src/ZPublisher/tests/testHTTPRequest.py", line 764, in test_processInputs_with_file_upload_gets_iterator
    self.assertEqual(list(f), ['test\n'])
TypeError: 'FileUpload' object is not iterable

I took that to mean the way FileUpload.__init__ puts methods from the underlying field storage into __dict__ doesn't work anymore. I thought new-style classes either need a __getattribute__ hook to intercept some of these, or in some cases like __iter__ even that isn't enough and it needs the method to be defined explicitly.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch May 31, 2017

Member

Final test failure is fixed and the master branch is green on Py{27,34,35,36}.

Products.MailHost is the last optional but default dependency of Zope which isn't ported yet.

And of course lots of untested code in Zope is likely still wrong ;)

Member

hannosch commented May 31, 2017

Final test failure is fixed and the master branch is green on Py{27,34,35,36}.

Products.MailHost is the last optional but default dependency of Zope which isn't ported yet.

And of course lots of untested code in Zope is likely still wrong ;)

@gotcha

This comment has been minimized.

Show comment
Hide comment
@gotcha

gotcha Jun 1, 2017

Member

untested code ? really ? :)

Member

gotcha commented Jun 1, 2017

untested code ? really ? :)

@icemac icemac added the enhancement label Jun 7, 2017

@icemac icemac added this to the 4.0b1 milestone Jun 7, 2017

@tseaver

This comment has been minimized.

Show comment
Hide comment
@tseaver

tseaver Jun 7, 2017

Member

#148 is the biggest unresolved blocker I know of for "core" Zope: I set up an instance yesterday on Python 3.6 and was able to exercise all the default ZMI options except those involving file uploads.

Member

tseaver commented Jun 7, 2017

#148 is the biggest unresolved blocker I know of for "core" Zope: I set up an instance yesterday on Python 3.6 and was able to exercise all the default ZMI options except those involving file uploads.

@hannosch

This comment has been minimized.

Show comment
Hide comment
@hannosch

hannosch Sep 15, 2017

Member

All the dependencies are done (MailHost is now entirely optional). Closing this in favor of the b1 milestone, which tracks general issues - not only those about Python 3 compatibility.

Member

hannosch commented Sep 15, 2017

All the dependencies are done (MailHost is now entirely optional). Closing this in favor of the b1 milestone, which tracks general issues - not only those about Python 3 compatibility.

@hannosch hannosch closed this Sep 15, 2017

@hannosch hannosch moved this from Open to Done in Zope 4 beta release Sep 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment