More fixes for doc building with python 3 #1383

Merged
merged 8 commits into from Apr 2, 2013

Conversation

Projects
None yet
5 participants
Member

dmcdougall commented Oct 13, 2012

In #1361, @cjmayo pointed out some extra fixes that needed to be implemented to make doc building under python 3 work properly. These have been modified to not break backward compatibility with doc building in python 2.

Member

dmcdougall commented Oct 13, 2012

@cjmayo Let me know if this works with python 3. I have no reason to believe it shouldn't.

Edit: Er, I mean let me know if it doesn't work with python 3.

Owner

mdboom commented Oct 13, 2012

+1. This works for me. Thanks for taking care of this -- building the docs under Python 3 is something that fell through the cracks during the Python 3 port.

Member

dmcdougall commented Oct 13, 2012

Thanks goes to @cjmayo for providing the original patches.

cjmayo commented Oct 14, 2012

df93eca and 19b1a05 work with Python 3 for me and I can get the docs to build (I can't formally verify but I have lots of html, images and pdfs and all the links I have tried in the html work). But it is a bit more complicated.

I can only get this to work with Python 3.3.0 and not with Python 3.2.3 (specifically Gentoo Linux python-3.2.3). 3.2 fails with:

sphinx 1.1.3-r3
reading sources... [ 80%] examples/pylab_examples/unicode_demo   
Sphinx error:
'ascii' codec can't decode byte 0xc3 in position 134: ordinal not
Building HTML failed.

and I haven't figured that out.

My recipe:

Dependencies needed for Python 3.3 compatibility:

Matplotlib library patches (most of these probably break Python 2 use so need work):

  • I have to revert the manual reference count patches to ft2font.cpp (for Python 2.7.3 also), and it seems others maybe have the same problem in Issue #1309. But for yet others it fixed it #1054???
  • To cope with the seeming 2to3 import exception bug http://bugs.python.org/issue16214
--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,8 +125,7 @@
 """
 from __future__ import print_function

-import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
-       traceback, exceptions
+import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, traceback

 from docutils.parsers.rst import directives
 from docutils import nodes
  • hash functions in Python 3 take bytes not strings:
--- lib/matplotlib/sphinxext/mathmpl.py.orig
+++ lib/matplotlib/sphinxext/mathmpl.py
@@ -65,7 +65,7 @@
 def latex2html(node, source):
     inline = isinstance(node.parent, nodes.TextElement)
     latex = node['latex']
-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

     destdir = os.path.join(setup.app.builder.outdir, '_images', 'mathmpl')
     if not os.path.exists(destdir):

Matplotlib documentation build system patches:

  • make.py, not sure what the final proposal was from #1361 I just do:
--- doc/make.py.orig
+++ doc/make.py
@@ -5,108 +5,9 @@
 import glob
 import os
 import shutil
+from shutil import ignore_patterns, copytree
 import sys

-### Begin compatibility block for pre-v2.6: ###
-#
-# ignore_patterns and copytree funtions are copies of what is included
-# in shutil.copytree of python v2.6 and later.
-#
....
--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = open(gallery_path, 'w')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)
Owner

mdboom commented Oct 15, 2012

Sorry for the noise. I was wrong about this "working for me". I was inadvertently using the Python 2.x installation of Sphinx. I'm working through some remaining Python 3 issues and will have a PR against this soon.

Member

dmcdougall commented Oct 15, 2012

@cjmayo Have those docutils patches been merged upstream? We might not be able to get this in for 1.2 if we have to wait for upstream changes.

cjmayo commented Oct 15, 2012

Yes they are direct from upstream:
http://docutils.sourceforge.net/docs/dev/repository.html#id5
It's just the git interface is a bit easier to browse.

But they are only needed for Python 3.3. All the other the other matplotlib fixes will be needed for Python 3.2 too and there is something else needed for 3.2 for that 'ascii' codec error.

Member

dmcdougall commented Oct 15, 2012

Having major headaches with python2/3 unicode differences. Bleh. I'm trying to use the utilities from six.py but it claims six is not a module...

Member

dmcdougall commented Oct 16, 2012

@cjmayo I have an idea about that ascii error on python 3.2. I'm pretty sure in python 3.2 things like u'asd' are invalid syntax, but in python 3.3 u'asd' is allowed.

Member

dmcdougall commented Oct 16, 2012

I'm moving the commits from #1361 into this branch.

Member

dmcdougall commented Oct 16, 2012

@mdboom Are you saying running python3.2 make.py is not sufficient, and that I need a separate python3 build of sphinx installed?

Owner

mdboom commented Oct 16, 2012

@dmcdougall: Yes -- you'll need to install sphinx in Python 3 (using pip-3.2 install sphinx or similar, or from a package), but then you also need to make sure that the Python 3 version of sphinx-build is the default on the path, probably by manually overriding the PATH environment variable.

Member

dmcdougall commented Oct 16, 2012

$ which sphinx-build
/opt/local/Library/Frameworks/Python.framework/Versions/3.2/bin/sphinx-build
$ python --version
Python 3.2.3
$ python make.py
Running Sphinx v1.1.3
Initializing GitHub plugin
loading pickled environment... done
animation, api, axes_grid, event_handling, misc, mplot3d, old_animation, pylab_examples, tests, units, user_interfaces, widgets, 
building [html]: targets for 566 source files that are out of date
updating environment: 0 added, 53 changed, 0 removed
reading sources... [  5%] api/artist_api                                                                                          
Exception occurred:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/sphinx/ext/inheritance_diagram.py", line 146, in _class_info
    builtins = list(vars(__builtin__).values())
NameError: global name '__builtin__' is not defined
The full traceback has been saved in /var/folders/rs/pqvv7l192xs426r_016501f80000gn/T/sphinx-err-v0lp1g.log, if you want to report the issue to the developers.
Owner

mdboom commented Oct 16, 2012

@dmcdougall: You're farther along than I ever got at this point. That traceback is too promising -- it suggests that the inheritance_diagram hasn't yet been updated for Python 3 (I guess). I wonder if disabling that plugin in our conf.py let's it go further, or whether there are just other problems hiding behind that one?

Member

dmcdougall commented Oct 16, 2012

@mdboom I'm using all of the changes in this PR. I added some commits earlier today. I also merged the ones from #1361 to keep everything together.

MacPorts doesn't seem to sport a py31/py32 version of sphinx. I have no experience with sphinx. My guess is that support for python 3 is minimal. I'm not sure where to go from here.

Member

dmcdougall commented Oct 16, 2012

@mdboom Commenting out the inheritance diagram in conf.py it mostly works. I get the following error, but the doc build still completes!

<autodoc>:0: ERROR: Unknown interpreted text role "function".
/Users/damon/git/github/matplotlib/doc/devel/documenting_mpl.rst:384: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.patches matplotlib.lines matplotlib.text
   :parts: 2
/Users/damon/git/github/matplotlib/doc/devel/transformations.rst:5: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.transforms matplotlib.path
   :parts: 1
/Users/damon/git/github/matplotlib/doc/examples/pylab_examples/multipage_pdf.rst:8: WARNING: Exception occurred in plotting multipage_pdf
 from /Users/damon/git/github/matplotlib/doc/mpl_examples/pylab_examples/multipage_pdf.py:
Traceback (most recent call last):
  File "/Users/damon/python/lib/matplotlib/sphinxext/plot_directive.py", line 515, in run_code
    exec(code, ns)
  File "<string>", line 36
    d['Author'] = u'Jouni K. Sepp\xe4nen'
Member

dmcdougall commented Oct 16, 2012

One of them is a unicode issue, because u'asd' is not valid syntax in py3.2.

Owner

mdboom commented Oct 16, 2012

It's great that it completes. That unicode issue is probably fixable -- something along the lines of

'Jouni K. Sepp\xe4nen'.decode('latin-1')

If that's the only problem, we're in pretty good shape.

Member

dmcdougall commented Oct 16, 2012

I'm also getting this: loading pickled environment... failed: unsupported pickle protocol: 3.

Edit: but the build still completes.

Member

dmcdougall commented Oct 16, 2012

Er, nevermind. Turns out the pickling thing gets done to save time on subsequent runs? Since I keep switching between python2 and python3 the pickled environment gets corrupted but it gets regenerated correctly. Disregard my previous comment.

cjmayo commented Oct 16, 2012

I can get past the ascii error with Python 3.2 with an updated version of the gen_gallery.py patch above:

--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -20,7 +20,7 @@
 {%% endblock %%}
 """

-import os, glob, re, sys, warnings
+import io, os, glob, re, sys, warnings
 import matplotlib.image as image

 multiimage = re.compile('(.*?)(_\d\d){1,2}')
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = io.open(gallery_path, 'w', encoding='utf8')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)

And a new patch to plot_directive.py:

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,6 +125,7 @@
 """
 from __future__ import print_function

+import io
 import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
        traceback, exceptions

@@ -667,7 +668,7 @@
         else:
             function_name = None

-        with open(source_file_name, 'r') as fd:
+        with io.open(source_file_name, 'r', encoding='utf8') as fd:
             code = fd.read()
         output_base = os.path.basename(source_file_name)
     else:
@@ -809,7 +810,7 @@

     # copy script (if necessary)
     target_name = os.path.join(dest_dir, output_base + source_ext)
-    with open(target_name, 'w') as f:
+    with io.open(target_name, 'w', encoding='utf8') as f:
         if source_file_name == rst_file:
             code_escaped = unescape_doctest(code)
         else:

This now gets me through the html generation and also png creation. Fails now in using graphviz.

Couple of comments on the recent comments.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

Member

dmcdougall commented Oct 16, 2012

@cjmayo Cheers for the patch updates. For some reason I don't have a problem with the current ones, but I'll add your updates in since they work for you.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Probably. I didn't know this existed. I have never worked with python 3, so this whole PR/frustration is a combination of me learning and getting headaches.

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

Nope. All the testing I did was the generate the documentation under python2, and the build succeeds. I have not checked the output.

Member

dmcdougall commented Oct 16, 2012

I am getting a new error building the docs under python 3:

Running Sphinx v1.1.3

Exception occurred:
  File "/Users/damon/python/lib/matplotlib/dates.py", line 122, in <module>
    from dateutil.rrule import rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY, \
  File "/Users/damon/python/lib/dateutil/rrule.py", line 55
    raise ValueError, "Can't create weekday with n == 0"
                    ^
SyntaxError: invalid syntax

I have no idea why this is happening. 2to3 should have fixed this. Argh! I think it's time for me to call it a day and look at this tomorrow.

cjmayo commented Oct 17, 2012

With the last two patches I can now get to the end of the doc building with Python 3.2.3, but I have to use sphinx tip (6bb9dfbdf66e) - sphinx 1.1.3 does not work for me.

Member

dmcdougall commented Oct 17, 2012

@cjmayo Progress! Interesting that you need current Sphinx HEAD. What errors do you get with sphinx 1.1.3?

We have two options on proceeding depending on your response to my above question:

  1. We may be able to patch our sphinxext files to work with sphinx v1.1.3.
  2. If 1) is not possible, then this pull request will fester until the problem is solved upstream.
Owner

mdboom commented Oct 17, 2012

At the end of the day, the most important thing is to be able to produce the docs under some version of Python (and Python 2.x is currently working). Having them build under 3.x is of course very desirable, but if that can't be done with a stable version of Sphinx, than there's not much point to putting this is our release either. IMHO, of course.

Member

dmcdougall commented Oct 17, 2012

@mdboom Agreed; there are more important v1.2.x bugs currently milestoned. Furthermore, I think if we are waiting in either case the v1.2.x milestone is unnecessary.

I will keep this PR open and remove the milestone. If anybody disagrees then adding it back on is fine.

ariddell commented Dec 7, 2012

Even with these changes, I'm encountering an import error related to this bug: RuntimeError: dictionary changed size during iteration from colors.py #1401

Member

pelson commented Dec 10, 2012

Thanks @ariddell - the error you are describing is a matplotlib issue relating to python3.3 to try and keep the issues separate, this build process fix is for the documentation building with a currently supported python (lets say python3.2 for sake of argument). So whilst I think what you highlight is important, it should be addressed as a completely independent issue (#1401).

Cheers,

Member

pelson commented Dec 10, 2012

@dmcdougall - do you remember the status of this? Is it worth re-visiting and getting this PR merged?

I'm getting the error with 3.2.3.

I've setup a clean virtualenv, pip install'd only numpy and git://github.com/dmcdougall/matplotlib.git@py3_docs_2#egg=matplotlib and I get this:

Python 3.2.3 (default, Oct 19 2012, 19:53:16) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.sphinxext.plot_directive
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/__init__.py", line 151, in <module>
    from matplotlib.rcsetup import (defaultParams,
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/rcsetup.py", line 20, in <module>
    from matplotlib.colors import is_color_like
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/colors.py", line 205, in <module>
    for k, v in cnames.items():
RuntimeError: dictionary changed size during iteration

Python path is: ['', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/distribute-0.6.24-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/pip-1.1-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/lib-dynload', '/usr/lib/python3.2', '/usr/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages']

Hope this helps. Thanks for all your work on this.

Member

pelson commented Dec 10, 2012

I'm getting the error with 3.2.3.

Ah... Ok - thanks for pointing that out. In that case I think it is very pertinent for this PR.

Cheers,

Member

pelson commented Mar 29, 2013

@dmcdougall - I'd be happy to merge this. Is there more work to be done before I do, or shall I go ahead and press the green button?

Member

dmcdougall commented Apr 1, 2013

@pelson It appears that @cjmayo is currently operational with these changes, and that @ariddell is having issues with that dictionary error. That error, however, appears to have been resolved in the comments in #1401. So, I think we should merge this. If there are further dictionary errors, we can resolve those separately.

Cheers @cjmayo for the help.

@pelson pelson added a commit that referenced this pull request Apr 2, 2013

@pelson pelson Merge pull request #1383 from dmcdougall/py3_docs_2
More fixes for doc building with python 3
1d6abc1

@pelson pelson merged commit 1d6abc1 into matplotlib:v1.2.x Apr 2, 2013

1 check passed

default The Travis build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment