Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

More fixes for doc building with python 3 #1383

Merged
merged 8 commits into from

5 participants

@dmcdougall
Collaborator

In #1361, @cjmayo pointed out some extra fixes that needed to be implemented to make doc building under python 3 work properly. These have been modified to not break backward compatibility with doc building in python 2.

@dmcdougall
Collaborator

@cjmayo Let me know if this works with python 3. I have no reason to believe it shouldn't.

Edit: Er, I mean let me know if it doesn't work with python 3.

@mdboom
Owner

+1. This works for me. Thanks for taking care of this -- building the docs under Python 3 is something that fell through the cracks during the Python 3 port.

@dmcdougall
Collaborator

Thanks goes to @cjmayo for providing the original patches.

@cjmayo

df93eca and 19b1a05 work with Python 3 for me and I can get the docs to build (I can't formally verify but I have lots of html, images and pdfs and all the links I have tried in the html work). But it is a bit more complicated.

I can only get this to work with Python 3.3.0 and not with Python 3.2.3 (specifically Gentoo Linux python-3.2.3). 3.2 fails with:

sphinx 1.1.3-r3
reading sources... [ 80%] examples/pylab_examples/unicode_demo   
Sphinx error:
'ascii' codec can't decode byte 0xc3 in position 134: ordinal not
Building HTML failed.

and I haven't figured that out.

My recipe:

Dependencies needed for Python 3.3 compatibility:

Matplotlib library patches (most of these probably break Python 2 use so need work):

  • I have to revert the manual reference count patches to ft2font.cpp (for Python 2.7.3 also), and it seems others maybe have the same problem in Issue #1309. But for yet others it fixed it #1054???

  • To cope with the seeming 2to3 import exception bug http://bugs.python.org/issue16214

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,8 +125,7 @@
 """
 from __future__ import print_function

-import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
-       traceback, exceptions
+import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, traceback

 from docutils.parsers.rst import directives
 from docutils import nodes
  • hash functions in Python 3 take bytes not strings:
--- lib/matplotlib/sphinxext/mathmpl.py.orig
+++ lib/matplotlib/sphinxext/mathmpl.py
@@ -65,7 +65,7 @@
 def latex2html(node, source):
     inline = isinstance(node.parent, nodes.TextElement)
     latex = node['latex']
-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

     destdir = os.path.join(setup.app.builder.outdir, '_images', 'mathmpl')
     if not os.path.exists(destdir):

Matplotlib documentation build system patches:

  • make.py, not sure what the final proposal was from #1361 I just do:
--- doc/make.py.orig
+++ doc/make.py
@@ -5,108 +5,9 @@
 import glob
 import os
 import shutil
+from shutil import ignore_patterns, copytree
 import sys

-### Begin compatibility block for pre-v2.6: ###
-#
-# ignore_patterns and copytree funtions are copies of what is included
-# in shutil.copytree of python v2.6 and later.
-#
....
--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = open(gallery_path, 'w')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)
@mdboom
Owner

Sorry for the noise. I was wrong about this "working for me". I was inadvertently using the Python 2.x installation of Sphinx. I'm working through some remaining Python 3 issues and will have a PR against this soon.

@dmcdougall
Collaborator

@cjmayo Have those docutils patches been merged upstream? We might not be able to get this in for 1.2 if we have to wait for upstream changes.

@cjmayo

Yes they are direct from upstream:
http://docutils.sourceforge.net/docs/dev/repository.html#id5
It's just the git interface is a bit easier to browse.

But they are only needed for Python 3.3. All the other the other matplotlib fixes will be needed for Python 3.2 too and there is something else needed for 3.2 for that 'ascii' codec error.

@dmcdougall
Collaborator

Having major headaches with python2/3 unicode differences. Bleh. I'm trying to use the utilities from six.py but it claims six is not a module...

@dmcdougall
Collaborator

@cjmayo I have an idea about that ascii error on python 3.2. I'm pretty sure in python 3.2 things like u'asd' are invalid syntax, but in python 3.3 u'asd' is allowed.

@dmcdougall
Collaborator

I'm moving the commits from #1361 into this branch.

@dmcdougall
Collaborator

@mdboom Are you saying running python3.2 make.py is not sufficient, and that I need a separate python3 build of sphinx installed?

@mdboom
Owner

@dmcdougall: Yes -- you'll need to install sphinx in Python 3 (using pip-3.2 install sphinx or similar, or from a package), but then you also need to make sure that the Python 3 version of sphinx-build is the default on the path, probably by manually overriding the PATH environment variable.

@dmcdougall
Collaborator
$ which sphinx-build
/opt/local/Library/Frameworks/Python.framework/Versions/3.2/bin/sphinx-build
$ python --version
Python 3.2.3
$ python make.py
Running Sphinx v1.1.3
Initializing GitHub plugin
loading pickled environment... done
animation, api, axes_grid, event_handling, misc, mplot3d, old_animation, pylab_examples, tests, units, user_interfaces, widgets, 
building [html]: targets for 566 source files that are out of date
updating environment: 0 added, 53 changed, 0 removed
reading sources... [  5%] api/artist_api                                                                                          
Exception occurred:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/sphinx/ext/inheritance_diagram.py", line 146, in _class_info
    builtins = list(vars(__builtin__).values())
NameError: global name '__builtin__' is not defined
The full traceback has been saved in /var/folders/rs/pqvv7l192xs426r_016501f80000gn/T/sphinx-err-v0lp1g.log, if you want to report the issue to the developers.
@mdboom
Owner

@dmcdougall: You're farther along than I ever got at this point. That traceback is too promising -- it suggests that the inheritance_diagram hasn't yet been updated for Python 3 (I guess). I wonder if disabling that plugin in our conf.py let's it go further, or whether there are just other problems hiding behind that one?

@dmcdougall
Collaborator

@mdboom I'm using all of the changes in this PR. I added some commits earlier today. I also merged the ones from #1361 to keep everything together.

MacPorts doesn't seem to sport a py31/py32 version of sphinx. I have no experience with sphinx. My guess is that support for python 3 is minimal. I'm not sure where to go from here.

@dmcdougall
Collaborator

@mdboom Commenting out the inheritance diagram in conf.py it mostly works. I get the following error, but the doc build still completes!

<autodoc>:0: ERROR: Unknown interpreted text role "function".
/Users/damon/git/github/matplotlib/doc/devel/documenting_mpl.rst:384: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.patches matplotlib.lines matplotlib.text
   :parts: 2
/Users/damon/git/github/matplotlib/doc/devel/transformations.rst:5: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.transforms matplotlib.path
   :parts: 1
/Users/damon/git/github/matplotlib/doc/examples/pylab_examples/multipage_pdf.rst:8: WARNING: Exception occurred in plotting multipage_pdf
 from /Users/damon/git/github/matplotlib/doc/mpl_examples/pylab_examples/multipage_pdf.py:
Traceback (most recent call last):
  File "/Users/damon/python/lib/matplotlib/sphinxext/plot_directive.py", line 515, in run_code
    exec(code, ns)
  File "<string>", line 36
    d['Author'] = u'Jouni K. Sepp\xe4nen'
@dmcdougall
Collaborator

One of them is a unicode issue, because u'asd' is not valid syntax in py3.2.

@mdboom
Owner

It's great that it completes. That unicode issue is probably fixable -- something along the lines of

'Jouni K. Sepp\xe4nen'.decode('latin-1')

If that's the only problem, we're in pretty good shape.

@dmcdougall
Collaborator

I'm also getting this: loading pickled environment... failed: unsupported pickle protocol: 3.

Edit: but the build still completes.

@dmcdougall
Collaborator

Er, nevermind. Turns out the pickling thing gets done to save time on subsequent runs? Since I keep switching between python2 and python3 the pickled environment gets corrupted but it gets regenerated correctly. Disregard my previous comment.

@cjmayo

I can get past the ascii error with Python 3.2 with an updated version of the gen_gallery.py patch above:

--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -20,7 +20,7 @@
 {%% endblock %%}
 """

-import os, glob, re, sys, warnings
+import io, os, glob, re, sys, warnings
 import matplotlib.image as image

 multiimage = re.compile('(.*?)(_\d\d){1,2}')
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = io.open(gallery_path, 'w', encoding='utf8')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)

And a new patch to plot_directive.py:

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,6 +125,7 @@
 """
 from __future__ import print_function

+import io
 import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
        traceback, exceptions

@@ -667,7 +668,7 @@
         else:
             function_name = None

-        with open(source_file_name, 'r') as fd:
+        with io.open(source_file_name, 'r', encoding='utf8') as fd:
             code = fd.read()
         output_base = os.path.basename(source_file_name)
     else:
@@ -809,7 +810,7 @@

     # copy script (if necessary)
     target_name = os.path.join(dest_dir, output_base + source_ext)
-    with open(target_name, 'w') as f:
+    with io.open(target_name, 'w', encoding='utf8') as f:
         if source_file_name == rst_file:
             code_escaped = unescape_doctest(code)
         else:

This now gets me through the html generation and also png creation. Fails now in using graphviz.

Couple of comments on the recent comments.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

@dmcdougall
Collaborator

@cjmayo Cheers for the patch updates. For some reason I don't have a problem with the current ones, but I'll add your updates in since they work for you.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Probably. I didn't know this existed. I have never worked with python 3, so this whole PR/frustration is a combination of me learning and getting headaches.

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

Nope. All the testing I did was the generate the documentation under python2, and the build succeeds. I have not checked the output.

@dmcdougall
Collaborator

I am getting a new error building the docs under python 3:

Running Sphinx v1.1.3

Exception occurred:
  File "/Users/damon/python/lib/matplotlib/dates.py", line 122, in <module>
    from dateutil.rrule import rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY, \
  File "/Users/damon/python/lib/dateutil/rrule.py", line 55
    raise ValueError, "Can't create weekday with n == 0"
                    ^
SyntaxError: invalid syntax

I have no idea why this is happening. 2to3 should have fixed this. Argh! I think it's time for me to call it a day and look at this tomorrow.

@cjmayo

With the last two patches I can now get to the end of the doc building with Python 3.2.3, but I have to use sphinx tip (6bb9dfbdf66e) - sphinx 1.1.3 does not work for me.

@dmcdougall
Collaborator

@cjmayo Progress! Interesting that you need current Sphinx HEAD. What errors do you get with sphinx 1.1.3?

We have two options on proceeding depending on your response to my above question:

1) We may be able to patch our sphinxext files to work with sphinx v1.1.3.
2) If 1) is not possible, then this pull request will fester until the problem is solved upstream.

@mdboom
Owner

At the end of the day, the most important thing is to be able to produce the docs under some version of Python (and Python 2.x is currently working). Having them build under 3.x is of course very desirable, but if that can't be done with a stable version of Sphinx, than there's not much point to putting this is our release either. IMHO, of course.

@dmcdougall
Collaborator

@mdboom Agreed; there are more important v1.2.x bugs currently milestoned. Furthermore, I think if we are waiting in either case the v1.2.x milestone is unnecessary.

I will keep this PR open and remove the milestone. If anybody disagrees then adding it back on is fine.

@ariddell

Even with these changes, I'm encountering an import error related to this bug: RuntimeError: dictionary changed size during iteration from colors.py #1401

@pelson
Collaborator

Thanks @ariddell - the error you are describing is a matplotlib issue relating to python3.3 to try and keep the issues separate, this build process fix is for the documentation building with a currently supported python (lets say python3.2 for sake of argument). So whilst I think what you highlight is important, it should be addressed as a completely independent issue (#1401).

Cheers,

@pelson
Collaborator

@dmcdougall - do you remember the status of this? Is it worth re-visiting and getting this PR merged?

@ariddell

I'm getting the error with 3.2.3.

I've setup a clean virtualenv, pip install'd only numpy and git://github.com/dmcdougall/matplotlib.git@py3_docs_2#egg=matplotlib and I get this:

Python 3.2.3 (default, Oct 19 2012, 19:53:16) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.sphinxext.plot_directive
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/__init__.py", line 151, in <module>
    from matplotlib.rcsetup import (defaultParams,
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/rcsetup.py", line 20, in <module>
    from matplotlib.colors import is_color_like
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/colors.py", line 205, in <module>
    for k, v in cnames.items():
RuntimeError: dictionary changed size during iteration

Python path is: ['', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/distribute-0.6.24-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/pip-1.1-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/lib-dynload', '/usr/lib/python3.2', '/usr/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages']

Hope this helps. Thanks for all your work on this.

@pelson
Collaborator

I'm getting the error with 3.2.3.

Ah... Ok - thanks for pointing that out. In that case I think it is very pertinent for this PR.

Cheers,

@pelson
Collaborator

@dmcdougall - I'd be happy to merge this. Is there more work to be done before I do, or shall I go ahead and press the green button?

@dmcdougall
Collaborator

@pelson It appears that @cjmayo is currently operational with these changes, and that @ariddell is having issues with that dictionary error. That error, however, appears to have been resolved in the comments in #1401. So, I think we should merge this. If there are further dictionary errors, we can resolve those separately.

Cheers @cjmayo for the help.

@pelson pelson merged commit 1d6abc1 into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
8 doc/conf.py
@@ -174,10 +174,10 @@
# Additional stuff for the LaTeX preamble.
latex_preamble = """
- \usepackage{amsmath}
- \usepackage{amsfonts}
- \usepackage{amssymb}
- \usepackage{txfonts}
+ \\usepackage{amsmath}
+ \\usepackage{amsfonts}
+ \\usepackage{amssymb}
+ \\usepackage{txfonts}
"""
# Documents to append as an appendix to all manuals.
View
8 doc/make.py
@@ -89,20 +89,20 @@ def copytree(src, dst, symlinks=False, ignore=None):
copy2(srcname, dstname)
# catch the Error from the recursive copytree so that we can
# continue with other files
- except Error, err:
+ except Error as err:
errors.extend(err.args[0])
- except EnvironmentError, why:
+ except EnvironmentError as why:
errors.append((srcname, dstname, str(why)))
try:
copystat(src, dst)
- except OSError, why:
+ except OSError as why:
if WindowsError is not None and isinstance(why, WindowsError):
# Copying file access times may fail on Windows
pass
else:
errors.extend((src, dst, str(why)))
if errors:
- raise Error, errors
+ raise Error(errors)
### End compatibility block for pre-v2.6 ###
View
6 doc/sphinxext/gen_gallery.py
@@ -121,18 +121,18 @@ def gen_gallery(app, doctree):
gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
if os.path.exists(gallery_path):
- fh = file(gallery_path, 'r')
+ fh = open(gallery_path, 'r')
regenerate = fh.read() != content
fh.close()
else:
regenerate = True
if regenerate:
- fh = file(gallery_path, 'w')
+ fh = open(gallery_path, 'w')
fh.write(content)
fh.close()
for key in app.builder.status_iterator(
- thumbnails.iterkeys(), "generating thumbnails... ",
+ iter(thumbnails.keys()), "generating thumbnails... ",
length=len(thumbnails)):
image.thumbnail(key, thumbnails[key], 0.3)
View
19 doc/sphinxext/gen_rst.py
@@ -2,6 +2,7 @@
generate the rst files for the examples by iterating over the pylab examples
"""
from __future__ import print_function
+import io
import os, glob
import os
@@ -37,15 +38,18 @@ def generate_example_rst(app):
continue
fullpath = os.path.join(root,fname)
- contents = file(fullpath).read()
+ if sys.version_info[0] >= 3:
+ contents = io.open(fullpath, encoding='utf8').read()
+ else:
+ contents = io.open(fullpath).read()
# indent
relpath = os.path.split(root)[-1]
datad.setdefault(relpath, []).append((fullpath, fname, contents))
- subdirs = datad.keys()
+ subdirs = list(datad.keys())
subdirs.sort()
- fhindex = file(os.path.join(exampledir, 'index.rst'), 'w')
+ fhindex = open(os.path.join(exampledir, 'index.rst'), 'w')
fhindex.write("""\
.. _examples-index:
@@ -77,7 +81,7 @@ def generate_example_rst(app):
os.makedirs(outputdir)
subdirIndexFile = os.path.join(rstdir, 'index.rst')
- fhsubdirIndex = file(subdirIndexFile, 'w')
+ fhsubdirIndex = open(subdirIndexFile, 'w')
fhindex.write(' %s/index.rst\n\n'%subdir)
fhsubdirIndex.write("""\
@@ -122,14 +126,17 @@ def generate_example_rst(app):
) and
not noplot_regex.search(contents))
if not do_plot:
- fhstatic = file(outputfile, 'w')
+ fhstatic = open(outputfile, 'w')
fhstatic.write(contents)
fhstatic.close()
if not out_of_date(fullpath, outrstfile):
continue
- fh = file(outrstfile, 'w')
+ if sys.version_info[0] >= 3:
+ fh = io.open(outrstfile, 'w', encoding='utf8')
+ else:
+ fh = io.open(outrstfile, 'w')
fh.write('.. _%s-%s:\n\n'%(subdir, basename))
title = '%s example code: %s'%(subdir, fname)
#title = '<img src=%s> %s example code: %s'%(thumbfile, subdir, fname)
View
2  lib/matplotlib/sphinxext/mathmpl.py
@@ -65,7 +65,7 @@ def latex2png(latex, filename, fontset='cm'):
def latex2html(node, source):
inline = isinstance(node.parent, nodes.TextElement)
latex = node['latex']
- name = 'math-%s' % md5(latex).hexdigest()[-10:]
+ name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]
destdir = os.path.join(setup.app.builder.outdir, '_images', 'mathmpl')
if not os.path.exists(destdir):
View
4 lib/matplotlib/sphinxext/plot_directive.py
@@ -125,8 +125,8 @@
"""
from __future__ import print_function
-import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
- traceback, exceptions
+import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap
+import traceback
from docutils.parsers.rst import directives
from docutils import nodes
Something went wrong with that request. Please try again.