Skip to content
This repository

More fixes for doc building with python 3 #1383

Merged
merged 8 commits into from about 1 year ago

5 participants

Damon McDougall Michael Droettboom Chris Mayo Allen Riddell Phil Elson
Damon McDougall
Collaborator

In #1361, @cjmayo pointed out some extra fixes that needed to be implemented to make doc building under python 3 work properly. These have been modified to not break backward compatibility with doc building in python 2.

Damon McDougall
Collaborator

@cjmayo Let me know if this works with python 3. I have no reason to believe it shouldn't.

Edit: Er, I mean let me know if it doesn't work with python 3.

Michael Droettboom
Owner

+1. This works for me. Thanks for taking care of this -- building the docs under Python 3 is something that fell through the cracks during the Python 3 port.

Damon McDougall
Collaborator

Thanks goes to @cjmayo for providing the original patches.

Chris Mayo

df93eca and 19b1a05 work with Python 3 for me and I can get the docs to build (I can't formally verify but I have lots of html, images and pdfs and all the links I have tried in the html work). But it is a bit more complicated.

I can only get this to work with Python 3.3.0 and not with Python 3.2.3 (specifically Gentoo Linux python-3.2.3). 3.2 fails with:

sphinx 1.1.3-r3
reading sources... [ 80%] examples/pylab_examples/unicode_demo   
Sphinx error:
'ascii' codec can't decode byte 0xc3 in position 134: ordinal not
Building HTML failed.

and I haven't figured that out.

My recipe:

Dependencies needed for Python 3.3 compatibility:

Matplotlib library patches (most of these probably break Python 2 use so need work):

  • I have to revert the manual reference count patches to ft2font.cpp (for Python 2.7.3 also), and it seems others maybe have the same problem in Issue #1309. But for yet others it fixed it #1054???

  • To cope with the seeming 2to3 import exception bug http://bugs.python.org/issue16214

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,8 +125,7 @@
 """
 from __future__ import print_function

-import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
-       traceback, exceptions
+import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, traceback

 from docutils.parsers.rst import directives
 from docutils import nodes
  • hash functions in Python 3 take bytes not strings:
--- lib/matplotlib/sphinxext/mathmpl.py.orig
+++ lib/matplotlib/sphinxext/mathmpl.py
@@ -65,7 +65,7 @@
 def latex2html(node, source):
     inline = isinstance(node.parent, nodes.TextElement)
     latex = node['latex']
-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

     destdir = os.path.join(setup.app.builder.outdir, '_images', 'mathmpl')
     if not os.path.exists(destdir):

Matplotlib documentation build system patches:

  • make.py, not sure what the final proposal was from #1361 I just do:
--- doc/make.py.orig
+++ doc/make.py
@@ -5,108 +5,9 @@
 import glob
 import os
 import shutil
+from shutil import ignore_patterns, copytree
 import sys

-### Begin compatibility block for pre-v2.6: ###
-#
-# ignore_patterns and copytree funtions are copies of what is included
-# in shutil.copytree of python v2.6 and later.
-#
....
--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = open(gallery_path, 'w')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)
Michael Droettboom
Owner

Sorry for the noise. I was wrong about this "working for me". I was inadvertently using the Python 2.x installation of Sphinx. I'm working through some remaining Python 3 issues and will have a PR against this soon.

Damon McDougall
Collaborator

@cjmayo Have those docutils patches been merged upstream? We might not be able to get this in for 1.2 if we have to wait for upstream changes.

Chris Mayo

Yes they are direct from upstream:
http://docutils.sourceforge.net/docs/dev/repository.html#id5
It's just the git interface is a bit easier to browse.

But they are only needed for Python 3.3. All the other the other matplotlib fixes will be needed for Python 3.2 too and there is something else needed for 3.2 for that 'ascii' codec error.

Damon McDougall
Collaborator

Having major headaches with python2/3 unicode differences. Bleh. I'm trying to use the utilities from six.py but it claims six is not a module...

Damon McDougall
Collaborator

@cjmayo I have an idea about that ascii error on python 3.2. I'm pretty sure in python 3.2 things like u'asd' are invalid syntax, but in python 3.3 u'asd' is allowed.

Damon McDougall
Collaborator

I'm moving the commits from #1361 into this branch.

Damon McDougall
Collaborator

@mdboom Are you saying running python3.2 make.py is not sufficient, and that I need a separate python3 build of sphinx installed?

Michael Droettboom
Owner

@dmcdougall: Yes -- you'll need to install sphinx in Python 3 (using pip-3.2 install sphinx or similar, or from a package), but then you also need to make sure that the Python 3 version of sphinx-build is the default on the path, probably by manually overriding the PATH environment variable.

Damon McDougall
Collaborator
$ which sphinx-build
/opt/local/Library/Frameworks/Python.framework/Versions/3.2/bin/sphinx-build
$ python --version
Python 3.2.3
$ python make.py
Running Sphinx v1.1.3
Initializing GitHub plugin
loading pickled environment... done
animation, api, axes_grid, event_handling, misc, mplot3d, old_animation, pylab_examples, tests, units, user_interfaces, widgets, 
building [html]: targets for 566 source files that are out of date
updating environment: 0 added, 53 changed, 0 removed
reading sources... [  5%] api/artist_api                                                                                          
Exception occurred:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/sphinx/ext/inheritance_diagram.py", line 146, in _class_info
    builtins = list(vars(__builtin__).values())
NameError: global name '__builtin__' is not defined
The full traceback has been saved in /var/folders/rs/pqvv7l192xs426r_016501f80000gn/T/sphinx-err-v0lp1g.log, if you want to report the issue to the developers.
Michael Droettboom
Owner

@dmcdougall: You're farther along than I ever got at this point. That traceback is too promising -- it suggests that the inheritance_diagram hasn't yet been updated for Python 3 (I guess). I wonder if disabling that plugin in our conf.py let's it go further, or whether there are just other problems hiding behind that one?

Damon McDougall
Collaborator

@mdboom I'm using all of the changes in this PR. I added some commits earlier today. I also merged the ones from #1361 to keep everything together.

MacPorts doesn't seem to sport a py31/py32 version of sphinx. I have no experience with sphinx. My guess is that support for python 3 is minimal. I'm not sure where to go from here.

Damon McDougall
Collaborator

@mdboom Commenting out the inheritance diagram in conf.py it mostly works. I get the following error, but the doc build still completes!

<autodoc>:0: ERROR: Unknown interpreted text role "function".
/Users/damon/git/github/matplotlib/doc/devel/documenting_mpl.rst:384: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.patches matplotlib.lines matplotlib.text
   :parts: 2
/Users/damon/git/github/matplotlib/doc/devel/transformations.rst:5: ERROR: Unknown directive type "inheritance-diagram".

.. inheritance-diagram:: matplotlib.transforms matplotlib.path
   :parts: 1
/Users/damon/git/github/matplotlib/doc/examples/pylab_examples/multipage_pdf.rst:8: WARNING: Exception occurred in plotting multipage_pdf
 from /Users/damon/git/github/matplotlib/doc/mpl_examples/pylab_examples/multipage_pdf.py:
Traceback (most recent call last):
  File "/Users/damon/python/lib/matplotlib/sphinxext/plot_directive.py", line 515, in run_code
    exec(code, ns)
  File "<string>", line 36
    d['Author'] = u'Jouni K. Sepp\xe4nen'
Damon McDougall
Collaborator

One of them is a unicode issue, because u'asd' is not valid syntax in py3.2.

Michael Droettboom
Owner

It's great that it completes. That unicode issue is probably fixable -- something along the lines of

'Jouni K. Sepp\xe4nen'.decode('latin-1')

If that's the only problem, we're in pretty good shape.

Damon McDougall
Collaborator

I'm also getting this: loading pickled environment... failed: unsupported pickle protocol: 3.

Edit: but the build still completes.

Damon McDougall
Collaborator

Er, nevermind. Turns out the pickling thing gets done to save time on subsequent runs? Since I keep switching between python2 and python3 the pickled environment gets corrupted but it gets regenerated correctly. Disregard my previous comment.

Chris Mayo

I can get past the ascii error with Python 3.2 with an updated version of the gen_gallery.py patch above:

--- doc/sphinxext/gen_gallery.py.orig
+++ doc/sphinxext/gen_gallery.py
@@ -20,7 +20,7 @@
 {%% endblock %%}
 """

-import os, glob, re, sys, warnings
+import io, os, glob, re, sys, warnings
 import matplotlib.image as image

 multiimage = re.compile('(.*?)(_\d\d){1,2}')
@@ -121,18 +121,18 @@

     gallery_path = os.path.join(app.builder.srcdir, '_templates', 'gallery.html')
     if os.path.exists(gallery_path):
-        fh = file(gallery_path, 'r')
+        fh = open(gallery_path, 'r')
         regenerate = fh.read() != content
         fh.close()
     else:
         regenerate = True
     if regenerate:
-        fh = file(gallery_path, 'w')
+        fh = io.open(gallery_path, 'w', encoding='utf8')
         fh.write(content)
         fh.close()

     for key in app.builder.status_iterator(
-        thumbnails.iterkeys(), "generating thumbnails... ",
+        iter(thumbnails.keys()), "generating thumbnails... ",
         length=len(thumbnails)):
         image.thumbnail(key, thumbnails[key], 0.3)

And a new patch to plot_directive.py:

--- lib/matplotlib/sphinxext/plot_directive.py.orig
+++ lib/matplotlib/sphinxext/plot_directive.py
@@ -125,6 +125,7 @@
 """
 from __future__ import print_function

+import io
 import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
        traceback, exceptions

@@ -667,7 +668,7 @@
         else:
             function_name = None

-        with open(source_file_name, 'r') as fd:
+        with io.open(source_file_name, 'r', encoding='utf8') as fd:
             code = fd.read()
         output_base = os.path.basename(source_file_name)
     else:
@@ -809,7 +810,7 @@

     # copy script (if necessary)
     target_name = os.path.join(dest_dir, output_base + source_ext)
-    with open(target_name, 'w') as f:
+    with io.open(target_name, 'w', encoding='utf8') as f:
         if source_file_name == rst_file:
             code_escaped = unescape_doctest(code)
         else:

This now gets me through the html generation and also png creation. Fails now in using graphviz.

Couple of comments on the recent comments.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

Damon McDougall
Collaborator

@cjmayo Cheers for the patch updates. For some reason I don't have a problem with the current ones, but I'll add your updates in since they work for you.

On the 'Jouni K. Sepp\xe4nen' would from __future__ import unicode_literals work as used in examples/pylab_examples/unicode_demo.py?

Probably. I didn't know this existed. I have never worked with python 3, so this whole PR/frustration is a combination of me learning and getting headaches.

Have you tested:

-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]

on Python 2?

Nope. All the testing I did was the generate the documentation under python2, and the build succeeds. I have not checked the output.

Damon McDougall
Collaborator

I am getting a new error building the docs under python 3:

Running Sphinx v1.1.3

Exception occurred:
  File "/Users/damon/python/lib/matplotlib/dates.py", line 122, in <module>
    from dateutil.rrule import rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY, \
  File "/Users/damon/python/lib/dateutil/rrule.py", line 55
    raise ValueError, "Can't create weekday with n == 0"
                    ^
SyntaxError: invalid syntax

I have no idea why this is happening. 2to3 should have fixed this. Argh! I think it's time for me to call it a day and look at this tomorrow.

Chris Mayo

With the last two patches I can now get to the end of the doc building with Python 3.2.3, but I have to use sphinx tip (6bb9dfbdf66e) - sphinx 1.1.3 does not work for me.

Damon McDougall
Collaborator

@cjmayo Progress! Interesting that you need current Sphinx HEAD. What errors do you get with sphinx 1.1.3?

We have two options on proceeding depending on your response to my above question:

1) We may be able to patch our sphinxext files to work with sphinx v1.1.3.
2) If 1) is not possible, then this pull request will fester until the problem is solved upstream.

Michael Droettboom
Owner

At the end of the day, the most important thing is to be able to produce the docs under some version of Python (and Python 2.x is currently working). Having them build under 3.x is of course very desirable, but if that can't be done with a stable version of Sphinx, than there's not much point to putting this is our release either. IMHO, of course.

Damon McDougall
Collaborator

@mdboom Agreed; there are more important v1.2.x bugs currently milestoned. Furthermore, I think if we are waiting in either case the v1.2.x milestone is unnecessary.

I will keep this PR open and remove the milestone. If anybody disagrees then adding it back on is fine.

Allen Riddell

Even with these changes, I'm encountering an import error related to this bug: RuntimeError: dictionary changed size during iteration from colors.py #1401

Phil Elson
Collaborator

Thanks @ariddell - the error you are describing is a matplotlib issue relating to python3.3 to try and keep the issues separate, this build process fix is for the documentation building with a currently supported python (lets say python3.2 for sake of argument). So whilst I think what you highlight is important, it should be addressed as a completely independent issue (#1401).

Cheers,

Phil Elson
Collaborator

@dmcdougall - do you remember the status of this? Is it worth re-visiting and getting this PR merged?

Allen Riddell

I'm getting the error with 3.2.3.

I've setup a clean virtualenv, pip install'd only numpy and git://github.com/dmcdougall/matplotlib.git@py3_docs_2#egg=matplotlib and I get this:

Python 3.2.3 (default, Oct 19 2012, 19:53:16) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.sphinxext.plot_directive
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/__init__.py", line 151, in <module>
    from matplotlib.rcsetup import (defaultParams,
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/rcsetup.py", line 20, in <module>
    from matplotlib.colors import is_color_like
  File "/home/abr/.virtualenvs/matplotlib-py3.2/src/matplotlib/lib/matplotlib/colors.py", line 205, in <module>
    for k, v in cnames.items():
RuntimeError: dictionary changed size during iteration

Python path is: ['', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/distribute-0.6.24-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages/pip-1.1-py3.2.egg', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/lib-dynload', '/usr/lib/python3.2', '/usr/lib/python3.2/plat-linux2', '/home/abr/.virtualenvs/matplotlib-py3.2/lib/python3.2/site-packages']

Hope this helps. Thanks for all your work on this.

Phil Elson
Collaborator

I'm getting the error with 3.2.3.

Ah... Ok - thanks for pointing that out. In that case I think it is very pertinent for this PR.

Cheers,

Phil Elson
Collaborator

@dmcdougall - I'd be happy to merge this. Is there more work to be done before I do, or shall I go ahead and press the green button?

Damon McDougall
Collaborator

@pelson It appears that @cjmayo is currently operational with these changes, and that @ariddell is having issues with that dictionary error. That error, however, appears to have been resolved in the comments in #1401. So, I think we should merge this. If there are further dictionary errors, we can resolve those separately.

Cheers @cjmayo for the help.

Phil Elson pelson merged commit 1d6abc1 into from April 02, 2013
Phil Elson pelson closed this April 02, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
8  doc/conf.py
@@ -174,10 +174,10 @@
174 174
 
175 175
 # Additional stuff for the LaTeX preamble.
176 176
 latex_preamble = """
177  
-   \usepackage{amsmath}
178  
-   \usepackage{amsfonts}
179  
-   \usepackage{amssymb}
180  
-   \usepackage{txfonts}
  177
+   \\usepackage{amsmath}
  178
+   \\usepackage{amsfonts}
  179
+   \\usepackage{amssymb}
  180
+   \\usepackage{txfonts}
181 181
 """
182 182
 
183 183
 # Documents to append as an appendix to all manuals.
8  doc/make.py
@@ -89,20 +89,20 @@ def copytree(src, dst, symlinks=False, ignore=None):
89 89
                 copy2(srcname, dstname)
90 90
         # catch the Error from the recursive copytree so that we can
91 91
         # continue with other files
92  
-        except Error, err:
  92
+        except Error as err:
93 93
             errors.extend(err.args[0])
94  
-        except EnvironmentError, why:
  94
+        except EnvironmentError as why:
95 95
             errors.append((srcname, dstname, str(why)))
96 96
     try:
97 97
         copystat(src, dst)
98  
-    except OSError, why:
  98
+    except OSError as why:
99 99
         if WindowsError is not None and isinstance(why, WindowsError):
100 100
             # Copying file access times may fail on Windows
101 101
             pass
102 102
         else:
103 103
             errors.extend((src, dst, str(why)))
104 104
     if errors:
105  
-        raise Error, errors
  105
+        raise Error(errors)
106 106
 
107 107
 ### End compatibility block for pre-v2.6 ###
108 108
 
6  doc/sphinxext/gen_gallery.py
@@ -121,18 +121,18 @@ def gen_gallery(app, doctree):
19  doc/sphinxext/gen_rst.py
@@ -2,6 +2,7 @@
2 2
 generate the rst files for the examples by iterating over the pylab examples
3 3
 """
4 4
 from __future__ import print_function
  5
+import io
5 6
 import os, glob
6 7
 
7 8
 import os
@@ -37,15 +38,18 @@ def generate_example_rst(app):
37 38
                 continue
38 39
 
39 40
             fullpath = os.path.join(root,fname)
40  
-            contents = file(fullpath).read()
  41
+            if sys.version_info[0] >= 3:
  42
+                contents = io.open(fullpath, encoding='utf8').read()
  43
+            else:
  44
+                contents = io.open(fullpath).read()
41 45
             # indent
42 46
             relpath = os.path.split(root)[-1]
43 47
             datad.setdefault(relpath, []).append((fullpath, fname, contents))
44 48
 
45  
-    subdirs = datad.keys()
  49
+    subdirs = list(datad.keys())
46 50
     subdirs.sort()
47 51
 
48  
-    fhindex = file(os.path.join(exampledir, 'index.rst'), 'w')
  52
+    fhindex = open(os.path.join(exampledir, 'index.rst'), 'w')
49 53
     fhindex.write("""\
50 54
 .. _examples-index:
51 55
 
@@ -77,7 +81,7 @@ def generate_example_rst(app):
77 81
             os.makedirs(outputdir)
78 82
 
79 83
         subdirIndexFile = os.path.join(rstdir, 'index.rst')
80  
-        fhsubdirIndex = file(subdirIndexFile, 'w')
  84
+        fhsubdirIndex = open(subdirIndexFile, 'w')
81 85
         fhindex.write('    %s/index.rst\n\n'%subdir)
82 86
 
83 87
         fhsubdirIndex.write("""\
@@ -122,14 +126,17 @@ def generate_example_rst(app):
122 126
                                   ) and
123 127
                        not noplot_regex.search(contents))
124 128
             if not do_plot:
125  
-                fhstatic = file(outputfile, 'w')
  129
+                fhstatic = open(outputfile, 'w')
126 130
                 fhstatic.write(contents)
127 131
                 fhstatic.close()
128 132
 
129 133
             if not out_of_date(fullpath, outrstfile):
130 134
                 continue
131 135
 
132  
-            fh = file(outrstfile, 'w')
  136
+            if sys.version_info[0] >= 3:
  137
+                fh = io.open(outrstfile, 'w', encoding='utf8')
  138
+            else:
  139
+                fh = io.open(outrstfile, 'w')
133 140
             fh.write('.. _%s-%s:\n\n'%(subdir, basename))
134 141
             title = '%s example code: %s'%(subdir, fname)
135 142
             #title = '<img src=%s> %s example code: %s'%(thumbfile, subdir, fname)
2  lib/matplotlib/sphinxext/mathmpl.py
@@ -65,7 +65,7 @@ def latex2png(latex, filename, fontset='cm'):
65 65
 def latex2html(node, source):
66 66
     inline = isinstance(node.parent, nodes.TextElement)
67 67
     latex = node['latex']
68  
-    name = 'math-%s' % md5(latex).hexdigest()[-10:]
  68
+    name = 'math-%s' % md5(latex.encode()).hexdigest()[-10:]
69 69
 
70 70
     destdir = os.path.join(setup.app.builder.outdir, '_images', 'mathmpl')
71 71
     if not os.path.exists(destdir):
4  lib/matplotlib/sphinxext/plot_directive.py
@@ -125,8 +125,8 @@
125 125
 """
126 126
 from __future__ import print_function
127 127
 
128  
-import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap, \
129  
-       traceback, exceptions
  128
+import sys, os, glob, shutil, imp, warnings, cStringIO, re, textwrap
  129
+import traceback
130 130
 
131 131
 from docutils.parsers.rst import directives
132 132
 from docutils import nodes
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.