update Sphinx to version 0.6.2 or 0.6.3

[jhpalmieri]

As the summary says. The spkg can be found at

http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg

After installing it and before building any docs, you will probably need to delete the directory SAGE_ROOT/devel/sage/doc/output -- this holds cached information which may not be compatible between different versions of Sphinx.

Sphinx 0.6.3 also uses Jinja2. Sage includes Jinja, not Jinja2. As long as you're connected to a network, the installation of Sphinx 0.6.3 downloads Jinja2 and installs it -- it can coexist with Jinja, so it shouldn't mess anything up. If you're not connected to a network, and for putting in the Sage tar ball, there is an spkg for Jinja2:

http://sage.math.washington.edu/home/palmieri/SPKG/jinja2-2.1.1.p0.spkg

For producing a new Sage tarball, since we're adding a new spkg, I think we need to modify the files spkg/install and spkg/standard/deps; I've attached new versions of these.

Finally, the patch below ("inheritance_example.patch") is not required. It provides an illustration of a new capability of Sphinx 0.6.3 (and I think it's quite nice), but it's not something that needs to be installed or reviewed for this ticket.

CC: @mwhansen @sagetrac-sage-combinat

Component: packages: standard

Author: John Palmieri

Reviewer: John Cremona, Tim Dumol, Mitesh Patel

Merged: sage-4.2.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/6586

[jhpalmieri]

comment:1

Oh, I forgot: you need to apply the patch at #6585 for the reference manual to build. This version of Sphinx also prints a lot of new warnings, too.

[sagetrac-mvngu]

comment:2

I installed the updated SPKG with

./sage -f http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p0.spkg

and applied the patch at #6585. When doing

./sage -docbuild reference html

I got the following errors:

mvngu@sage sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux]$ ./sage -docbuild reference html
Traceback (most recent call last):
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 667, in <module>
    getattr(get_builder(name), type)()
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 348, in _wrapper
    for module_name in self.get_modified_modules():
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 412, in get_modified_modules
    env = self.get_sphinx_environment()
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 403, in get_sphinx_environment
    return BuildEnvironment.frompickle(config, env_pickle)
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/local/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/environment.py", line 210, in frompickle
    env = pickle.load(picklefile)
AttributeError: 'module' object has no attribute 'RedirStream'

[jhpalmieri]

comment:3

I think I got this error, too. Once I deleted doc/output, things worked for me. (I think there is something stored in the old version of the documentation which confuses the new version of Sphinx.)

[sagetrac-mvngu]

comment:5

The upgraded Sphinx SPKG can't handle binary encodings in the file:

sage/combinat/partition.py

In particular, the following sections from that source file:

line:2233

sage: p = loads('x\x9c\x85\x8e\xbd\xaa\xc2@\x10\x85\x89\xff.>\xc4\x94\xda\x04\x15|\x04\xb1\xb1\x90\x0b[\x87I\x18\x935\xc9\xae\xb33\xda\t\xd7\xc2\xf76"biw\x0e\x9c\x9f\xef\xbfW\x08\x96\x94\x16\xa1\xcd\x9dGM\xcf\x18\xd5\xa9\x0b\xde\x1c>Jv\x91PIt\xbf\xcd|m8Y\xdc\xb9w\xe3\xfe\xdc&\xf5\xbb\x1d\x9d/%u^\xa9\xa4hZ\xac)\xfb\x18\x1e\xd8d\xfd\xf8\xe3\xa1\x1df\x1e[\xe2\x91\xdd|\x97!\x1ca\xb5\x84\n\xaf\xdd\x02\xbc\xbe\x05\x1a\x12\x01\xad\xd0C\x88@|\xc1\x064\xc0\x9a\xc7v\x16\xf2\x13\x15\x9a\x15\r\x8a\xf0\xe47\xf9;ixj\x13_u \xd8\x81\x98K\x9e>\x01\x13iVH')

line:2294

sage: p = loads('x\x9c\x85\x8e\xbd\x0e\x820\x14\x85\x03\x8a?\x8d\x0f\xd1Q\x97\x06y\x07\xe3\xaa&\x9d\xc9\xa5\xb9\x96\n\xb4^Z\xdcLt\xf0\xbd\xc5 qt;\'9?\xdf#V\x1e4\n\xe5\x9a\xc2X\x08\xe2\nm0\xc18\xcb\x0e\xa3\xf2\xfb\x16!\xa0\x0f\xbbcn+F\xd1\xe6I\xf1\x9d&k\x19UC\xbb5V{al@\x8d-k\xa0\xc2|44\x95Q\xf6:Q"\x93\xdcB\x834\x93\xe9o\x99\xbb3\xdf\xa6\xbc\x84[\xbf\xc0\xf5\xf7\x87\x7f 8R\x075\x0f\x8eg4\x97+W\\P\x85\\\xd5\xe0=-\xfeC\x0fIFK\x19\xd9\xb2g\x80\x9e\x81u\x85x\x03w\x0eT\xb1')

line:2839

sage: dmp = 'x\x9ck`J.NLO\xd5K\xce\xcfM\xca\xccK,\xd1+H,*\xc9,\xc9\xcc\xcf\xe3\n\x80\xb1\x8a\xe3\x93\x81DIQbf^I1W!\xa3fc!Sm!\xb3F(7\x92x!Km!k(GnbE<\xc8\x88B6\x88\xb9E\x99y\xe9\xc5z@\x05\xa9\xe9\xa9E\\\xb9\x89\xd9\xa9\xf10N!{(\xa3QkP!Gq(c^\x06\x90c\x0c\xe4p\x96&\xe9\x01\x00\xc2\xe53\xfd'

When these lines are removed, the HTML version of the reference manual builds OK. So a possible fix for line 2233 is to replace everything in the TEST block starting from that line by:

sage: p = PartitionsGreatestLE(10,2)
sage: p == loads(dumps(p))
True

We can do something similar for line 2294; replace everything in the TEST block starting from line 2294 by:

sage: p = PartitionsGreatestEQ(10,2)
sage: p == loads(dumps(p))
True

As for the TEST block starting from line 2839, I have no idea how to fix it so that the reference manual would successfully builds. Someone on the sage-combinat team to the rescue?

[jhpalmieri]

comment:6

Replying to @jhpalmieri:

Oh, I forgot: you need to apply the patch at #6585 for the reference manual to build. This version of Sphinx also prints a lot of new warnings, too.

I think that this patch fixes the issue with binary encodings; at least it did for me.

[sagetrac-mvngu]

comment:7

With the patch at #6585 and the SPKG, building the reference manual on sage.math takes much longer than previously.

[jhpalmieri]

comment:8

Replying to @sagetrac-mvngu:

With the patch at #6585 and the SPKG, building the reference manual on sage.math takes much longer than previously.

I agree: on my mac, it took 28 minutes to build the html version with the old one, and 43 minutes with the new one. With the --jsmath option on, the old one took 6 minutes and the new one took 15 minutes. I was getting similar comparisons on sage.math. (This is with me deleting devel/sage/doc/output between each build, to level the playing field.)

It's strange: for the tutorial and developer's guide, building is about the same or slightly faster with the new version. I also made a modified version of the developer's guide by duplicating some sections (so that the resulting PDF file was 559 pages); the new version was slightly faster there, too. Maybe it's all of the math in the reference manual? Maybe it's the large number of files? Maybe it's the autogeneration of files?

[qed777]

comment:9

Cursory profiling with #6187's "testreference" target indicates that the rewritten autodoc extension is significantly slower than before. I think autodoc now also uses far more memory. Attempting to build the full reference manual on my machine, Sphinx "stalled" at 97% of modules read. I think the kernel was too busy swapping everything but Sphinx to disk.

Perhaps there's a more efficient, parallelizable representation. Can we subclass and/or Cythonize sphinx.ext.autodoc.Documenter?

[qed777]

comment:10

Minor progress: It seems that a custom autodoc-skip-member method now overrides Sphinx's default behavior, which is to skip private methods (__init__, etc.). To keep these out of the reference manual, we might use, e.g.,

def skip_NestedClass(app, what, name, obj, skip, options):
    """docstring"""
    skip_nested = str(obj).find("sage.misc.misc") != -1 and name.find("MainClass.NestedClass") != -1
    return skip or skip_nested

in conf.py. Unfortunately, this does not significantly reduce the build time or memory usage.

[qed777]

comment:11

Apparently, the culprit is sphinx.pycode's Python source analyzer, which sphinx.ext.autodoc uses to extract attribute docstrings. From Sphinx-0.6.2-py2.6.egg/sphinx/pycode/__init__.py:

    The docstrings can either be in special '#:' comments before the assignment
    or in a docstring after it.

According to the analyzer, there are just two places to find such docstrings in the Sage library:

• combinat/designs/block_design.py
• schemes/elliptic_curves/ell_field.py
  Still, resource use is high, because there is a lot of caching going on. We can reduce it with weak references:

--- __init__.py.orig    2009-08-01 09:58:27.303927239 -0700
+++ __init__.py 2009-08-01 11:35:17.477865562 -0700
@@ -120,10 +120,11 @@ class PycodeError(Exception):
             res += ' (exception was: %r)' % self.args[1]
         return res
 
-
+import weakref
 class ModuleAnalyzer(object):
     # cache for analyzer objects -- caches both by module and file name
-    cache = {}
+#    cache = {}
+    cache = weakref.WeakValueDictionary()
 
     @classmethod
     def for_string(cls, string, modname, srcname='<string>'):

This does reduce build time and memory usage, but I don't know if the output is identical.

[qed777]

comment:12

For sage -docbuild reference html --jsmath, at least, the results are close:

$ du -s output_v0*
95550   output_v051
97392   output_v062
97388   output_v062_mod

The differences are in environmental.pickle or stem from using a function as a default argument in

• combinat/words/word_content.py
• rings/polynomial/groebner_fan.py

[qed777]

comment:13

Clearly, it's time for a break.

[jhpalmieri]

comment:14

Nice observation. Making the change with weak_references does indeed speed things up:

0.5.1     28 minutes
0.6.2     43 minutes
0.6.2     33 minutes   (version using weakref)

With no other changes, it includes all of the private methods, as already pointed out (and I assume it did before using weakref -- this may account for some of the increased time, since it's producing more output), but it also includes (for some classes, anyway), an entry for __weakref)_. See the documentation for sage/algebras/steenrod_algebra.py, for instance: it lists as an attribute of SteenrodAlgebraFactory

__weakref__
   list of weak references to the object (if defined)

[qed777]

comment:15

Actually, noticing __weakref__ for that module is what led to the comment about autodoc-skip-member. I think this was before I "patched" Sphinx, but I may be mistaken. I'm also not sure about why __weakref__ appears spontaneously. Perhaps a super class?

[qed777]

comment:16

See #6664 for the autodoc-skip-member issue.

[qed777]

comment:17

Also, I've asked about the merits of the weakref workaround on sphinx-dev.

[qed777]

Example inheritance diagram for "The Steenrod algebra". This is just for fun.

[qed777]

comment:18

Attachment: inheritance_example.patch.gz

The new inheritance diagram extension has potential. I've attached a minimal example. Please do not merge it.

The options are limited, however. For example, I could not find a directive that auto-generates a diagram for each module. Also, it seems there's no way to color-code the nodes/classes, e.g., to indicate their modular provenance. Maybe it's possible to do one or more of these ourselves, in builder.py, conf.py, or by patching Sphinx.

[qed777]

comment:19

I should add that this requires a local installation of GraphViz.

[jhpalmieri]

Attachment: deps.gz

diff between old and new install

[jhpalmieri]

Attachment: install.patch.gz

Attachment: deps.patch.gz

diff between old and new deps

[qed777]

comment:52

Both the latest Sphinx and Jinja2 packages appear to install without network activity. The changes to install and deps look good to me, but I haven't done a complete installation from scratch.

To the extent that it counts, my review is positive.

[TimDumol]

Changed reviewer from Mitesh Patel to Mitesh Patel, Tim Dumol

[TimDumol]

comment:53

Documentation builds perfectly. Sphinx 0.6.3 takes only 2 minutes more sys time than 0.5.1 to build all the docs, for me.

Haven't done a complete install yet, but I'll do so once I finish downloading the 4.1.2.alpha2 tarball.

[qed777]

Changed reviewer from Mitesh Patel, Tim Dumol to John Cremona, Tim Dumol, Mitesh Patel

[qed777]

comment:54

I updated spkg/install and spkg/standard/deps, replaced the old Sphinx spkg with the new one in spkg/standard, and copied the Jinja2 package to spkg/standard. With these changes, sage -bdist 100 generates the expected binary distributions on sage.math from both binary and compiled source distributions. Moreover, sage -sdist 200 makes a source distribution that compiles as expected.

There's a failed test:

mpatel@sage sage-200$ ./sage -t devel/sage/sage/misc/trace.py 
sage -t  "devel/sage/sage/misc/trace.py"                    
**********************************************************************
File "/scratch/mpatel/source/sage-200/devel/sage/sage/misc/trace.py", line 61:
    sage: print s.before[s.before.find('-'):]
Expected:
    ---...
    ipdb> c
    2 * 5
Got:
    --Call--
    > /scratch/mpatel/source/sage-200/local/lib/python2.6/site-packages/sage/rings/arith.py(1944)factor()
       1943 
    -> 1944 def factor(n, proof=None, int_=False, algorithm='pari', verbose=0, **kwds):
       1945     """
    
    ipdb> c
    2 * 5
    <BLANKLINE>
**********************************************************************
1 items had failures:
   1 of  10 in __main__.example_1
***Test Failed*** 1 failures.
For whitespace errors, see the file /scratch/mpatel/source/sage-200/tmp/.doctest_trace.py
         [3.7 s]
exit code: 1024
 
----------------------------------------------------------------------
The following tests failed:


        sage -t  "devel/sage/sage/misc/trace.py"
Total time for all tests: 3.7 seconds

I think it's not related to this ticket.

[qed777]

comment:55

Problem: Somehow, Sphinx 0.6.3 breaks the sidebar toggle (cf. #6507) in the live docs. In Firebug's console:

unexpected end of XML source
<span class="math">('#searchbox').show(0);

[qed777]

comment:56

Patching Sphinx 0.6.3 should fix this and perhaps also a similar problem reported on sage-devel:

--- sphinx/themes/basic/layout.html.orig        2009-10-04 17:07:16.000000000 -0700
+++ sphinx/themes/basic/layout.html     2009-10-04 17:07:28.000000000 -0700
@@ -81,7 +81,7 @@
               {{ _('Enter search terms or a module, class or function name.') }}
               </p>
           </div>
-          <script type="text/javascript">$('#searchbox').show(0);</script>
+          <script type="text/javascript">jQuery('#searchbox').show(0);</script>
           {%- endif %}
           {%- endblock %}
         </div>

[qed777]

comment:57

Or perhaps not.

[TimDumol]

comment:58

mpatel: The problem with the sidebar is caused by compression. Line comments ("//") are used in the sidebar code, therefore when the code is compressed, the entire function is commented out. Changing the line comments to block comments ("/* */") fixed the problem for me.

[TimDumol]

comment:59

A patch for the doc sidebar problem is now up at #7126.

[qed777]

comment:60

If this comment at #7126 is correct, we should still patch sphinx/themes/basic/layout.html. I can do this.

[TimDumol]

comment:61

Replying to @qed777:

Problem: Somehow, Sphinx 0.6.3 breaks the sidebar toggle (cf. #6507) in the live docs. In Firebug's console:

unexpected end of XML source
<span class="math">('#searchbox').show(0);

#7141 adds a patch that fixes this.

[mwhansen]

comment:62

Merged both the jinja2 and sphinx in spkg.

[mwhansen]

Merged: sage-4.2.alpha0