upgrade sphinx to version 1.0.4

[sagetrac-mvngu]

As the subject says. See this sage-devel thread for some background on why we need to upgrade Sphinx.

Apply: (starting from sage-4.6.1.alpha1)

1. The patch package http://sage.math.washington.edu/home/jdemeyer/spkg/patch-2.5.9.spkg (see Add GNU patch 2.5.9 as a standard package. #9418)
2. The Sphinx package at http://sage.math.washington.edu/home/jdemeyer/spkg/sphinx-1.0.4.p3.spkg
3. Apply attachment: 10118_sphinx_sagelib.2.patch to the Sage library.
4. Apply attachment: trac-10118_sagenb.patch to the Sage notebook library.

See also #10290 (Upgrade Pygments to version 1.3.1) for syntax highlighting to work properly.

Follow-up (p4 spkg): #10350

CC: @hivert @jhpalmieri @sagetrac-sage-combinat @novoselt

Component: packages: standard

Keywords: sphinx spkg

Author: Minh Van Nguyen, Jeroen Demeyer

Reviewer: Jeroen Demeyer, Minh Van Nguyen

Merged: sage-4.6.1.alpha2

Issue created by migration from https://trac.sagemath.org/ticket/10118

[qed777]

comment:1

I think Florent Hivert mentioned possibly working on the Sphinx and Pygments packages. Florent, did you happen to start upgrading them?

[kiwifb]

comment:3

Sphinx-1.0.4 works well in sage-4.5.3 and 4.6alpha3 the only problem is it break
the following test:

sage -t -force_lib "/usr/lib/python2.6/site-packages/sagenb/misc/sphinxify.py"
File "/usr/lib/python2.6/site-packages/sagenb/misc/sphinxify.py", line 69:
    sage: sphinxify('A test')
Expected:
    '<div class="docstring">\n    \n  <p>A test</p>\n\n\n</div>'
Got:
    '\n<div class="docstring">\n    \n  <p>A test</p>\n\n\n</div>'
**********************************************************************
File "/usr/lib/python2.6/site-packages/sagenb/misc/sphinxify.py", line 71:
    sage: sphinxify('**Testing**\n`monospace`')
Expected:
    '<div class="docstring"...<strong>Testing</strong>\n<span class="math"...</p>\n\n\n</div>'
Got:
    '\n<div class="docstring">\n    \n  <p><strong>Testing</strong>\n<span class="math">monospace</span></p>\n\n\n</div>'
**********************************************************************
File "/usr/lib/python2.6/site-packages/sagenb/misc/sphinxify.py", line 73:
    sage: sphinxify('`x=y`')
Expected:
    '<div class="docstring">\n    \n  <p><span class="math">x=y</span></p>\n\n\n</div>'
Got:
    '\n<div class="docstring">\n    \n  <p><span class="math">x=y</span></p>\n\n\n</div>'File "/usr/lib/python2.6/site-packages/sagenb/misc/sphinxify.py", line 69:
    sage: sphinxify('A test')
Expected:
    '<div class="docstring">\n    \n  <p>A test</p>\n\n\n</div>'
Got:
    '\n<div class="docstring">\n    \n  <p>A test</p>\n\n\n</div>'
**********************************************************************
File "/usr/lib/python2.6/site-packages/sagenb/misc/sphinxify.py", line 71:
    sage: sphinxify('**Testing**\n`monospace`')
Expected:
    '<div class="docstring"...<strong>Testing</strong>\n<span class="math"...</p>\n\n\n</div>'
Got:
    '\n<div class="docstring">\n    \n  <p><strong>Testing</strong>\n<span class="math">monospace</span></p>\n\n\n</div>'
**********************************************************************
File "/usr/lib/python2.6/site-packages/sagenb/misc/sphinxify.py", line 73:
    sage: sphinxify('`x=y`')
Expected:
    '<div class="docstring">\n    \n  <p><span class="math">x=y</span></p>\n\n\n</div>'
Got:
    '\n<div class="docstring">\n    \n  <p><span class="math">x=y</span></p>\n\n\n</div>'

[hivert]

comment:4

Replying to @qed777:

I think Florent Hivert mentioned possibly working on the Sphinx and Pygments packages. Florent, did you happen to start upgrading them?

Hi Mitesh,

I didn't had the time to work on it (and even more generally on sage) recently. My previous attempt goes back to July with version 1.0.1 and showed some difficult hard stuff but Sphinx itself seemed to bu the cause. This is very cool if they are solved with 1.0.4. There is a chance that I can find some time in the next two weeks. However, if Francois beat me, I won't complain :-)
The broken tests seem so be very easy to solve. My plan is then to solve #9128 (Sphinx should be aware of all.py to find its links).

[kiwifb]

comment:5

Well the situation for me is that on sage-on-gentoo we use system packages. And sphinx 1.0.4 has been marked stable so all sage-on-gentoo users were automatically updated to it. We test a number of things that way, if there is a real problem we pin sage to use a particular version.

I won't beat you to it, it's kind of crunch time here (the semester and the year is ending in New Zealand with all that implies in terms of marking).

[dandrake]

comment:6

Can someone post a spkg for this? I can try to review, but I need something to review...

[sagetrac-mvngu]

comment:7

Replying to @dandrake:

Can someone post a spkg for this? I can try to review, but I need something to review...

I'm currently (just taking a break to answer your comment) working on an upgraded Sphinx package.

[sagetrac-mvngu]

Attachment: trac-10118_sphinx-spkg.patch.gz

[sagetrac-mvngu]

Attachment: trac-10118_sage-lib.patch.gz

Attachment: trac-10118_sagenb.patch.gz

[sagetrac-mvngu]

Author: Minh Van Nguyen

[sagetrac-mvngu]

comment:8

See the ticket description for instructions on which patch/spkg to apply. Here are the results of my testing.

• bsd.math -- Both the HTML and PDF versions of the documentation built OK.

• cicero.skynet -- The HTML version of the documentation built OK, with numerous warnings regarding a missing LaTeX installation. This also meant that I can't build the PDF version of the documentation.

• cleo.skynet -- Sage 4.6.1.alpha0 fails to build on this machine, hence can't test building the documentation.

• eno.skynet -- The HTML version of the documentation built OK, with numerous warnings regarding a missing LaTeX installation. This also meant that I can't build the PDF version of the documentation.

• flavius.skynet -- The PDF version of the documentation built fine. The HTML version built with numerous warnings due to missing the dvipng command.

• gcc11 -- The HTML documentation built OK, with some warnings due to an incomplete LaTeX installation. Can't build the PDF version of the documentation due to this reason.

• gcc16 -- The HTML documentation built OK, with some warnings due to an incomplete LaTeX installation. Can't build the PDF version of the documentation due to this reason.

• hawk (David Kirkby's OpenSolaris box) -- Both the PDF and HTML versions of the documentation built fine.

• iras.skynet -- Sage 4.6.1.alpha0 fails to build on this machine, hence can't test building the documentation.

• lena.skynet -- The HTML version of the documentation built OK, with numerous warnings about a missing LaTeX installation, hence cannot parse math expressions typeset using LaTeX. The missing LaTeX installation also prevents us from building the PDF version of the documentation.

• rh.math -- Has a LaTeX installation, but missing utf8x.def, hence unable to successfully build the PDF version of the documentation. The HTML version of the documentation built with numerous warnings due to missing utf8x.def.

• rosemary.math -- Sage 4.6.1.alpha0 fails to build on this machine, hence can't test building the standard documentation.

• sage.math -- Both the HTML and PDF versions of the documentation built fine.

• sextus.skynet -- The HTML version of the documentation built OK, with numerous warnings about a missing LaTeX installation, hence cannot parse math expressions typeset using LaTeX. The missing LaTeX installation also prevents us from building the PDF version of the documentation.

• taurus.skynet -- The HTML documentation built fine. This machine doesn't have a LaTeX installation, so I can't build the PDF version of the documentation.

What the above tests mean is that the HTML documentation can be built without a LaTeX installation, but the resulting documentation would not render LaTeX typeset math expressions. For best result, you should install a LaTeX distribution on your system before building the Sage documentation. Of course, when you compile Sage, the configure script that is first run does warn about having a LaTeX installation if your system doesn't already have a LaTeX installation.

I'm CC'ing the sage-combinat team because attachment: trac-10118_sage-lib.patch touches files in the combinatorics module.

[qed777]

comment:10

You should be able to build 4.6.1.alpha0 on cleo with export SAGE_DEBUG=yes (#9897, #10120). This should also help on iras, but you'll likely also have problems with cddlib (#10204).

By the way, do you think we could add rosemary.math as a Sage Buildbot buildslave?

[sagetrac-mvngu]

comment:11

Replying to @qed777:

By the way, do you think we could add rosemary.math as a Sage Buildbot buildslave?

See the following sage-devel thread for a follow-up to your question:

http://groups.google.com/group/sage-devel/browse_thread/thread/7cf390f110f6cea6

[jdemeyer]

Changed keywords from none to sphinx spkg

[jdemeyer]

comment:12

Replying to @sagetrac-mvngu:

• eno.skynet -- The HTML version of the documentation built OK, with numerous warnings regarding a missing LaTeX installation. This also meant that I can't build the PDF version of the documentation.

If possible, I would like to see

• See the resulting HTML of a Sphinx run without LaTeX.
• The log file of running Sphinx without LaTeX.

Would it make sense for make doc-html to first test whether latex is installed? If there is no latex, give an error and don't build the documentation.

[sagetrac-mvngu]

Attachment: dochtml.log

[sagetrac-mvngu]

comment:13

Attachment: docpdf.log

Replying to @jdemeyer:

If possible, I would like to see

• See the resulting HTML of a Sphinx run without LaTeX.

See the URL

http://mvngu.googlecode.com/hg/10118-sphinx/en/index.html

• The log file of running Sphinx without LaTeX.

See the log files:

1. attachment: dochtml.log
2. attachment: docpdf.log

Would it make sense for make doc-html to first test whether latex is installed? If there is no latex, give an error and don't build the documentation.

That would result in a user, whose system doesn't have LaTeX but can compile Sage OK, without at least a way to build the vast majority of the Sage standard documentation. Currently, on a system without LaTeX you could still build the HTML version of the Sage documentation, but any expressions typeset in LaTeX could not be rendered using LaTeX. This means that the documentation does build, but you won't be able to see LaTeX math expressions typeset beautifully; you can still read the corresponding LaTeX typesetting, though. However, if you build the HTML version with jsMath rendering of mathematical expressions, then you don't need LaTeX at all. The only case where you really need LaTeX is when you build the PDF version of the documentation.

So LaTeX is not necessary for building the Sage documentation. But if you don't have LaTeX, there's very little reason why the documentation builder should abort and refuse to build the HTML version of the documentation. The resulting "broken" HTML built documentation is still very readable; you just don't get beautifully typeset math expressions. And if you have jsMath installed, build the documentation with jsMath.

[jdemeyer]

comment:14

Replying to @sagetrac-mvngu:

Replying to @jdemeyer:

If possible, I would like to see

• See the resulting HTML of a Sphinx run without LaTeX.

http://mvngu.googlecode.com/hg/10118-sphinx/en/index.html

Thanks! The output does indeed look quite readable without LaTeX installed.

[jdemeyer]

comment:15

I'm all in favour of making the pdf documentation actually work!
I am preparing a .p1 package with some minor changes, hang on...

[jdemeyer]

Reviewer: Jeroen Demeyer

[jdemeyer]

Attachment: sphinx-1.0.4.p1.patch.gz

Reviewer patch, p0 to p1

[jdemeyer]

comment:16

Minh, your patches look fine, I made some minor changes to Makefile and spkg-install.

New spkg: http://sage.math.washington.edu/home/jdemeyer/spkg/sphinx-1.0.4.p1.spkg

spkg patch: attachment: sphinx-1.0.4.p1.patch

Works on sage.math.washington.edu, it would probably be good to test on other machines too.

[jdemeyer]

comment:34

Replying to @jhpalmieri:

Oh, sorry, my mistake. The line I mentioned does get used when producing the latex/PDF docs, but not the html. For the html docs, the file pngmath.py (in sphinx/ext) has been changed from the old version of Sphinx to use utf8x instead of utf8. We could try a patch like this:

--- ../src/sphinx/ext/pngmath.py	2010-07-24 03:07:36.000000000 -0700
+++ pngmath.py	2010-11-17 10:20:51.000000000 -0800
@@ -34,7 +34,7 @@
 
 DOC_HEAD = r'''
 \documentclass[12pt]{article}
-\usepackage[utf8x]{inputenc}
+\usepackage[utf8]{inputenc}
 \usepackage{amsmath}
 \usepackage{amsthm}
 \usepackage{amssymb}

It also might be worth investigating why the Sphinx people changed this in the first place.

Sounds like a good idea. I will try it when I have time.

[jdemeyer]

Attachment: 10118_sphinx_sagelib.patch.gz

[jdemeyer]

Attachment: 10118_sphinx_sagelib.2.patch.gz

Sage library patch, replaces trac-10118_sage-lib.patch

[jdemeyer]

comment:37

Minh, I removed your hunk removing (`this link <../../installation/index.html>`_ should take you to a local copy of the installation guide). because

1. I like the local link, I think we should try to keep it.
2. There are other places in the documentation with the same problem, so we might as well fix them all at once in one ticket.

[jdemeyer]

Changed author from Minh Van Nguyen to Minh Van Nguyen, Jeroen Demeyer

[jdemeyer]

Merged: sage-4.6.1.alpha2

[jdemeyer]

Attachment: sphinx-1.0.4.p3.patch.gz

Patch from p0 to p3, for review

[sagetrac-mvngu]

comment:39

Attachment: trac-10118_use-utf8.patch.gz

Here are some trivial problems in the file SPKG.txt of version ".p3" of the Sphinx package:

• In the section

== Dependencies ==

This depends on Jinja >= 2, Pygments >= 0.8, and docutils >= 0.4.

we should have "Pygments >= 1.3.1" due to ticket #10290. But this is no biggy; it's a trivial typo that can either (i) be fixed in a new ticket, or (ii) fixed in the current ticket.

• In the section

* patches/pngmath.patch: This replaces \usepackage[utf8x]{inputenc}
  by \usepackage[utf8]{inputenc} in the LaTeX preamble for building
  images in the HTML documentation.  This change is done because some
  LaTeX installations have only utf8, not utf8x (utf8x adds support for
  much more Unicode characters, but these are needed used to typeset the
  mathematics in the Sage documentation).

the fragment

but these are needed used to typeset the

should be changed to

but these are used to typeset the

Tested on the following platforms:

• {sage.math, bsd.math, hawk}: Both HTML and PDF versions of documentation built OK. For the PDF version, building the reference manual produced some warnings about "unusable reference target", which are to do with relative links to other documents in the Sage standard documentation.

• {cleo, iras}.skynet: Sage 4.6.1.alpha2 fails to build on this machine, hence the documentation doesn't get built at all.

• {cicero, eno, lena, sextus, taurus, t2}: The HTML version of the documentation mostly built OK, but with warnings about a missing LaTeX installation. Consequently, the PDF version of the documentation can't be built.

• flavius.skynet -- The PDF version of the documentation built fine, but with some warnings about unusable relative links. The HTML version built with numerous warnings due to missing the dvipng command.

• {gcc11, gcc16}.fsffrance.org: The HTML version of all documents in the standard documentation mostly built OK, but with warnings about the missing command "dvipng". This machine has a LaTeX installation, but it is missing the file titlesec.sty, hence building the PDF version of any document in the standard documentation would hang at the error message:

Style option: `fancybox' v1.3 <2000/09/19> (tvz)
)

! LaTeX Error: File `titlesec.sty' not found.

Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)

• rh.math: The HTML version of all documents in the standard documentation mostly built OK, but with warnings about the missing command "dvipng". This means that LaTeX typesetting does not render at all in the HTML version. This machine has a LaTeX installation, but it is missing the file utf8x.def, hence building any document in the standard documentation would hang at the error message that prompts for utf8x.def. Note that we have the patch

diff -r -u src.old/sphinx/ext/pngmath.py src/sphinx/ext/pngmath.py              
--- src/sphinx/ext/pngmath.py   2010-07-24 12:07:36.000000000 +0200             
+++ src/sphinx/ext/pngmath.py   2010-11-18 09:38:05.428635584 +0100             
@@ -34,7 +34,7 @@
                                                                                
 DOC_HEAD = r'''                                                                
 \documentclass[12pt]{article}                                                  
-\usepackage[utf8x]{inputenc}                                                   
+\usepackage[utf8]{inputenc}                                                    
 \usepackage{amsmath}                                                           
 \usepackage{amsthm}                                                            
 \usepackage{amssymb}

to use only utf8 instead of utf8x. The same change needs to be made to the file doc/common/conf.py. See my reviewer patch attachment: trac-10118_use-utf8.patch. With this patch, we get pass the issue of the missing utf8x.def, but we now run into the following error:

Style option: `fancybox' v1.3 <2000/09/19> (tvz)
)

! LaTeX Error: File `titlesec.sty' not found.

Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)

So here are two choices:

1. Accept sphinx-1.0.4.p3.spkg as is. I'm OK with that package. The issues I pointed out above can be addressed in a follow-up ticket.
2. Fix the above issues in the current ticket.

[jdemeyer]

comment:40

Since this ticket is already merged, see #10350.

[jdemeyer]

comment:41

Since Minh agreed on this spkg, I'm setting this to positive_review. Further changes are at #10350.

[jdemeyer]

Changed reviewer from Jeroen Demeyer to Jeroen Demeyer, Minh Van Nguyen

[jhpalmieri]

comment:42

Regarding the last patch (trac-10118_use-utf8.patch):

• this is a bit of a regression: see Sage 4.3.1 reference manual: PDF version failed to build due to non-ASCII characters in docstring #8036. Right now, the problematic string from that ticket is not in the reference manual, but if the file schemes/elliptic_curves/BSD.py is added to the reference manual, then the pdf documentation won't build.

• with this patch, I think that fr/tutorial doesn't build in pdf format. Could someone else please check this?

• by default, Sphinx uses utf8, so if this patch worked, I think you could actually just comment out the line (in conf.py)

latex_elements['inputenc'] = '\\usepackage[utf8x]{inputenc}'

rather than replace utf8x with utf8.

[jdemeyer]

comment:43

I agree with John. Let's not apply the reviewer patch (it is only needed to build the pdf documentation, so I don't think there is a big issue).

[nexttime]

comment:45

In principle needs work:

Sphinx attempts to download Jinja2 during build, see the follow-up ticket #10350.

[nexttime]

comment:46

Replying to @nexttime:

In principle needs work:

Sphinx attempts to download Jinja2 during build, see the follow-up ticket #10350.

An upgraded Jinja2 spkg (2.2.5.p0), avoiding the download of a version >=2.2, can be found at #10423; needs review.