wrong cross-referencing in modindex of documentation

[haraldschilly]

The Sphinx-generated documentation has index files for modules. E.g. looking at http://doc.sagemath.org/html/en/a_tour_of_sage/py-modindex.html

Checking upon the first link in this list, it points to a non-existing page. It seems like as if sphinx doesn't correctly cross link to a sibling. I.e. instead of /html/en/a_tour_of_sage/...

http://doc.sagemath.org/html/en/a_tour_of_sage/algebras/sage/algebras/affine_nil_temperley_lieb.html#module-sage.algebras.affine_nil_temperley_lieb

it should be /html/en/reference/...

http://doc.sagemath.org/html/en/reference/algebras/sage/algebras/affine_nil_temperley_lieb.html#module-sage.algebras.affine_nil_temperley_lieb

Upstream (Sphinx) fix:

• pull request: Fix bug with copying of initial_data for Domains sphinx-doc/sphinx#2816
• released in: Sphinx 1.4.6
• corresponding Sphinx upgrade in Sage: Upgrade to Sphinx 1.5.x #22252

Upstream: Fixed upstream, in a later stable release.

CC: @paulmasson @embray @jdemeyer @hivert @egourgoulhon @slel

Component: documentation

Author: Erik Bray

Branch: dbfb2d6

Reviewer: Paul Masson

Issue created by migration from https://trac.sagemath.org/ticket/21044

[embray]

comment:3

As I found on the mailing list, I encountered this problem in the doctests, with an incorrect link being generated for sage.rings.padics.tutorial.

This behavior can be explained in part by this comment in the sphinx.ext.intersphinx source:

        # Duplicate values in different inventories will shadow each
        # other; which one will override which can vary between builds
        # since they are specified using an unordered dict.  To make
        # it more consistent, we sort the named inventories and then
        # add the unnamed inventories last.  This means that the
        # unnamed inventories will shadow the named ones but the named
        # ones can still be accessed when the name is specified.

In my specific case, entries for sage.rings.padics.tutorial are showing up in the objects.inv for each of

/home/embray/src/sagemath/sage-cygwin/local/share/doc/sage/html/en/reference/plotting
/home/embray/src/sagemath/sage-cygwin/local/share/doc/sage/html/en/reference/padics
/home/embray/src/sagemath/sage-cygwin/local/share/doc/sage/html/en/reference/plot3d	

Since these are stored as keys in a dict the order can nondeterministic. In my case plot3d ends up being last.

What's still mysterious is why sage.rings.padics.tutorials would even have reference in the plotting or plot3d docs' objects.inv. Neither of those documents even reference it.

[embray]

comment:4

Incidentally when I run my serial docs build, plot3d and plotting are built immediately after padics.

If I understand the build system right, however, each sub-document should be built in its own sphinx environment. Yet it seems as if there's some "leaking" between the environments during the initial pass, and this shouldn't happen.

[haraldschilly]

comment:5

"leaking" between the environments ...

oh, good point! that could explain why those wrong links show up at all. my wild guess: sphinx generates "temporary" files to cache some data-structures. on the next pass, sphinx doesn't clean up but somehow loads and reuses these caches. Maybe it's already enough to use find -cmin -5 or so to check recently modified files to pinpoint if/where such suspicious cache files are?

[embray]

comment:6

It's also resulting in huge objects.inv inventories for relatively small documents, because all these labels and references are being carried from one environment to the next.

It could also be the reason for some other issues I've had (for which it seems I haven't posted a ticket) with duplicate labels between some documents. The duplicate labels really shouldn't matter as long as they're not within the same document.

[embray]

comment:7

Ah, I've found it. This is a bug in Sphinx. A nasty case of "dict.copy() is a shallow copy"...

[embray]

Upstream: Reported upstream. No feedback yet.

[embray]

comment:8

Reported upstream: sphinx-doc/sphinx#2816

I have a fix in the sage end (via monkey patching) that I'm testing now.

[embray]

Changed upstream from Reported upstream. No feedback yet. to Reported upstream. Developers acknowledge bug.

[embray]

comment:10

Here's a workaround for this bug in the meantime. It's ugly, but that's to be expected. I've had this patch in my working branch for some time now and it has been well-tested.

---

New commits:

dbfb2d6

Monkey-patch for doc build issue caused by bug in Sphinx

[embray]

Commit: dbfb2d6

[embray]

Author: Erik Bray

[embray]

Branch: u/embray/patch-domain-init

[embray]

comment:11

Also a nice side-effect is that it results in smaller objects.inv files, which were previously full of unnecessary duplicates.

[embray]

Changed upstream from Reported upstream. Developers acknowledge bug. to Fixed upstream, in a later stable release.

[embray]

comment:12

The fix will be upstream in Sphinx 1.4.6, though that doesn't do us any immediate good since we're a ways from being able to upgrade Sphinx IIUC.

[paulmasson]

comment:14

After building Sage and the documentation with this branch, I randomly tested links in py-modindex.html files and didn't find any bad links. Needless to say, there is now no such file in either html/en/a_tour_of_sage as reported above, or in html/en/thematic_tutorials as currently appears in Google's webmaster tools.

I don't know enough about this part of Sage to comment on whether this is the best way to fix the problem, but it certainly does appear to deal with the issue. This should really get into 7.4 so that the next version of the documentation will look better to Google. What else needs to be tested for that to happen?

[paulmasson]

Reviewer: Paul Masson

[embray]

comment:16

In addition to fixing the specific issue in the ticket description, this fixes many other problems with building the docs in a single process that arose due to labels being shared between documents, resulting in "duplicate label" errors and similar.

[paulmasson]

comment:17

If there are no objections from other potential reviewers, setting this to positive review.

[vbraun]

Changed branch from u/embray/patch-domain-init to dbfb2d6

[jdemeyer]

comment:19

Replying to @embray:

that doesn't do us any immediate good since we're a ways from being able to upgrade Sphinx IIUC.

Why not? Did you actually try it?

[jdemeyer]

Changed commit from dbfb2d6 to none

[embray]

comment:20

I don't know why I wrote that 3 months ago. Last I recall I wasn't sure who was working on what w.r.t. Sphinx support or what the progress was on that.

[jdemeyer]

comment:21

Note: I will revert this in #22252.

[embray]

comment:22

Sounds good--thanks!