D4.13: Refactorisation of SageMath's Sphinx documentation system

[minrk]

• WP4: User Interfaces
• Lead Institution: Université Paris-Sud
• Due: 2017-08-31 (month 24)
• Nature: Other
• Task: T4.4 (T4.4: Refactor Sage 's Sphinx documentation system #72)
• Proposal: p. 47
• Final report (sources)

To do:

General

• Get in touch with the Sphinx team, and brainstorm about the plan (what could potentially be pushed upstream, ...)
  • Invitations sent for Sage Days 77
  • Informal discussions started with Georg Brandl (Sphinx lead dev) and Robert Lehmann
• Write an overview of Sphinx custom code in Sage's documentation build system: http://trac.sagemath.org/ticket/20080
• Misc documentation build improvements/cleanup:
  • http://trac.sagemath.org/ticket/19061
  • http://trac.sagemath.org/ticket/19127
  • http://trac.sagemath.org/ticket/19143
  • http://trac.sagemath.org/ticket/19153
  • http://trac.sagemath.org/ticket/19156
  • http://trac.sagemath.org/ticket/19166

Docbuilder

• Reduce disk space requirement: https://trac.sagemath.org/ticket/26115
• Make the Sage documentation build without requiring make doc-clean:
  • https://trac.sagemath.org/ticket/22588
• Build Sage documentation in $SAGE_SHARE: http://trac.sagemath.org/ticket/19963
• Profile Sphinx compilation of Sage's doc for time and memory and seek for optimizations.
  Indeed Sphinx currently takes up to 3Gb and a lot of time which feels too much.
• Cut down the large documents in the reference manual (in particular combinat) into smaller pieces for more efficient parallel build.
• Reproducible documentation build:
  • http://trac.sagemath.org/ticket/19858
  • http://trac.sagemath.org/ticket/20501
  • http://trac.sagemath.org/ticket/20574
• Allow for crosslinks between any two documents
• Make the Sage crosslink database accessible online.
  • This would enable crosslinking from outside the Sage sources, like we do in the Sage documentation for referencing Python's documentation.
• Use upstream's parallel build system instead of our own. Or, if the parallel build feature of upstream fails to meet our needs: generalize the parallel build feature, and push it upstream. Related: D3.11: HPC enabled Sage distribution #60 and D5.11: Refactor and Optimise the existing combinatorics Sage code using the new developed Pythran and Cython features. #109.
• Generalize the multiple-documents-with-crosslinks build feature, and push it upstream

autodoc

• Better support for Cython:
  • Use inspect instead of checking types sphinx-doc/sphinx#3644
  • PEP 580
• Remove all monkey patches to the Sphinx sources:
  • Use formatargspec of Sphinx: http://trac.sagemath.org/ticket/20545
  • Python 3 changes in sage_autodoc: http://trac.sagemath.org/ticket/20572
  • Various autodoc unforking: http://trac.sagemath.org/ticket/20577
  • Autodoc unforking again: https://trac.sagemath.org/ticket/22604
  • autodoc: use qualname on Python 3: https://trac.sagemath.org/ticket/27002
  • py3: make sage autodoc extension work for both python2 and python3: https://trac.sagemath.org/ticket/26949
• Replace deprecated sphinx.ext.pngmath: https://trac.sagemath.org/ticket/22560
• Fix syntax highlighting: http://trac.sagemath.org/ticket/19972
• Compile Cython code with binding=True: https://trac.sagemath.org/ticket/22747
• Testcases:
  • cysignals: http://cysignals.readthedocs.io/en/latest/
  • fpylll: http://fpylll.readthedocs.io/en/latest/

Package upgrades

• http://trac.sagemath.org/ticket/18497
• http://trac.sagemath.org/ticket/22252
• http://trac.sagemath.org/ticket/22490
• https://trac.sagemath.org/ticket/23023
• https://trac.sagemath.org/ticket/24935
• https://trac.sagemath.org/ticket/26033

[jdemeyer]

I am going to look again at this now.

[nthiery]

I am going to look again at this now.

Excellent! That indeed looks like the major next big task. For this deliverable, but in the first place because the Sage developers are looking forward to it! Feel free to ping again the Sphinx devs to encourage them to come again to Cernay.

[jdemeyer]

Feel free to ping again the Sphinx devs to encourage them to come again to Cernay.

I think it was @hivert who invited them, certainly not me.

[jdemeyer]

Some tickets needing review:

• https://trac.sagemath.org/ticket/22604
• https://trac.sagemath.org/ticket/22490
• https://trac.sagemath.org/ticket/22560
• https://trac.sagemath.org/ticket/22588

[jdemeyer]

I am currently working on improving support for Cython functions in Sphinx: sphinx-doc/sphinx#3623.

I'm using cysignals as test case: http://cysignals.readthedocs.io/en/latest/pysignals.html#cysignals.pysignals.setsignal

[jdemeyer]

As a second use-case, I will improve the documentation for fpylll too: fplll/fpylll#22

[jdemeyer]

I reorganized the issue description a bit. Personally, I worked mostly on the "autodoc" part and I think there is already more than enough stuff to do in that part.

Maybe we should sit together with the people interested in this issue and see what can be done?

[bpilorget]

@jdemeyer @hivert This deliverable is officially due for 31/08/2017. Can you please report on the progress you have achieved?
Thanks in advance

[jdemeyer]

Would be okay to delay this deliverable?

[nthiery]

Hi Jeroen!
How is the report writing going? Could you use some help, maybe from @minrk embray?
Cheers,

[embray]

There's so much work yet to do here; in fact I'd say the work has barely begun, and Jeroen hasn't received much support on it. I have a few specific fixes I would like to try out that I think will improve memory usage in Sage's documentation build, but it hardly constitutes "refactoring". I'm not sure what you want us to do with this. At best we can say "minor progress has been made towards improvments".

In particular Jeroen's done a lot of work "unforking" parts of Sage's docbuild--that is, reintegrating changes we've made to code from Sphinx back into upstream. But that's very technical. I suppose it can be worded in terms of making upstream contributions that are mutually beneficial to both projects.

Unfortunately some of the work I've done for D3.5 (#64) proposes forking a new bit of Sphinx that I needed to fork in order to get everything working right: https://trac.sagemath.org/ticket/25914 I did this a month ago but haven't gotten anyone to look at it yet. I believe the changes I needed to make should mostly be easy to get accepted upstream though, and it's a high priority for me to do just that, but only once my changes are accepted into Sage in the first place.

[nthiery]

There's so much work yet to do here;

Yeah, that's life; if we wanted to have an extension for this deliverable, we should have asked for it months ago. But we did not, so we have to submit a report next week. So let's not worry about it, and just honestly report on the current state: all the good work that was already done (the unforking was explicitly part of the refactoring!), and the specific plan on how we want to achieve that plan within the next year, with explicit mention of the manpower that we will dedicate to this, including e.g. a plan for a joint sprint with the two of you, and maybe others?

[jdemeyer]

Working on the report...

[jdemeyer]

Concerning work to do: nobody has tried to assign me to any >M36 deliverable, so I'm essentially free after august :-) I certainly do plan to continue working on this and I will write that in the report.

Also PEP 580 is partially motivated by this deliverable. Unfortunately, due to the current crisis in CPython governance, that's unlikely to be accepted (or rejected) in 2018.

[embray]

Also PEP 580 is partially motivated by this deliverable. Unfortunately, due to the current crisis in CPython governance, that's unlikely to be accepted (or rejected) in 2018.

Right, I meant to mention that. That would probably be worth adding to the report.

[nthiery]

I just finished reviewing the draft of report. That's a good start, thanks again!

I made some edits, and left TODO's; for some of them, you may want to ping me back if unsure what to do.

[jdemeyer]

I'll first finish a draft for D4.10 and then come back here.

[jdemeyer]

Quick question: do we have an "official" way to refer to ODK workshops? Should I just write "the April 2016 OpenDreamKit workshop in Cernay"?

[jdemeyer]

You wrote

\TODO{Sphinx configuration for Sage packages: template in sage-sample
  and utilities in sage-package}

but I disagree. The whole point is to use a standard Sphinx setup, so we don't need "Sphinx configuration for Sage packages".

[jdemeyer]

\TODO{Any relevant visual?} -> I don't think so

[nthiery]

Quick question: do we have an "official" way to refer to ODK workshops? Should I just write "the April 2016 OpenDreamKit workshop in Cernay"?

This one was Sage Days 77. Let's say "the Sage Days 77 workshop organized by OpenDreamKit" or some similar phrasing.

[nthiery]

On Tue, Aug 28, 2018 at 03:20:25AM -0700, jdemeyer wrote: You wrote \TODO{Sphinx configuration for Sage packages: template in sage-sample and utilities in sage-package} but I disagree. The whole point is to use a standard Sphinx setup, so we don't need "Sphinx configuration for Sage packages".

Sorry if I have been unclear; the whole point is indeed to use a vanilla Sphinx installation. Nevertheless Sage will still use some custom configuration, if not just for theming, right? this we want to share with Sage packages one way or the other.

[jdemeyer]

Just pushed another update. I also added a reference to T3.3 (packaging) since this is actually quite relevant for packaging too.

[jdemeyer]

Nevertheless Sage will still use some custom configuration

I'm not sure about this actually.

[nthiery]

I'm not sure about this actually.

The less custom we have, the better :-) Nevertheless, things like our custom trac roles for Wikipedia, trac, and such, the default role `` -> maths, the thebelab configuration, that we want to use plugins like intersphinx to crosslink to Python's standard documentation. That's all configuration that will differ from Sphinx's default value, right?

[jdemeyer]

OK, I see your point now. But what (if anything) should we write about it?

[jdemeyer]

Regarding

\TODO{Compiling the documentation to Jupyter notebooks:
  rst2ipynb, contributions to https://github.com/QuantEcon/sphinxcontrib-jupyter/
  https://trac.sagemath.org/ticket/25909;
  NT can write that if needed; I believe it's not reported elsewhere;
  question: I don't know whether it fits the scope of the deliverable}

1. Can we claim it as part of ODK?

2. What it does it quite related to ThebeLab. So, if we report on it, it makes sense to do that on D4.7 (D4.7: Full featured Jupyter interface for GAP, PARI/GP, Singular #96), where we already report on ThebeLab. (EDIT: on the other hand, we could report it here to give a better impression that we actually did something for D4.13)

[nthiery]

OK, I see your point now. But what (if anything) should we write about it?

What do you think of a brief phrase stating something like: it is beneficial for Sage packages to use the same theming and configuration as Sage for building their documentation. We simplified the process by implementing the sage-package utility (which contains among other things a copy of the Sage-Sphinx configuration), and illustrating its usage in the sage-sample package template. We are investigating for the best way to avoid the aforementioned copy.

[nthiery]

1. Can we claim it as part of ODK?

Most of it (but of course not the development of sphinxcontrib-jupyter itself; just the contributions) was done by Olivier Cayrol and myself. So yes.

2. What it does it quite related to ThebeLab. So, if we report on it, it makes sense to do that on D4.7 ([1]#96), where we already report on ThebeLab.

I agree it's borderline. Here, most of the work is Sphinx related. Whereas for ThebeLab most of the work was related to Jupyter, with just a tiny bit of Sphinx configuration. Hence it seems to fit a bit better here.

[jdemeyer]

@nthiery I pushed another update adding some of the stuff we discussed today. Please review in particular the last section.

[jdemeyer]

I made some final fixes to the report. For me, it is done.

[embray]

I'll give it another read over. Based on my last look at it I don't think I'll have much to add on the content; I'm mostly just proof-reading.

[embray]

Unfortunately, the process of
getting a PEP accepted is quite slow: at the time of writing this report, the PEP had not been
accepted (nor rejected).

I wonder if it's worth mentioning in a footnote that Guido was mostly positive about the PEP, which might have been helpful in getting it done faster if not for the unfortunate (for your PEP) timing of his retirement from the BDFL role.

[embray]

In other words, the current wording leaves some doubt as to whether or not it will be accepted. I think, from what I've seen so far, it likely will be accepted, eventually, in some form or other. Even if some of the details change I don't think anyone can disagree (well, I'm sure 1 or 2 people do) that refactoring CPython's function types to have a more consistent interface is a good thing.

[jdemeyer]

In other words, the current wording leaves some doubt as to whether or not it will be accepted. I think, from what I've seen so far, it likely will be accepted, eventually, in some form or other.

Personally, I'm more pessimistic (or realistic?). I think that there is also a real possibility of it just being ignored, neither accepted or rejected (this is certainly the case for many Python PRs).

[nthiery]

Thanks Jeroen and Erik for the polishing!
I just finished reviewing and pushed some changes; most minor, and a bit of refactoring here and there. I also included the pep as appendix. Let me know if you want to change something (e.g. about the PEP) or whether I should just submit as is.

[embray]

OK, I remain optimistic about it in the long term, but you're in the thick of it so I know how frustrating it's been and can easily see how you can be pessimistic (there is precedent for PEPs that looked good in principle but just stalled out indefinitely). So if you're not comfortable making any claims with confidence then let's just let it be.

[nthiery]

Submitted!
Thanks Jeroen for the long term work on this project and the report; thanks Erik for the proofreading!

[IzabelaFaguet]

Accepted by the EU on December 5th.