New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

D4.13: Refactorisation of SageMath's Sphinx documentation system #87

Open
minrk opened this Issue Sep 8, 2015 · 42 comments

Comments

Projects
None yet
6 participants
@minrk
Contributor

minrk commented Sep 8, 2015

  • WP4: User Interfaces
  • Lead Institution: Université Paris-Sud
  • Due: 2017-08-31 (month 24)
  • Nature: Other
  • Task: T4.4

See page 48 of the proposal.

To do:

General

Docbuilder

  • Reduce disk space requirement: https://trac.sagemath.org/ticket/26115
  • Make the Sage documentation build without requiring make doc-clean:
  • Build Sage documentation in $SAGE_SHARE: http://trac.sagemath.org/ticket/19963
  • Profile Sphinx compilation of Sage's doc for time and memory and seek for optimizations.
    Indeed Sphinx currently takes up to 3Gb and a lot of time which feels too much.
  • Cut down the large documents in the reference manual (in particular combinat) into smaller pieces for more efficient parallel build.
  • Reproducible documentation build:
  • Allow for crosslinks between any two documents
  • Make the Sage crosslink database accessible online.
    • This would enable crosslinking from outside the Sage sources, like we do in the Sage documentation for referencing Python's documentation.
  • Use upstream's parallel build system instead of our own. Or, if the parallel build feature of upstream fails to meet our needs: generalize the parallel build feature, and push it upstream. Related: #60 and #109.
  • Generalize the multiple-documents-with-crosslinks build feature, and push it upstream

autodoc

Package upgrades

@minrk minrk added this to the D4.13 milestone Sep 8, 2015

@minrk minrk self-assigned this Nov 3, 2015

@nthiery nthiery assigned hivert and unassigned minrk Mar 22, 2016

@nthiery nthiery modified the milestones: Month 24: 2017-08-31, D4.13 Mar 22, 2016

@jdemeyer jdemeyer changed the title from D4.13: Refactorisation of Sage 's Sphinx documentation system to D4.13: Refactorisation of SageMath's Sphinx documentation system Jun 16, 2016

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Mar 2, 2017

Contributor

I am going to look again at this now.

Contributor

jdemeyer commented Mar 2, 2017

I am going to look again at this now.

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Mar 2, 2017

Contributor
Contributor

nthiery commented Mar 2, 2017

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Mar 3, 2017

Contributor

Feel free to ping again the Sphinx devs to encourage them to come again to Cernay.

I think it was @hivert who invited them, certainly not me.

Contributor

jdemeyer commented Mar 3, 2017

Feel free to ping again the Sphinx devs to encourage them to come again to Cernay.

I think it was @hivert who invited them, certainly not me.

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Apr 13, 2017

Contributor

I am currently working on improving support for Cython functions in Sphinx: sphinx-doc/sphinx#3623.

I'm using cysignals as test case: http://cysignals.readthedocs.io/en/latest/pysignals.html#cysignals.pysignals.setsignal

Contributor

jdemeyer commented Apr 13, 2017

I am currently working on improving support for Cython functions in Sphinx: sphinx-doc/sphinx#3623.

I'm using cysignals as test case: http://cysignals.readthedocs.io/en/latest/pysignals.html#cysignals.pysignals.setsignal

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Apr 14, 2017

Contributor

As a second use-case, I will improve the documentation for fpylll too: fplll/fpylll#22

Contributor

jdemeyer commented Apr 14, 2017

As a second use-case, I will improve the documentation for fpylll too: fplll/fpylll#22

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Apr 15, 2017

Contributor

I reorganized the issue description a bit. Personally, I worked mostly on the "autodoc" part and I think there is already more than enough stuff to do in that part.

Maybe we should sit together with the people interested in this issue and see what can be done?

Contributor

jdemeyer commented Apr 15, 2017

I reorganized the issue description a bit. Personally, I worked mostly on the "autodoc" part and I think there is already more than enough stuff to do in that part.

Maybe we should sit together with the people interested in this issue and see what can be done?

@jdemeyer jdemeyer self-assigned this Apr 24, 2017

@bpilorget

This comment has been minimized.

Show comment
Hide comment
@bpilorget

bpilorget Aug 16, 2017

Contributor

@jdemeyer @hivert This deliverable is officially due for 31/08/2017. Can you please report on the progress you have achieved?
Thanks in advance

Contributor

bpilorget commented Aug 16, 2017

@jdemeyer @hivert This deliverable is officially due for 31/08/2017. Can you please report on the progress you have achieved?
Thanks in advance

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 29, 2017

Contributor

Would be okay to delay this deliverable?

Contributor

jdemeyer commented Aug 29, 2017

Would be okay to delay this deliverable?

@bpilorget bpilorget added UGe and removed PS labels Jan 12, 2018

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 18, 2018

Contributor

Hi Jeroen!
How is the report writing going? Could you use some help, maybe from @minrk embray?
Cheers,

Contributor

nthiery commented Aug 18, 2018

Hi Jeroen!
How is the report writing going? Could you use some help, maybe from @minrk embray?
Cheers,

@embray

This comment has been minimized.

Show comment
Hide comment
@embray

embray Aug 20, 2018

Collaborator

There's so much work yet to do here; in fact I'd say the work has barely begun, and Jeroen hasn't received much support on it. I have a few specific fixes I would like to try out that I think will improve memory usage in Sage's documentation build, but it hardly constitutes "refactoring". I'm not sure what you want us to do with this. At best we can say "minor progress has been made towards improvments".

In particular Jeroen's done a lot of work "unforking" parts of Sage's docbuild--that is, reintegrating changes we've made to code from Sphinx back into upstream. But that's very technical. I suppose it can be worded in terms of making upstream contributions that are mutually beneficial to both projects.

Unfortunately some of the work I've done for D3.5 (#64) proposes forking a new bit of Sphinx that I needed to fork in order to get everything working right: https://trac.sagemath.org/ticket/25914 I did this a month ago but haven't gotten anyone to look at it yet. I believe the changes I needed to make should mostly be easy to get accepted upstream though, and it's a high priority for me to do just that, but only once my changes are accepted into Sage in the first place.

Collaborator

embray commented Aug 20, 2018

There's so much work yet to do here; in fact I'd say the work has barely begun, and Jeroen hasn't received much support on it. I have a few specific fixes I would like to try out that I think will improve memory usage in Sage's documentation build, but it hardly constitutes "refactoring". I'm not sure what you want us to do with this. At best we can say "minor progress has been made towards improvments".

In particular Jeroen's done a lot of work "unforking" parts of Sage's docbuild--that is, reintegrating changes we've made to code from Sphinx back into upstream. But that's very technical. I suppose it can be worded in terms of making upstream contributions that are mutually beneficial to both projects.

Unfortunately some of the work I've done for D3.5 (#64) proposes forking a new bit of Sphinx that I needed to fork in order to get everything working right: https://trac.sagemath.org/ticket/25914 I did this a month ago but haven't gotten anyone to look at it yet. I believe the changes I needed to make should mostly be easy to get accepted upstream though, and it's a high priority for me to do just that, but only once my changes are accepted into Sage in the first place.

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 20, 2018

Contributor
Contributor

nthiery commented Aug 20, 2018

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 21, 2018

Contributor

Working on the report...

Contributor

jdemeyer commented Aug 21, 2018

Working on the report...

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 21, 2018

Contributor

Concerning work to do: nobody has tried to assign me to any >M36 deliverable, so I'm essentially free after august :-) I certainly do plan to continue working on this and I will write that in the report.

Also PEP 580 is partially motivated by this deliverable. Unfortunately, due to the current crisis in CPython governance, that's unlikely to be accepted (or rejected) in 2018.

Contributor

jdemeyer commented Aug 21, 2018

Concerning work to do: nobody has tried to assign me to any >M36 deliverable, so I'm essentially free after august :-) I certainly do plan to continue working on this and I will write that in the report.

Also PEP 580 is partially motivated by this deliverable. Unfortunately, due to the current crisis in CPython governance, that's unlikely to be accepted (or rejected) in 2018.

@embray

This comment has been minimized.

Show comment
Hide comment
@embray

embray Aug 21, 2018

Collaborator

Also PEP 580 is partially motivated by this deliverable. Unfortunately, due to the current crisis in CPython governance, that's unlikely to be accepted (or rejected) in 2018.

Right, I meant to mention that. That would probably be worth adding to the report.

Collaborator

embray commented Aug 21, 2018

Also PEP 580 is partially motivated by this deliverable. Unfortunately, due to the current crisis in CPython governance, that's unlikely to be accepted (or rejected) in 2018.

Right, I meant to mention that. That would probably be worth adding to the report.

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 23, 2018

Contributor

Working on the report
Thanks @jdemeyer; I am looking forward to it! Feel free to commit early versions for early feedback.

Contributor

nthiery commented Aug 23, 2018

Working on the report
Thanks @jdemeyer; I am looking forward to it! Feel free to commit early versions for early feedback.

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 23, 2018

Contributor

Concerning work to do: nobody has tried to assign me to any >M36 deliverable, so I'm essentially free after august :-) I certainly do plan to continue working on this and I will write that in the report.

We indeed should discuss what the priorities are for ODK and how to make the best out of this last year.
From the top of my head: Sphinx, Python 3, packaging, modularization, and dev workflow would all benefit much from your expertise.

Contributor

nthiery commented Aug 23, 2018

Concerning work to do: nobody has tried to assign me to any >M36 deliverable, so I'm essentially free after august :-) I certainly do plan to continue working on this and I will write that in the report.

We indeed should discuss what the priorities are for ODK and how to make the best out of this last year.
From the top of my head: Sphinx, Python 3, packaging, modularization, and dev workflow would all benefit much from your expertise.

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 23, 2018

Contributor

I just pushed a first draft of this report. I admit that it's quite technical, but then again: this is technical material and there is not much cool stuff to show.

It's also more of a work plan rather than a report on finished work, but that's the honest reality.

Contributor

jdemeyer commented Aug 23, 2018

I just pushed a first draft of this report. I admit that it's quite technical, but then again: this is technical material and there is not much cool stuff to show.

It's also more of a work plan rather than a report on finished work, but that's the honest reality.

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 23, 2018

Contributor

Thanks Jeroen! That's a technical deliverable; given sufficient context, a technical report is fine. And being honest is our brand mark. I'll have a look soon.

Contributor

nthiery commented Aug 23, 2018

Thanks Jeroen! That's a technical deliverable; given sufficient context, a technical report is fine. And being honest is our brand mark. I'll have a look soon.

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 24, 2018

Contributor

I just finished reviewing the draft of report. That's a good start, thanks again!

I made some edits, and left TODO's; for some of them, you may want to ping me back if unsure what to do.

Contributor

nthiery commented Aug 24, 2018

I just finished reviewing the draft of report. That's a good start, thanks again!

I made some edits, and left TODO's; for some of them, you may want to ping me back if unsure what to do.

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 27, 2018

Contributor

I'll first finish a draft for D4.10 and then come back here.

Contributor

jdemeyer commented Aug 27, 2018

I'll first finish a draft for D4.10 and then come back here.

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 28, 2018

Contributor

Quick question: do we have an "official" way to refer to ODK workshops? Should I just write "the April 2016 OpenDreamKit workshop in Cernay"?

Contributor

jdemeyer commented Aug 28, 2018

Quick question: do we have an "official" way to refer to ODK workshops? Should I just write "the April 2016 OpenDreamKit workshop in Cernay"?

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 28, 2018

Contributor

You wrote

\TODO{Sphinx configuration for Sage packages: template in sage-sample
  and utilities in sage-package}

but I disagree. The whole point is to use a standard Sphinx setup, so we don't need "Sphinx configuration for Sage packages".

Contributor

jdemeyer commented Aug 28, 2018

You wrote

\TODO{Sphinx configuration for Sage packages: template in sage-sample
  and utilities in sage-package}

but I disagree. The whole point is to use a standard Sphinx setup, so we don't need "Sphinx configuration for Sage packages".

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 28, 2018

Contributor

\TODO{Any relevant visual?} -> I don't think so

Contributor

jdemeyer commented Aug 28, 2018

\TODO{Any relevant visual?} -> I don't think so

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 28, 2018

Contributor
Contributor

nthiery commented Aug 28, 2018

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 28, 2018

Contributor
Contributor

nthiery commented Aug 28, 2018

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 28, 2018

Contributor

Just pushed another update. I also added a reference to T3.3 (packaging) since this is actually quite relevant for packaging too.

Contributor

jdemeyer commented Aug 28, 2018

Just pushed another update. I also added a reference to T3.3 (packaging) since this is actually quite relevant for packaging too.

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 28, 2018

Contributor

Nevertheless Sage will still use some custom configuration

I'm not sure about this actually.

Contributor

jdemeyer commented Aug 28, 2018

Nevertheless Sage will still use some custom configuration

I'm not sure about this actually.

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 28, 2018

Contributor
Contributor

nthiery commented Aug 28, 2018

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 28, 2018

Contributor

OK, I see your point now. But what (if anything) should we write about it?

Contributor

jdemeyer commented Aug 28, 2018

OK, I see your point now. But what (if anything) should we write about it?

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 28, 2018

Contributor

Regarding

\TODO{Compiling the documentation to Jupyter notebooks:
  rst2ipynb, contributions to https://github.com/QuantEcon/sphinxcontrib-jupyter/
  https://trac.sagemath.org/ticket/25909;
  NT can write that if needed; I believe it's not reported elsewhere;
  question: I don't know whether it fits the scope of the deliverable}
  1. Can we claim it as part of ODK?

  2. What it does it quite related to ThebeLab. So, if we report on it, it makes sense to do that on D4.7 (#96), where we already report on ThebeLab. (EDIT: on the other hand, we could report it here to give a better impression that we actually did something for D4.13)

Contributor

jdemeyer commented Aug 28, 2018

Regarding

\TODO{Compiling the documentation to Jupyter notebooks:
  rst2ipynb, contributions to https://github.com/QuantEcon/sphinxcontrib-jupyter/
  https://trac.sagemath.org/ticket/25909;
  NT can write that if needed; I believe it's not reported elsewhere;
  question: I don't know whether it fits the scope of the deliverable}
  1. Can we claim it as part of ODK?

  2. What it does it quite related to ThebeLab. So, if we report on it, it makes sense to do that on D4.7 (#96), where we already report on ThebeLab. (EDIT: on the other hand, we could report it here to give a better impression that we actually did something for D4.13)

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 28, 2018

Contributor
Contributor

nthiery commented Aug 28, 2018

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 28, 2018

Contributor
Contributor

nthiery commented Aug 28, 2018

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 28, 2018

Contributor

@nthiery I pushed another update adding some of the stuff we discussed today. Please review in particular the last section.

Contributor

jdemeyer commented Aug 28, 2018

@nthiery I pushed another update adding some of the stuff we discussed today. Please review in particular the last section.

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 31, 2018

Contributor

I made some final fixes to the report. For me, it is done.

Contributor

jdemeyer commented Aug 31, 2018

I made some final fixes to the report. For me, it is done.

@embray

This comment has been minimized.

Show comment
Hide comment
@embray

embray Aug 31, 2018

Collaborator

I'll give it another read over. Based on my last look at it I don't think I'll have much to add on the content; I'm mostly just proof-reading.

Collaborator

embray commented Aug 31, 2018

I'll give it another read over. Based on my last look at it I don't think I'll have much to add on the content; I'm mostly just proof-reading.

@embray

This comment has been minimized.

Show comment
Hide comment
@embray

embray Aug 31, 2018

Collaborator

Unfortunately, the process of
getting a PEP accepted is quite slow: at the time of writing this report, the PEP had not been
accepted (nor rejected).

I wonder if it's worth mentioning in a footnote that Guido was mostly positive about the PEP, which might have been helpful in getting it done faster if not for the unfortunate (for your PEP) timing of his retirement from the BDFL role.

Collaborator

embray commented Aug 31, 2018

Unfortunately, the process of
getting a PEP accepted is quite slow: at the time of writing this report, the PEP had not been
accepted (nor rejected).

I wonder if it's worth mentioning in a footnote that Guido was mostly positive about the PEP, which might have been helpful in getting it done faster if not for the unfortunate (for your PEP) timing of his retirement from the BDFL role.

@embray

This comment has been minimized.

Show comment
Hide comment
@embray

embray Aug 31, 2018

Collaborator

In other words, the current wording leaves some doubt as to whether or not it will be accepted. I think, from what I've seen so far, it likely will be accepted, eventually, in some form or other. Even if some of the details change I don't think anyone can disagree (well, I'm sure 1 or 2 people do) that refactoring CPython's function types to have a more consistent interface is a good thing.

Collaborator

embray commented Aug 31, 2018

In other words, the current wording leaves some doubt as to whether or not it will be accepted. I think, from what I've seen so far, it likely will be accepted, eventually, in some form or other. Even if some of the details change I don't think anyone can disagree (well, I'm sure 1 or 2 people do) that refactoring CPython's function types to have a more consistent interface is a good thing.

@jdemeyer

This comment has been minimized.

Show comment
Hide comment
@jdemeyer

jdemeyer Aug 31, 2018

Contributor

In other words, the current wording leaves some doubt as to whether or not it will be accepted. I think, from what I've seen so far, it likely will be accepted, eventually, in some form or other.

Personally, I'm more pessimistic (or realistic?). I think that there is also a real possibility of it just being ignored, neither accepted or rejected (this is certainly the case for many Python PRs).

Contributor

jdemeyer commented Aug 31, 2018

In other words, the current wording leaves some doubt as to whether or not it will be accepted. I think, from what I've seen so far, it likely will be accepted, eventually, in some form or other.

Personally, I'm more pessimistic (or realistic?). I think that there is also a real possibility of it just being ignored, neither accepted or rejected (this is certainly the case for many Python PRs).

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 31, 2018

Contributor

Thanks Jeroen and Erik for the polishing!
I just finished reviewing and pushed some changes; most minor, and a bit of refactoring here and there. I also included the pep as appendix. Let me know if you want to change something (e.g. about the PEP) or whether I should just submit as is.

Contributor

nthiery commented Aug 31, 2018

Thanks Jeroen and Erik for the polishing!
I just finished reviewing and pushed some changes; most minor, and a bit of refactoring here and there. I also included the pep as appendix. Let me know if you want to change something (e.g. about the PEP) or whether I should just submit as is.

@embray

This comment has been minimized.

Show comment
Hide comment
@embray

embray Aug 31, 2018

Collaborator

OK, I remain optimistic about it in the long term, but you're in the thick of it so I know how frustrating it's been and can easily see how you can be pessimistic (there is precedent for PEPs that looked good in principle but just stalled out indefinitely). So if you're not comfortable making any claims with confidence then let's just let it be.

Collaborator

embray commented Aug 31, 2018

OK, I remain optimistic about it in the long term, but you're in the thick of it so I know how frustrating it's been and can easily see how you can be pessimistic (there is precedent for PEPs that looked good in principle but just stalled out indefinitely). So if you're not comfortable making any claims with confidence then let's just let it be.

@nthiery nthiery added the Submitted label Aug 31, 2018

@nthiery

This comment has been minimized.

Show comment
Hide comment
@nthiery

nthiery Aug 31, 2018

Contributor

Submitted!
Thanks Jeroen for the long term work on this project and the report; thanks Erik for the proofreading!

Contributor

nthiery commented Aug 31, 2018

Submitted!
Thanks Jeroen for the long term work on this project and the report; thanks Erik for the proofreading!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment