Updating doc building to Sphinx 4.3.2 (python 3) #2662

barsch · 2020-06-29T16:33:30Z

This PR updates the docs build process to Sphinx 3.1.1. After unsucessfully trying to fix the old doc builder I decided to restart the whole process from scratch and reimplement existing features afterwards.

Bootstrap theme got replaced by more lightweight "Read the Docs" theme - however I didn't spend to much time on styling yet as this is something which should be tackled after the build process is running smooth
stylesheet/javascript/image optimization is done via gulp
played a bit with sphinx-gallery but didn't like it, so its again a modified version of matplotlib.sphinxext.plot_directive, which work good enough for me
all plots using basemap or mlpy are failing -> segmentation fault on ubuntu as well on win10 for basemap 1.2.1 on py38
docs were in pretty bad shape - a lot of warnings have been fixed - there are still many left
currently only the html builder has been tested - latex/docset etc need work
the structure of the start page probably needs some tweak - I suggest to remove all help/mailing list options from there and link to a github page instead - we should focus on documentation only IMHO

A preview of the current results can be seen here: https://docs.obspy.org/sphinx3/

Feel free to participate - running the doc building takes unfortunatly ages ... If someone could rewrite all examples using basemap to cartopy would be a big help.

megies · 2020-06-30T09:07:14Z

Awesome stuff @barsch. I like the overall looks, and I'm real happy that we'll have a proper navigation again (with the left side bar).

Can you put the exact version numbers of the build environment somewhere, I can help set up the cron jobs later.. or.. we could have @d-chambers and @heavelock give a shot at building docs in gh-actions? Might be the right time to give it a try, since we would have to set up docs building on new server from scratch more or less anyway?

Also, can you put the build log as a gist maybe.. interesting to see if it got any more readable at all compared to the 80-page warning/error logs we used to have

barsch · 2020-06-30T09:10:18Z

@megies its already build on the new server (manually atm) - log can be found here https://docs.obspy.org/sphinx3/log.txt

megies · 2020-06-30T09:28:25Z

OK, oof, still such a bookshelf of warnings.

barsch · 2020-06-30T09:31:27Z

jupp - and still takes ages to run - its not fun working on it ...

megies · 2020-06-30T10:21:13Z

No kidding, it might easily be the most painful thing to work on in the whole project.

adigitoleo · 2020-11-08T04:30:21Z

Hey I was browsing the docs and saw that there are some bits where sphinx links to the builtin time module, just because time is used as a parameter name:

It's a very small thing and it seems like you've got enough work cut out already, but I thought I'd point it out since you're updating the docs. It might be solved using the :samp: sphinx role, but I've never used it:

https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-samp

heavelock · 2021-02-10T19:44:57Z

Hey! I had some spare time today and I thought about finally working on building docs on GithubActions. Coming from GitlabCI this action systems seems to be weird but I think I managed to make the builder run.

Check it out on PR against my fork:
heavelock#9

It's based on obspy/sphinx3 branch that was rebased on top of current master

From some of the important changes in the code of docs itself I changed m2r dependency to m2r2 that is a fork of the former one which is not maintained anymore.

heavelock · 2021-02-15T10:52:01Z

Oh dammit. I accidentally pushed to wrong remote and my changes got in here. Sorry for that as I meant to first ask before doing that ...

Anyway, should I remove my changes or can they stay here?

My changes include:

Rebasing on top of current master
Add GH action for building in CI
GH action for exporting the artifact with built docs (Working!)
Added cartopy to dependencies of docs so we can start fixing plotting routines.
Pinned sphinx==3.4.3 as 3.5.0 that was released yesterday has a bug that prevents from building.

barsch · 2021-02-15T10:53:11Z

All fine - knock yourself out ;)

heavelock · 2021-02-15T13:02:47Z

Since we are at the docs, do you think it would beneficial for us to start using some linter for docs?
In my projects I'm using doc8 which is equivalent to flake8 for rst docs.

What do you think?

heavelock · 2021-02-15T13:07:17Z

Basemap -> Cartopy conversion progress:

basemap_with_beachball_read_events.py
basemap_plot_with_beachballs2.py
basemap_plot_with_beachballs.py
basemap_plot_custom.py ----- This plot will be crippled until Inventory.plot and Catalog.plot will not be converted to Cartopy

barsch · 2021-02-15T13:34:16Z

@heavelock first of all big thanks for looking into this! Two rants from my side while you are on it:

can you also look into this: Updating doc building to Sphinx 4.3.2 (python 3) #2662 (comment)
important for me would be to have always the possibility to run the docs build process locally in case gh-actions fails to work/they charge money in the future or for whatever reason

heavelock · 2021-02-15T15:38:03Z

Yeah, sure. Not today though, I have to come back to my normal tasks for now. I will take a look on that tomorrow.

And the second point:
Definitely. How about having a dedicated docker image for building it?

barsch · 2021-02-15T15:39:52Z

docker would be perfect imho

heavelock · 2021-02-15T15:41:14Z

perf. I had it already done but then I ditched it because the sphinx-action I am using here does not support using a custom image as argument. I could fork it and have our own action for that or just create a custom workflow. Positive side of the action is that it is parsing the warnings from sphinx and dropping them in diff.

heavelock · 2021-02-17T07:48:32Z

I testes the :samp: directive but it breaks the rendering. I noticed that this problem occurs only for class variables, it is not cross-referenced for standard :param:. I guesss it might be a bug upstream. I will post it to the sphinx's bugtracker.

Edit: Bugfix for that will be included in 4.0.0:
sphinx-doc/sphinx#8902

heavelock · 2021-02-17T08:14:39Z

@barsch Would you mind forking this repo as organization? https://github.com/ammaraskar/sphinx-action
It is the action that I used for building docs. Unfortunately, I don't see a possibility of hanging a docker image used by it without modifying the code.

barsch · 2021-02-17T08:34:27Z

@barsch Would you mind forking this repo as organization? https://github.com/ammaraskar/sphinx-action

done: https://github.com/obspy/sphinx-action

ThomasLecocq · 2022-01-10T17:16:58Z

my turn to give it a go :-)

ThomasLecocq · 2022-01-11T09:41:09Z

2022 changes

Rebased on current master
Changed the template for class so that the Private and Special methods don't create a new page, but are listed/inline on the Class's page. This results in a gigantic reduction of output html (there was one for each of __eq__, __del__ etc, for each class). It makes longer pages, but all the info (and cross-refs) are preserved: see example [after/before]:
Disabled rendering hires & pdf for plots, it takes a lot of time for no real usefulness
removed the ":numbered:" param to the tutorial
The build on github actions + @heavelock 's docker takes ~20 min, and the artifact works smooth

Example output: http://msnoise.org/obspy-doc/

on my local machine, the first build took also ~20 minutes, successive builds (adding one comma to one RST file): less than 2 minutes. This means, running a doc builder that doesn't redo everything from scratch could produce docs in a flash. That'd require having some workflow which would include some "persistence":

first run ever (ever)

create conda env (+cache it)
pull master branch
build the doc
upload the whole package content as artifact

successive builds:

activate doc environment
download the artifact
pull/checkout the branch (!pulling changes the "version" and makes a full new build, need to "patch" !)
build doc
export .build/html folder somewhere

TODO still:

revert or adapt the changelog.rst vs changelog.md
try to shrink the 3554 (!) warnings
fix gallery plots
find a way to reduce slow processes:

====================== slowest reading durations =======================
452.730 packages/autogen/obspy.imaging.cm
42.885 packages/autogen/obspy.core.inventory.network.Network.plot
34.954 tutorial/code_snippets/beamforming_fk_analysis
29.142 packages/autogen/obspy.core.event.Catalog.plot
9.388 tutorial/code_snippets/waveform_plotting_tutorial

trichter · 2022-01-14T16:09:30Z

I reverted my changes here and created a new branch at PR #2947.

ThomasLecocq · 2022-01-17T11:52:28Z

work in progress to make the event & inventory cartopy-proof (might put that in a separate PR, not sure yet):

trichter · 2022-01-17T12:44:32Z

might put that in a separate PR, not sure yet

What fits best for you. I skimmed through changes introduced by this PR and I didn't find any obvious glitches. Maybe we can try to get rid of the image replacements of _images/*.png to not grow the size of the repository unnecessarily. Some stuff I don't understand (e.g. gulpfile.js), so maybe @barsch can also have look after you finished.

barsch · 2022-01-17T15:11:34Z

gulp (https://gulpjs.com/docs/en/getting-started/quick-start) is just a build tool to automate CSS, JavaScript and static image management, e.g. minifying code, optimizing images, preprocessing CSS, etc.

I wouldn't try to automate this via GH actions if not needed as this has failed in the past after updating dependencies etc. Instead I suggest we just leave the the gulpfile.js in the root and once some theme changes are needed one just runs the script locally and creates new static minimized and compressed .css and .js files

trichter · 2022-01-17T16:33:58Z

I checked ObsPy's sectret tab and:

Secrets are not passed to workflows that are triggered by a pull request from a fork.

Because we want to see doc builds also from forks, it will not work to deploy the doc via SSH in a run triggered in the doc building workflow. We either need a separate github action checking in a time interval for artifacts to upload. Or straightforward, the server can check for runs and relevant artifacts on its own. So the doc building would still be on github actions and can be maintained in this repository, but the server fetches and deploys the docs.

barsch · 2022-01-17T16:36:52Z

Or straightforward, the server can check for runs and relevant artifacts on its own. So the doc building would still be on github actions and can be maintained in this repository, but the server fetches and deploys the docs.

I very much prefer that.

trichter · 2022-01-17T16:45:28Z

Great. @barsch Do you want to write the necessary server-side code? I could also do it. This PR can serve as test load.

ThomasLecocq · 2022-01-17T16:47:20Z

Great. @barsch Do you want to write the necessary server-side code? I could also do it. This PR can serve as test load.

@trichter I propose we do it you & me when the PR is ready. I'm almost done with the final plots.

barsch · 2022-01-17T16:55:35Z

there should be code which could be recycled here: https://github.com/obspy/obspy/tree/master/misc/scripts/docs_buildbot

Great. @barsch Do you want to write the necessary server-side code? I could also do it. This PR can serve as test load.

I'm happy to maintain/admin the server and obspy-reporter, but as stated before coding for obspy is not really on my agenda any more - didn't use obspy since 10 years or so and its about time other take over ;)

(but I'm actually very impressed and quite happy to see huge progress here)

ThomasLecocq · 2022-01-17T17:03:30Z

all look OK now (didn't yet port all the "resolution" stuff, but as a PoC it works)

ThomasLecocq · 2022-01-17T17:26:44Z

YESSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS http://msnoise.org/obspy-doc/packages/autogen/obspy.core.event.Event.plot.html#obspy.core.event.Event.plot

barsch · 2022-01-17T17:47:16Z

hmm

https://docs.obspy.org/sphinx3/gallery.html

vs.

http://msnoise.org/obspy-doc/gallery.html

for whatever reason custom.min.css is not used

ThomasLecocq · 2022-01-17T17:55:25Z

hmm

https://docs.obspy.org/sphinx3/gallery.html

vs.

http://msnoise.org/obspy-doc/gallery.html

for whatever reason custom.min.css is not used

ah, indeed, that... checking now- difference is that on your site div has class "gallery" and not on mine

ThomasLecocq · 2022-01-17T17:58:36Z

CI was 🟢 , which is very satisfactory :-)
now rebuilding the doc locally to update http://msnoise.org/obspy-doc/gallery.html & then I think it'll be good to merge.

And move on with, on the DOC side: the doc builder and on the code side: remove all basemap stuff

🎉 tadaaaaaaaaaaaaaaaaaaa

ThomasLecocq · 2022-01-17T18:09:21Z

249 commits +2,927 −3,259 ... wow

I will squash to "1 commit" to merge in master :-)

megies added the docs issues that only affect the content of documentation pages, docstrings etc label Jun 30, 2020

megies mentioned this pull request Jun 30, 2020

wip: gh-actions #2591

Merged

barsch mentioned this pull request Feb 9, 2021

Add support for eventtype in Client.get_events() #2780

Merged

9 tasks

heavelock force-pushed the sphinx3 branch from 09f442f to 0e8e8e5 Compare February 15, 2021 08:48

heavelock mentioned this pull request Feb 17, 2021

Class variable name cross referencing sphinx-doc/sphinx#8902

Closed

heavelock marked this pull request as ready for review November 6, 2021 20:31

heavelock mentioned this pull request Nov 6, 2021

Docsbuilding docker image & GH Action #2792

Closed

ThomasLecocq mentioned this pull request Jan 5, 2022

Cannot build the documentation in Python 3 #2893

Closed

ThomasLecocq force-pushed the sphinx3 branch from 81abf15 to f4f7504 Compare January 11, 2022 08:59

ThomasLecocq changed the title ~~[WIP] Updating doc building to Sphinx 3.1.1~~ [WIP] Updating doc building to Sphinx 4.3.2 (python 3) Jan 11, 2022

replace the final basemap stuff with cartopy

b7a6f50

ThomasLecocq added 4 commits January 17, 2022 18:05

PEP errors

6d2c888

get rid of basemap

a9214b7

get rid of basemap

9b27d74

add cartopy check to sources.py

b0e8b28

ThomasLecocq added 2 commits January 17, 2022 18:28

linter ....

7dfe9b3

should probably be a better way to do this, but test would pass

e35b633

ThomasLecocq added 2 commits January 17, 2022 18:55

reverting image changes

613ebee

div gallery

73c5077

trichter approved these changes Jan 17, 2022

View reviewed changes

ThomasLecocq merged commit 9659cf3 into master Jan 17, 2022

d-chambers mentioned this pull request Jan 17, 2022

Bundled autosummary will stop working with Sphinx 2 #2393

Closed

barsch deleted the sphinx3 branch March 7, 2022 23:39

QuLogic mentioned this pull request Mar 10, 2022

Fix bibliography building with pybtex 0.22+ #2392

Closed

4 tasks

megies mentioned this pull request Mar 29, 2022

cannot find the link to download the pdf doc #3033

Closed

Updating doc building to Sphinx 4.3.2 (python 3) #2662

Updating doc building to Sphinx 4.3.2 (python 3) #2662

Conversation

barsch commented Jun 29, 2020 • edited

megies commented Jun 30, 2020

barsch commented Jun 30, 2020 • edited

megies commented Jun 30, 2020

barsch commented Jun 30, 2020

megies commented Jun 30, 2020

adigitoleo commented Nov 8, 2020

heavelock commented Feb 10, 2021 • edited

heavelock commented Feb 15, 2021 • edited

barsch commented Feb 15, 2021

heavelock commented Feb 15, 2021

heavelock commented Feb 15, 2021 • edited

barsch commented Feb 15, 2021

heavelock commented Feb 15, 2021

barsch commented Feb 15, 2021

heavelock commented Feb 15, 2021

heavelock commented Feb 17, 2021 • edited

heavelock commented Feb 17, 2021

barsch commented Feb 17, 2021

ThomasLecocq commented Jan 10, 2022

ThomasLecocq commented Jan 11, 2022 • edited

trichter commented Jan 14, 2022

ThomasLecocq commented Jan 17, 2022 • edited

trichter commented Jan 17, 2022

barsch commented Jan 17, 2022

trichter commented Jan 17, 2022

barsch commented Jan 17, 2022

trichter commented Jan 17, 2022 • edited

ThomasLecocq commented Jan 17, 2022

barsch commented Jan 17, 2022

ThomasLecocq commented Jan 17, 2022

ThomasLecocq commented Jan 17, 2022

barsch commented Jan 17, 2022 • edited

ThomasLecocq commented Jan 17, 2022 • edited by barsch

ThomasLecocq commented Jan 17, 2022 • edited

ThomasLecocq commented Jan 17, 2022 • edited

barsch commented Jun 29, 2020 •

edited

barsch commented Jun 30, 2020 •

edited

heavelock commented Feb 10, 2021 •

edited

heavelock commented Feb 15, 2021 •

edited

heavelock commented Feb 15, 2021 •

edited

heavelock commented Feb 17, 2021 •

edited

ThomasLecocq commented Jan 11, 2022 •

edited

ThomasLecocq commented Jan 17, 2022 •

edited

trichter commented Jan 17, 2022 •

edited

barsch commented Jan 17, 2022 •

edited

ThomasLecocq commented Jan 17, 2022 •

edited by barsch

ThomasLecocq commented Jan 17, 2022 •

edited

ThomasLecocq commented Jan 17, 2022 •

edited