DOC: Doc build for a single doc made much faster, and clean up #24428
Conversation
| import inspect | ||
| import importlib | ||
| import logging | ||
| import warnings | ||
|
|
||
| import jinja2 | ||
| from sphinx.ext.autosummary import _import_by_name | ||
| from numpydoc.docscrape import NumpyDocString | ||
| from numpydoc.docscrape_sphinx import SphinxDocString |
There was a problem hiding this comment.
you can remove the raw_input yes? as we only build on py3 now?
There was a problem hiding this comment.
good point, and actually it wasn't used, so should still run in py2
There was a problem hiding this comment.
right, its actualy totally fine to have only py3 doc building now.
Codecov Report
@@ Coverage Diff @@
## master #24428 +/- ##
==========================================
- Coverage 92.3% 43% -49.3%
==========================================
Files 163 163
Lines 51966 51966
==========================================
- Hits 47967 22349 -25618
- Misses 3999 29617 +25618
Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## master #24428 +/- ##
==========================================
+ Coverage 92.3% 92.32% +0.02%
==========================================
Files 163 166 +3
Lines 51969 52328 +359
==========================================
+ Hits 47968 48310 +342
- Misses 4001 4018 +17
Continue to review full report at Codecov.
|
|
@TomAugspurger if you have time and can take a look at this. Besides simplifying the code to build the docs, this will allow to build a single page much faster. That will be useful for fixing the warnings, and I'd like to time the build of each page, and see if we can reduce the time of the slowest examples. |
doc/make.py
Outdated
| pass | ||
| return single_doc[len('pandas.'):] | ||
| else: | ||
| raise ValueError('--single value should be a valid path to a ' |
There was a problem hiding this comment.
Could you print out single_doc here, so the user sees what they passed?
| @@ -326,7 +214,7 @@ def main(): | |||
| help='command to run: {}'.format(', '.join(cmds))) | |||
| argparser.add_argument('--num-jobs', | |||
| type=int, | |||
| default=1, | |||
| default=0, | |||
There was a problem hiding this comment.
In the DocBuilder init, the default is 1. Make them both the same?
What does a value of 0 do?
There was a problem hiding this comment.
Good point. with 0 I don't pass a -j value to sphinx-build, while with 1, I pass -j 1, which is actually the same (unless sphinx changes its default).
It's a bit weird, because the -j parameter is supposed to make sphinx-build run with multiple cores, but it actually doesn't work, and the process takes exactly the same with -j 1 and -j 4. It could make sense to remove this option, but I guess at some point sphinx will be fixed at some point, so I guess it's worth leaving it.
I'm happy setting our default to 0 (use sphinx default), 1, or auto, just let me know if you have a preference.
I address your other comment.
There was a problem hiding this comment.
Hmm I think multiple cores speeds things up for me, at least for a full doc build.
There was a problem hiding this comment.
I checked it with time in Linux, 1 vs 4 cores, it took 17m30s in both cases (with a difference of less than 5 seconds between them). I tried it in the past with a different computer (also Linux) and was the same.
* upstream/master: DOC: Fixing broken references in the docs (pandas-dev#24497) DOC: Splitting api.rst in several files (pandas-dev#24462) Fix misdescription in escapechar (pandas-dev#24490) Floor and ceil methods during pandas.eval which are provided by numexpr (pandas-dev#24355) BUG: Pandas any() returning false with true values present (GH pandas-dev#23070) (pandas-dev#24434) Misc separable pieces of pandas-dev#24024 (pandas-dev#24488) use capsys.readouterr() as named tuple (pandas-dev#24489) REF/TST: replace capture_stderr with pytest capsys fixture (pandas-dev#24496) TST- Fixing issue with test_parquet test unexpectedly passing (pandas-dev#24480) DOC: Doc build for a single doc made much faster, and clean up (pandas-dev#24428) BUG: Fix+test timezone-preservation in DTA.repeat (pandas-dev#24483) Implement reductions from pandas-dev#24024 (pandas-dev#24484)
…strings * upstream/master: TST: Skip db tests unless explicitly specified in -m pattern (pandas-dev#24492) Mix EA into DTA/TDA; part of 24024 (pandas-dev#24502) DOC: Fix building of a single API document (pandas-dev#24506) DOC: Fixing broken references in the docs (pandas-dev#24497) DOC: Splitting api.rst in several files (pandas-dev#24462) Fix misdescription in escapechar (pandas-dev#24490) Floor and ceil methods during pandas.eval which are provided by numexpr (pandas-dev#24355) BUG: Pandas any() returning false with true values present (GH pandas-dev#23070) (pandas-dev#24434) Misc separable pieces of pandas-dev#24024 (pandas-dev#24488) use capsys.readouterr() as named tuple (pandas-dev#24489) REF/TST: replace capture_stderr with pytest capsys fixture (pandas-dev#24496) TST- Fixing issue with test_parquet test unexpectedly passing (pandas-dev#24480) DOC: Doc build for a single doc made much faster, and clean up (pandas-dev#24428) BUG: Fix+test timezone-preservation in DTA.repeat (pandas-dev#24483) Implement reductions from pandas-dev#24024 (pandas-dev#24484)
The main goal of this PR is to be able to build single pages faster and without building anything else, so detecting and fixing the warnings (and other problems) in doc pages should be much more efficient.
Additionally, those things have been addressed:
index.rst.template) fromdoc/make.pytodoc/source/conf.py, which makes things simpler, and makes it possible to callsphinx-builddirectly--singleis used, do not build anything else than the required page (before all whatsnew pages where additionally build), which makes it much faster@jorisvandenbossche can you take a look please?