TST: run refguide-check on rst files in doc/* #14732

mattip · 2019-10-17T08:24:46Z

Run refguide-check on files under doc, skip the 'scipy-sphinx-theme', 'sphinxext', 'neps', and 'release' subdirectories for now.

Changes:

Add a --rst option to refguide-check.py to dive into the doc directory and implement it
Fix up line numbers in error reporting
Clean up some of the comments in refguide-check

Also

Skip running all the tests in the doctests in doc/source/dev/development_environment.rst

doc/source/release/1.11.0-notes.rst

mattip · 2019-10-17T08:38:20Z

The test-run failure demonstrates that the files are being checked :)

tools/refguide_check.py

mattip · 2019-10-17T15:02:58Z

Now that the log files of the refguide-check run prove this is working, I would like to remove the check from CI and get this merged. The next step would be to encourage others to work on fixing all the failures, and then eventually enable this in CI again when it is clean.

eric-wieser · 2019-10-17T15:22:28Z

nit: TST not TEST I think is our usual prefix

doc/source/dev/development_environment.rst

doc/source/user/c-info.ufunc-tutorial.rst

tools/refguide_check.py

mattip · 2019-10-17T16:34:23Z

squashed and force pushed

add a SKIPBLOCK option, but note it must be used on a contiguous block, no blank lines
clean up some reverted changes to files using the new SKIPBLOCK option
add the option to skip a file to the already existing option to skip a single directory or an entire tree

tools/refguide_check.py

bsipocz · 2019-10-17T17:08:49Z

I plan to have a review of this, but for now just a quick question:

add a SKIPBLOCK option, but note it must be used on a contiguous block, no blank lines

have you had a look into pytest-doctestplus? it already has a few directives to skip doctesting certain parts

mattip · 2019-10-17T17:41:35Z

I don't see a SKIPBLOCK directive. It would need to be on the level of a doctest runner, since at the level of the checker all you have is a single example in a block, not the entire block of examples.

We started with refguide in gh-9415 since it was felt doctest would never be up to the task at hand.

bsipocz · 2019-10-17T17:47:32Z

Isn't this the use case you're after?: https://github.com/astropy/astropy/blame/master/docs/nddata/utils.rst#L40

tools/refguide_check.py

mattip · 2019-10-17T18:01:05Z

yes, except the .. doctest-skip:: RST directive doesn't work with refguide (I just tried). Is that supposed to be respected by doctests or is a part of sphinx?

bsipocz · 2019-10-17T18:04:55Z

It's a plugin for pytest, so should be part of doctesting, sphinx doesn't do any code checking. For the bare minimum, you need doctest_plus = enabled in pytest.ini, but I haven't yet get it work with your testing infrastructure either.

And it also works for any text file format, I used it extensively for tex files to make sure code examples in a book manuscript work at least at the time sending it to the publisher :)

mattip · 2019-10-17T18:11:59Z

@bsipocz if I understand you correctly doctest_plus seems to be a totally different framework for docstring testing, so probably needs to be a separate PR/discussion on the mailing list.

bsipocz · 2019-10-17T18:17:45Z

I don't disagree about the mailing list, but what do you mean by totally different framework? It simply extends the standard doctest functionalities.

mattip · 2019-10-17T18:21:47Z

We are using tool/refguide_check.py to do docstring testing. It is not based on pytest, rather is a stand-alone tool that extends the stdlib doctests directly. I don't think we can make it part of a pytest workflow.

tools/refguide_check.py

mattip · 2019-11-04T16:31:37Z

Closes #14707

WarrenWeckesser · 2019-11-14T18:09:41Z

@mattip, overall this looks fine. There are a couple unresolved comments from @eric-wieser. I requested more information for one of them.

mattip · 2019-11-14T23:37:39Z

The test failure is due to a bad install of mingw32 and not connected to this PR.

WarrenWeckesser

LGTM. I'll hold off merging in case @eric-wieser has any additional comments on the last updates.

doc/source/reference/arrays.datetime.rst

charris · 2019-11-17T00:57:06Z

tools/refguide_check.py

 def check_items(all_dict, names, deprecated, others, module_name, dots=True):
+    """Check that `all_dict` is consistent with the `names` in `module_name`


Doesn't follow the docstring standard.

charris · 2019-11-17T00:58:48Z

tools/refguide_check.py

-      'Inf': np.inf,}
+      'Inf': np.inf,
+      'StringIO': io.StringIO,
+    }


Indentation looks off.

moved to column 1

tools/refguide_check.py

charris · 2019-11-17T01:00:48Z

tools/refguide_check.py

+
+
+def check_documentation(base_path, results, args, dots):
+    """Check examples in any *.rst located inside `base_path`.


Needs standard docstring.

fixed formatting

tools/refguide_check.py

charris · 2019-11-17T01:02:54Z

tools/refguide_check.py

+            sys.stderr.write(filename + ' ')
+            sys.stderr.flush()
+
+        tut_results = check_doctests_testfile(filename,


Linebreak after (.

charris · 2019-11-17T01:04:26Z

tools/refguide_check.py

-        for v in mod_results:
-            assert isinstance(v, tuple), v
+            mod_results = []
+            mod_results += check_items(all_dict, names, deprecated, others, module.__name__)


Line too long.

charris · 2019-11-17T01:05:52Z

Style nits.

mattip · 2019-11-18T11:44:30Z

style nits corrected

tools/refguide_check.py

eric-wieser

Looks good, thanks for the persistence.

I think my main confusion was over running the rst checker over python files - it all makes sense to me now we're not doing that. If it turns out I'm wrong and it would make sense to do that, we can follow up in a later PR.

charris · 2019-11-19T14:06:53Z

The docstrings are still not following the docstring standard https://numpydoc.readthedocs.io/en/latest/format.html.

mattip · 2019-11-22T05:59:11Z

The docstrings are still not following the docstring standard

@charris Which ones?

mattip · 2019-11-22T19:11:33Z

@charris there is now an issue about improving documentation xref gh-14961.

charris · 2019-11-23T15:41:45Z

Thanks Matti.

mattip added the 05 - Testing label Oct 17, 2019

eric-wieser reviewed Oct 17, 2019

View reviewed changes

doc/source/release/1.11.0-notes.rst Show resolved Hide resolved

eric-wieser reviewed Oct 17, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

eric-wieser reviewed Oct 17, 2019

View reviewed changes

doc/source/dev/development_environment.rst Outdated Show resolved Hide resolved

eric-wieser reviewed Oct 17, 2019

View reviewed changes

doc/source/user/c-info.ufunc-tutorial.rst Outdated Show resolved Hide resolved

eric-wieser reviewed Oct 17, 2019

View reviewed changes

tools/refguide_check.py Show resolved Hide resolved

TST: run refguide-check on rst files in doc/*

0b474e3

mattip force-pushed the refguide-rst branch from 89382d5 to 0b474e3 Compare October 17, 2019 16:31

eric-wieser reviewed Oct 17, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

TST: changes from review

5af0ca9

eric-wieser reviewed Oct 17, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

rgommers changed the title TEST: run refguide-check on rst files in doc/* TST: run refguide-check on rst files in doc/* Oct 18, 2019

eric-wieser reviewed Oct 29, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

eric-wieser reviewed Oct 29, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

mattip mentioned this pull request Oct 31, 2019

f2py user's guide uses python 2 syntax for print #14812

Closed

MAINT: fixes from review

620934a

eric-wieser reviewed Oct 31, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

eric-wieser reviewed Oct 31, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

MAINT: pep8, remove checks of *.py files from --rst check

96ec81b

WarrenWeckesser approved these changes Nov 15, 2019

View reviewed changes

charris reviewed Nov 17, 2019

View reviewed changes

doc/source/reference/arrays.datetime.rst Show resolved Hide resolved

charris reviewed Nov 17, 2019

View reviewed changes

tools/refguide_check.py Show resolved Hide resolved

charris reviewed Nov 17, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

charris reviewed Nov 17, 2019

View reviewed changes

MAINT: fixes from review

3b8cf7c

eric-wieser reviewed Nov 18, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

eric-wieser reviewed Nov 18, 2019

View reviewed changes

tools/refguide_check.py Outdated Show resolved Hide resolved

MAINT: fixes from review

e057823

eric-wieser approved these changes Nov 18, 2019

View reviewed changes

MAINT: remove deprecation warnings

dff3c1f

mattip mentioned this pull request Nov 22, 2019

DOC: add proper docstrings and documentation to refguide-check #14961

Closed

charris merged commit d76efa7 into numpy:master Nov 23, 2019

mattip mentioned this pull request Jan 26, 2020

MAINT: Use contextmanager in _run_doctests #15430

Merged

sethtroisi mentioned this pull request Jan 29, 2020

MAINT: Cleanup duplicate line in refguide_check #15464

Merged

mattip deleted the refguide-rst branch November 2, 2020 08:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST: run refguide-check on rst files in doc/* #14732

TST: run refguide-check on rst files in doc/* #14732

mattip commented Oct 17, 2019

mattip commented Oct 17, 2019

mattip commented Oct 17, 2019

eric-wieser commented Oct 17, 2019

mattip commented Oct 17, 2019

bsipocz commented Oct 17, 2019

mattip commented Oct 17, 2019

bsipocz commented Oct 17, 2019

mattip commented Oct 17, 2019

bsipocz commented Oct 17, 2019

mattip commented Oct 17, 2019

bsipocz commented Oct 17, 2019

mattip commented Oct 17, 2019

mattip commented Nov 4, 2019

WarrenWeckesser commented Nov 14, 2019

mattip commented Nov 14, 2019

WarrenWeckesser left a comment

charris Nov 17, 2019

mattip Nov 17, 2019

charris Nov 17, 2019

mattip Nov 17, 2019

charris Nov 17, 2019

mattip Nov 17, 2019

charris Nov 17, 2019

mattip Nov 17, 2019

charris Nov 17, 2019

mattip Nov 17, 2019

charris commented Nov 17, 2019

mattip commented Nov 18, 2019

eric-wieser left a comment

charris commented Nov 19, 2019

mattip commented Nov 22, 2019

mattip commented Nov 22, 2019

charris commented Nov 23, 2019

		def check_items(all_dict, names, deprecated, others, module_name, dots=True):
		"""Check that `all_dict` is consistent with the `names` in `module_name`



		def check_documentation(base_path, results, args, dots):
		"""Check examples in any *.rst located inside `base_path`.

TST: run refguide-check on rst files in doc/* #14732

TST: run refguide-check on rst files in doc/* #14732

Conversation

mattip commented Oct 17, 2019

mattip commented Oct 17, 2019

mattip commented Oct 17, 2019

eric-wieser commented Oct 17, 2019

mattip commented Oct 17, 2019

bsipocz commented Oct 17, 2019

mattip commented Oct 17, 2019

bsipocz commented Oct 17, 2019

mattip commented Oct 17, 2019

bsipocz commented Oct 17, 2019

mattip commented Oct 17, 2019

bsipocz commented Oct 17, 2019

mattip commented Oct 17, 2019

mattip commented Nov 4, 2019

WarrenWeckesser commented Nov 14, 2019

mattip commented Nov 14, 2019

WarrenWeckesser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charris commented Nov 17, 2019

mattip commented Nov 18, 2019

eric-wieser left a comment

Choose a reason for hiding this comment

charris commented Nov 19, 2019

mattip commented Nov 22, 2019

mattip commented Nov 22, 2019

charris commented Nov 23, 2019