DRAFT: enabling doctestplus

[bsipocz]

This is a WIP, and I'm opening a PR only to enable @seberg to push to this branch directly.

[seberg]

@bsipocz I pushed a small fix for a merge conflict. But, could you remind me of the correct incantation (currently?), I already added:

 norecursedirs =
+    # Skip potential temporary environments:
+    build
+    benchmarks/env

to the pytest.ini (both can contain a conftest.py which fixed the below issue for it), but I keep getting import errors like:

_pytest.pathlib.ImportPathMismatchError: ('random', '/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/random.py', PosixPath('/Users/sebastianb/forks/numpy/numpy/typing/tests/data/pass/random.py'))

so I think I am probably just approaching running pytest wrong.

EDIT: I think the problem was only that I didn't add --continue-on-collection-errors

[seberg]

There are now quite a few plots popping up again and failures due to deprecations. Maybe we should just ignore deprecation warnings in the docs. Or if we don't want to do that, just remove the examples for clearly deprecated functionality?

[mattip]

remove the examples for clearly deprecated functionality

+1

[mattip]

Would a rebase help clear some of the collection issues?

[seberg]

@bsipocz where could this go and is this OK? This fixes the plt.show() not popping up things. There are some that can just be plot directives, but the histogram re-uses the data and has comments, etc. so I thought this is the easier path.

Maybe we need distinct runs, also so that we can make sure that warnings are not raised as errors?

[bsipocz]

Indeed, this should be the fix for the mpl pop ups.

As for warnings, I think they really should be raised as errors, unless they are handled properly. For doctestplus we can use either +IGNORE_WARNINGS or +SHOW_WARNINGS

[seberg]

I think the collection issues are maybe all expected, i.e. trying to find doctests in places we shouldn't look or just something to do with how I ran it.

[bsipocz]

Thanks Sebastian for the commits! I'm sorry for being too slow with this, I'm trying to come back to it in a more timely manner.

[rgommers]

I'll note that the tool we currently use is tools/refguide_check.py. It is run in CircleCI only. It can also be run via runtests.py and should still be plugged into spin test.

If you want to iterate on CI, perhaps easiest to use that same job (or port it to a single GHA job).