gh-36591: Test that everything can be imported during ci

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes #1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes #12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

Currently you cannot import many modules directly. We use `pytest
--collect-only` to test during ci that the imports are working. As a
first step, we run pytest through sage, which makes sure that `sage.all`
is loaded. Once the reported errors are fixed (e.g. by filtering out
modules that fail because of FeatureNotPresentError), we can check that
importing without `sage.all` works.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- #12345: short description why this is a dependency
- #34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: #36591
Reported by: Tobias Diez
Reviewer(s): Matthias Köppe, Tobias Diez

.github/workflows/build.yml

@@ -172,11 +172,23 @@ jobs:
           MAKE: make -j2 --output-sync=recurse
           SAGE_NUM_THREADS: 2
 
+      - name: Check that all modules can be imported
+        run: |
+          # The following command checks that all modules can be imported.
+          # The output also includes a long list of modules together with the number of tests in each module.
+          # This can be ignored.
+          ../sage -python -m pip install pytest-xdist
+          ../sage -python -m pytest -c tox.ini -qq --doctest --collect-only || true
+        working-directory: ./worktree-image/src
+        env:
+          # Increase the length of the lines in the "short summary"
+          COLUMNS: 120
+
       - name: Pytest
         if: contains(github.ref, 'pytest')
         run: |
           ../sage -python -m pip install coverage pytest-xdist
-          ../sage -python -m coverage run -m pytest -c tox.ini --doctest-modules || true
+          ../sage -python -m coverage run -m pytest -c tox.ini --doctest || true
         working-directory: ./worktree-image/src
         env:
           # Increase the length of the lines in the "short summary"

.vscode/launch.json

@@ -1,6 +1,20 @@
 {
     "version": "0.2.0",
     "configurations": [
+        {
+            "name": "Sage: Pytest Doctests",
+            "type": "python",
+            "request": "launch",
+            "module": "pytest",
+            "args": [
+                "-c",
+                "src/tox.ini",
+                "--doctest",
+                "${file}"
+            ],
+            "console": "integratedTerminal",
+            "justMyCode": false
+        },
         {
             "name": "Sage: Pytest",
             "type": "python",
@@ -33,4 +47,4 @@
             "justMyCode": false
         }
     ],
-}
+}

src/bin/sage

@@ -1019,10 +1019,10 @@ if [ "$1" = '-pytest' -o "$1" = '--pytest' ]; then
             for a in $*; do
                 case $a in
                     -*) ;;
-                    *)  exec pytest --rootdir="$SAGE_SRC" --doctest-modules "$@"
+                    *)  exec pytest --rootdir="$SAGE_SRC" --doctest "$@"
                 esac
             done
-            exec pytest --rootdir="$SAGE_SRC" --doctest-modules "$@" "$SAGE_SRC"
+            exec pytest --rootdir="$SAGE_SRC" --doctest "$@" "$SAGE_SRC"
         else
             echo "Run 'sage -i pytest' to install"
             exit 1

src/conftest.py

@@ -22,8 +22,7 @@
     _patch_unwrap_mock_aware,
     get_optionflags,
 )
-from _pytest.pathlib import import_path, ImportMode
-
+from _pytest.pathlib import ImportMode, import_path
 from sage.doctest.parsing import SageDocTestParser, SageOutputChecker
 
 
@@ -141,10 +140,28 @@ def pytest_collect_file(
         # hit this here if someone explicitly runs `pytest some_file.pyx`.
         return IgnoreCollector.from_parent(parent)
     elif file_path.suffix == ".py":
-        if parent.config.option.doctestmodules:
+        if parent.config.option.doctest:
+            if file_path.name == "__main__.py":
+                # We don't allow tests to be defined in __main__.py files (because their import will fail).
+                return IgnoreCollector.from_parent(parent)
+            if file_path.name == "postprocess.py" and file_path.parent.name == "nbconvert":
+                # This is an executable file.
+                return IgnoreCollector.from_parent(parent)
             return SageDoctestModule.from_parent(parent, path=file_path)
 
 
+def pytest_addoption(parser):
+    # Add a command line option to run doctests
+    # (we don't use the built-in --doctest-modules option because then doctests are collected twice)
+    group = parser.getgroup("collect")
+    group.addoption(
+        "--doctest",
+        action="store_true",
+        default=False,
+        help="Run doctests in all .py modules",
+        dest="doctest",
+    )
+
 @pytest.fixture(autouse=True, scope="session")
 def add_imports(doctest_namespace: dict[str, Any]):
     """