Adding Documentation Notebooks to Tests

All documentation notebooks in toponymy/doc are tested automatically on a weekly basis, or on PRs as relevant. This helps to keep the documentation and the code in sync. However, there are a few things that are necessary to make sure that a new notebook is test-friendly.

1. All external LLM Namer calls should use OpenAINamer (it will be replaced by OllamaNamer in tests)
2. Use an existing dataset and its corresponding load function (see toponymy/tools/notebook_data_load.py ). Currently datasets include:
   • 20 newsgroups: load_newsgroups
   • Arxiv Machine Learning: load_arxiv_ml
   • Arxiv Category Theory: load_arxiv_ct
   • Arxiv Comp Sci: load_bundled_arxiv (the data is bundled into the library as part of the examples directory)
3. Make sure any save/load functionality uses notebook_output_dir (from toponymy/tools/notebook_data_load.py) in its path.
4. Make sure your embedding model is not using an LLM API call (SentenceTransformer("paraphrase-MiniLM-L3-v2") should work well enough for most notebook examples). (Support for fallbacks of embedding wrappers is not included yet)

Note: If these load helpers are not sufficient and you need a separate dataset you'll need to add a new set of load functions to toponymy/tools/notebook_data_load.py. See Notebook Testing Reference for information on how this works.

At this point, you can try running your notebook as if in a test environment by

NOTEBOOK_TESTING=true uv run toponymy/tools/notebook_runner.py doc/my_notebook_name.ipynb --instrument

Note by default this runs with the toponymy-uv kernel name so you can either pass another kernel name via --kernel-name , or register your environment as such:

uv run python -m ipykernel install --user --name toponymy-uv

Now, this may not run as expected, but here are some things to check for:

• Check that your dev environment has been set up (and includes Ollama if you have an OpenAINamer in your code).
• Check if it runs when bypassing the test dataset shrinking with use_small=False passed into your data loading function. If it runs on the full dataset under the test environment then there are a couple of options 1) try modifying some non-expository parameters and choices to get it to run to completion with the smaller dataset 2) run with use_small=False if the notebook runs to completion quickly anyway (say <2 mins like clustering_options.ipynb). Strong preference for 1) where possible.

At this point, take note of how long the notebook took to run (this should be handed to you if you ran with --instrument), and generously add a buffer to that for how long you think it could run in CI (a safe bet is to double your local run time). This will be your timeout.

Under NOTEBOOK_CONFIG in toponymy/tests/test_doc_notebooks.py, add an entry for your notebook:

"my_notebook_name.ipynb": {
	"has_openainamer": <True/False depending on if you used OpenAINamer>,
	"run_in_pr": False,
	"timeout": <your value from above in seconds>,
},

The run_in_pr setting should always be False unless the notebook falls into the Getting Started section of the documentation. In that case your notebook run time should also be fairly short since it will run tests on the notebook in every PR in addition to the weekly notebook test run.

Now you're ready to check if your notebook will pass tests. Depending on whether your notebook uses OpenAINamer or not, run either

uv run pytest -k test_doc_notebook_no_openainamer

or

uv run pytest -k test_doc_notebook_has_openainamer

Yes, this will run the other notebooks of the same flavour, but consider this also a sanity check that the other tests run successfully so you can tell the difference between a global issue or a notebook specific issue if things aren't working right.

Morally a notebook should not have to change anything that affects exposition to fit into the testing infrastructure. If there's something you're not sure how to adapt, file an issue and take a look at the Notebook Testing Reference to dive deeper into how the notebook tests work.