[doc] conf: enumerate llms_txt_exclude notebooks from git, not a live rglob

[ronny-anyscale]

Description

Part of making the incremental RtD doctree cache actually deliver its speedup (DOC-1047). #64414 (producer) and #64482 (consumer reaches the cache) are merged, but RtD still re-reads all ~3,967 docs on every preview: Sphinx restores the cache, then discards the doctree because conf.py evaluates to a different config than the cache was built with. Full audit in DOC-1344; this is the first cut — the single option Sphinx names as the doctree-rebuild trigger.

Root cause

llms_txt_exclude was built from a live rglob("*.ipynb") over doc/source, so its value depends on which notebooks exist when conf.py runs:

• Producer (Buildkite make html, clean tree): conf.py runs before sphinx-collections pulls, so it sees only the 59 git-tracked notebooks.
• Consumer (RtD make rtd, cache-restored tree): the cache captured the ~67 gitignored notebooks sphinx-collections pulled into _collections/…, so conf.py sees 59 + 67.

llms_txt_exclude is an env-affecting config value, so that difference makes Sphinx invalidate the whole restored doctree → [config changed ('llms_txt_exclude')] 3967 added (verified on build 4166140, PR #64482).

Fix

Enumerate notebooks from the git-tracked set (git ls-files "*.ipynb") — identical across environments for a given commit — with a filesystem-scan fallback for non-git checkouts. This equals what the clean producer and the published master build already compute (their conf.py also runs before the pull), so llms-full.txt is unchanged there.

Verified

• Determinism: adding an untracked .ipynb grows the old rglob (126→127) but leaves the git-tracked result stable (59→59).
• Correctness: on a clean checkout rglob == git ls-files; the local 126-vs-59 gap is exactly the 67 gitignored _collections pulled notebooks that cause the producer↔consumer mismatch.

Related issues

First cut for DOC-1344 (doctree-cache config portability); unblocks DOC-1047.

Additional information

Only fully verifiable on a live RtD PR build — the payoff is N changed instead of 3967 added on this PR's preview. This fixes only the env-rebuild trigger; the html_* env-var diffs (version_match, sidebars) still force HTML re-output (much cheaper — no re-read / notebook execution) and the collections function-ref hash remains — both enumerated in DOC-1344.

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 575cc93. Configure here.}

[cursor]

Wrong docname prefix from git

Medium Severity

git ls-files returns paths from the repository root (for example doc/source/...), but the shared conversion only strips .ipynb and never re-bases them under _conf_dir. Sphinx llms_txt_exclude entries must be docnames relative to the source directory, so fnmatch will not exclude tracked notebooks on normal git checkouts.

 

^{Reviewed by Cursor Bugbot for commit 575cc93. Configure here.}

[ronny-anyscale]

git ls-files here runs with cwd=_conf_dir (i.e. doc/source), so it returns paths relative to that directory, not the repository root — e.g. data/examples/foo.ipynb, not doc/source/data/examples/foo.ipynb. After stripping the .ipynb suffix those are already the source-relative docnames llms_txt_exclude matches against, so tracked notebooks are excluded correctly on a normal checkout.

Verified against the tracked set — the resulting docnames are _templates/template, data/examples/..., etc., with no doc/source/ prefix. So no change needed here.

[gemini-code-assist]

Code Review

This pull request updates doc/source/conf.py to retrieve the list of notebook files from the git-tracked set using git ls-files instead of a live filesystem scan, preventing Sphinx from discarding the restored doctree cache due to environment differences. Feedback suggests improving the robustness of this implementation by using the -z flag with git ls-files to safely handle special characters in filenames, and catching general exceptions to prevent build crashes on decoding errors.

[gemini-code-assist]

1. Handle special characters in filenames: Git ls-files might quote filenames with special characters or non-ASCII characters. This can cause pathlib.Path(nb) to fail to parse the path correctly. Using the -z option outputs paths verbatim, separated by NUL bytes (\0), which is the standard and most robust way to parse file paths from Git in Python.
2. Prevent build crashes on decoding errors: When text=True is passed to subprocess.run, Python decodes the output using the default system encoding. If the output contains invalid bytes for that encoding, it raises a UnicodeDecodeError (a subclass of ValueError). Since the current except block only catches (OSError, subprocess.CalledProcessError), a decoding error would crash the entire Sphinx build. Catching Exception ensures that any failure in the Git command safely falls back to the filesystem scan.

Suggested change

	try:
	_notebooks = subprocess.run(
	["git", "ls-files", "*.ipynb"],
	cwd=_conf_dir,
	capture_output=True,
	text=True,
	check=True,
	).stdout.splitlines()
	except (OSError, subprocess.CalledProcessError):
	try:
	_git_out = subprocess.run(
	["git", "ls-files", "-z", "*.ipynb"],
	cwd=_conf_dir,
	capture_output=True,
	text=True,
	check=True,
	).stdout
	_notebooks = [nb for nb in _git_out.split("\x00") if nb]
	except Exception: