chore: add CI integrity check for docs/doc-deps.yaml

[igerber]

Summary

• Add tests/test_doc_deps_integrity.py — a pure-stdlib (no PyYAML) line-parser CI check that guards docs/doc-deps.yaml against (1) stale referenced paths and (2) unmapped public diff_diff source modules. Wired into the docs-tests.yml doc-snippets job with the PYTHONPATH=. prefix (conftest imports diff_diff at collection, and that job doesn't pip install diff_diff).
• Map diff_diff/agent_workflow.py — the lone unmapped public module — to its agent guide (llms-autonomous.txt).
• Re-file TODO.md L174 (.txt guide-snippet validation) as not low-lift: a probe of the 112 guide code fences found only ~20% standalone-runnable (the rest are API-signature references, context fragments, or dataset-shape-specific). Remove the now-shipped L175 row.

The map is consumed by /docs-impact, /pre-merge-check, and CLAUDE.md but had no CI guard; this prevents future drift (0 stale today). Private (_-prefixed) modules and __init__.py are exempt by convention; intentional public exceptions go in an (initially empty) _ALLOWLIST.

Methodology references (required if estimator / math changes)

• Method name(s): N/A — no methodology/estimator/math changes (doc-deps map + CI wiring + new test only)
• Paper / source link(s): N/A
• Any intentional deviations from the source (and why): None

Validation

• Tests added/updated: tests/test_doc_deps_integrity.py (142 cases pass in 0.04s). Negative controls confirmed both guards fail-closed: a bogus path: fails test_referenced_paths_exist, and an unmapped public module fails test_all_public_sources_mapped.
• No-regression: tests/test_doc_snippets.py still collects (116 tests); no diff_diff/ source touched.

Security / privacy

• Confirm no secrets/PII in this PR: Yes

🤖 Generated with Claude Code

[github-actions]

Overall assessment

⚠️ Needs changes — there are no estimator or methodology changes in this diff, but the new CI guard does not enforce the full doc-deps grouping contract. That leaves a false-green case where CI can pass even though /docs-impact / /pre-merge-check would not have a resolvable docs mapping.

Executive summary

• No estimator, weighting, variance/SE, identification, or default-behavior code changed; I found no methodology or REGISTRY.md issues in scope.
• The added docs/doc-deps.yaml entry for diff_diff/agent_workflow.py closes the current unmapped-public-module gap; a repo-wide spot check shows no current public diff_diff/*.py offenders.
• P1: test_all_public_sources_mapped() treats any groups: member as “mapped,” but docs/doc-deps.yaml says group members resolve via the group’s first entry. A broken group can therefore pass CI while lookup still fails. (docs/doc-deps.yaml:L20-L21, tests/test_doc_deps_integrity.py:L89-L103)
• The .txt guide-validation deferral is properly tracked in TODO.md, so that deferred work is non-blocking under the project’s review policy. (TODO.md:L174-L174)
• No security or performance concerns stood out in the changed files.

Methodology

• No findings. The diff is limited to CI wiring, doc-deps metadata, TODO.md, and a new integrity test; no estimator/math/inference/default-behavior code changed, so docs/methodology/REGISTRY.md is not implicated here.

Code Quality

• No findings.

Performance

• No findings.

Maintainability

• No findings beyond the test-contract issue called out below.

Tech Debt

• P3 (informational) — Impact: none. The PR re-scopes the .txt guide-snippet limitation into TODO.md, which is the project’s accepted mechanism for deferrable documentation/test debt. Concrete fix: none required in this PR. (TODO.md:L174-L174)

Security

• No findings.

Documentation/Tests

• P1 — Impact: the new integrity test validates flat membership, not resolvable mapping. Per the map’s own contract, groups: members “resolve to the first entry” for doc lookup, but the test currently counts every group member as sufficient on its own. A future PR could add groups: foo: [diff_diff/foo.py, diff_diff/foo_helper.py] without adding sources: diff_diff/foo.py: ...; CI would still pass, while consumers would have no primary mapping to resolve through. Concrete fix: parse groups structurally and assert every group is non-empty and that members[0] exists in _SOURCE_KEYS; alternatively, only count group members as mapped when their group’s primary member has a sources: entry. (docs/doc-deps.yaml:L20-L21, tests/test_doc_deps_integrity.py:L89-L103)

Path to Approval

1. Strengthen tests/test_doc_deps_integrity.py so groups: only satisfy the mapping requirement when the group’s primary member (the first entry) also has a corresponding sources: entry. A dedicated assertion on that invariant is enough to move this PR to ✅.

Validation note: pytest is not installed in this sandbox, so I could not run the new test file directly. I did replicate its parser logic with a standalone Python check and confirmed that, with the added agent_workflow.py entry, the current tree has no unmapped public diff_diff/*.py modules.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 3809f220cf05811cbdba05bb46f3b5f43a9c1ba8

---

Overall assessment

✅ Looks good — the prior P1 from the last AI review is resolved, and I found no new unmitigated P0/P1 issues in the changed files.

Executive summary

• No estimator, weighting, variance/SE, identification, or default-behavior code changed in this diff, so there are no methodology findings and docs/methodology/REGISTRY.md is not implicated.
• The prior false-green issue is fixed: the new test_group_primaries_have_sources_entry() now enforces the documented groups: contract that members resolve through the first entry. See docs/doc-deps.yaml:L20-L21 and tests/test_doc_deps_integrity.py:L145-L160.
• diff_diff/agent_workflow.py is now mapped in docs/doc-deps.yaml, closing the unmapped public-module gap this PR set out to address. See docs/doc-deps.yaml:L455-L460.
• CI wiring is complete: the docs workflow now triggers on the new test file and runs the new integrity check with the same PYTHONPATH=. DIFF_DIFF_BACKEND=python setup as the snippet test. See .github/workflows/docs-tests.yml:L7-L20 and .github/workflows/docs-tests.yml:L69-L80.
• The .txt guide-snippet limitation remains properly tracked in TODO.md, so it is non-blocking under the project’s deferred-work policy. See TODO.md:L174.

Methodology

• No findings. Affected method(s): none. The diff is limited to docs CI wiring, doc-deps metadata, TODO tracking, and a new integrity test.

Code Quality

• No findings.

Performance

• No findings.

Maintainability

• No findings. The rereviewed contract gap from the previous review is now explicitly tested rather than left implicit in comments/docstrings. See tests/test_doc_deps_integrity.py:L76-L102 and tests/test_doc_deps_integrity.py:L145-L160.

Tech Debt

• Severity: P3 (informational). Impact: none in this PR. The remaining .txt guide-snippet validation limitation is explicitly tracked in TODO.md, which is the project-approved mechanism for deferred docs/test debt. Concrete fix: none required in this PR. See TODO.md:L174.

Security

• No findings.

Documentation/Tests

• No findings. The previous P1 is addressed: groups: are now checked for both non-emptiness and a resolvable primary sources: entry, matching the documented lookup contract. See docs/doc-deps.yaml:L20-L21 and tests/test_doc_deps_integrity.py:L145-L160.
• No findings. The workflow changes correctly ensure the new test runs in CI whenever relevant files change. See .github/workflows/docs-tests.yml:L21-L34 and .github/workflows/docs-tests.yml:L75-L80.

Validation note: pytest is not installed in this sandbox (pytest: command not found), so I could not execute tests/test_doc_deps_integrity.py directly. I did replicate the parser logic against the current tree and found 0 unmapped public diff_diff modules and 0 broken groups: primaries.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 481c2d78493324fa9465afd2c327fcf054ea533e

---

Overall assessment

✅ Looks good — this re-review resolves the prior P1, and I found no new unmitigated P0/P1 issues in the changed files. One non-blocking P2 test-parser robustness gap remains.

Executive summary

• The prior false-green issue is addressed: the new primary-resolution check now enforces that each groups: block is non-empty and that its first member has a sources: entry. See tests/test_doc_deps_integrity.py:L157-L173.
• No estimator, weighting, variance/SE, identification, or default-behavior code changed in this diff, so there are no methodology findings and docs/methodology/REGISTRY.md is not implicated.
• diff_diff/agent_workflow.py is now mapped in docs/doc-deps.yaml, and reproducing the PR’s scan logic against the current tree found no unmapped public diff_diff modules. See docs/doc-deps.yaml:L455-L460.
• CI wiring is complete: the docs workflow now triggers on the new test file and runs it with the same PYTHONPATH=. DIFF_DIFF_BACKEND=python setup needed by tests/conftest.py imports. See .github/workflows/docs-tests.yml:L69-L80 and tests/conftest.py:L185-L189.
• P2: _parse_groups() stops on any unindented non-whitespace line, so a valid YAML comment inserted between group definitions would silently skip later groups and can reintroduce false-green coverage for the exact primary-resolution contract this PR is meant to guard. See tests/test_doc_deps_integrity.py:L98-L104.

Methodology

• No findings. Affected method(s): none. The diff is limited to docs CI wiring, docs/doc-deps.yaml, TODO.md, and a new integrity test.

Code Quality

• No findings.

Performance

• No findings.

Maintainability

• No findings.

Tech Debt

• Severity: P3 (informational). Impact: none for approval. The remaining .txt guide executable-snippet gap is explicitly tracked in TODO.md, so it is non-blocking under the project’s deferred-work policy. Concrete fix: none required in this PR. See TODO.md:L174.

Security

• No findings.

Documentation/Tests

• Severity: P2. Impact: _parse_groups() terminates on any unindented non-whitespace line, not just the next real top-level YAML key. If a maintainer inserts a top-level comment between groups: entries, later groups disappear from _GROUPS, and test_group_primaries_have_sources_entry() can false-green on those skipped groups even though test_all_public_sources_mapped() still treats their members as mapped. Concrete fix: while inside groups:, ignore blank/comment lines and break only on the next actual top-level key such as sources:; then add a guard that the flattened _GROUPS members exactly match _GROUP_MEMBERS so truncation fails loudly. See tests/test_doc_deps_integrity.py:L98-L104 and tests/test_doc_deps_integrity.py:L117-L127.
• No other findings. The prior P1 is resolved, and the new agent_workflow.py mapping closes the public-module gap this PR set out to address. See docs/doc-deps.yaml:L20-L21, docs/doc-deps.yaml:L455-L460, and tests/test_doc_deps_integrity.py:L157-L173.

Validation note: pytest is not installed in this sandbox (pytest: command not found), so I could not execute tests/test_doc_deps_integrity.py directly. I did reproduce its source/mapping scan against the checked-out tree and found 0 current unmapped public modules and 0 broken group primaries.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 906f517160f9cc6e5f097fd907eda66dcc1283e5

---

Overall Assessment

✅ Looks good — this re-review resolves the prior parser false-green gap, and I found no new unmitigated P0/P1 issues in the changed files.

Executive Summary

• The prior groups: parser-truncation issue is addressed: _parse_groups() now ignores top-level comments while inside groups:, and test_doc_deps_parsed_nonempty() cross-checks structural vs. flat group-member parsing so future truncation fails closed. tests/test_doc_deps_integrity.py:L102-L141
• The new primary-resolution check matches the documented doc-deps contract that group members resolve through the first entry, so dead-end group mappings now fail explicitly instead of false-greening. docs/doc-deps.yaml:L20-L21, tests/test_doc_deps_integrity.py:L171-L186
• No estimator, weighting, variance/SE, identification, or default-behavior code changed in this diff, so there are no methodology findings.
• diff_diff/agent_workflow.py is now mapped in docs/doc-deps.yaml; reproducing the scan logic against the current checkout found 0 unmapped public diff_diff modules, 0 broken group primaries, and 0 missing referenced paths. docs/doc-deps.yaml:L455-L460, tests/test_doc_deps_integrity.py:L144-L186
• CI wiring is complete: the docs workflow now triggers on the new test file and runs it with the same PYTHONPATH=. DIFF_DIFF_BACKEND=python setup as the snippet job. .github/workflows/docs-tests.yml:L7-L10, .github/workflows/docs-tests.yml:L25-L29, .github/workflows/docs-tests.yml:L69-L80
• The .txt guide-snippet limitation remains explicitly tracked and re-scoped in TODO.md, so it is non-blocking under the project’s deferred-work policy. TODO.md:L174

Methodology

• No findings. Affected method(s): none. The diff is limited to docs CI wiring, docs/doc-deps.yaml, TODO.md, and a new integrity test. .github/workflows/docs-tests.yml:L45-L80, docs/doc-deps.yaml:L455-L460, tests/test_doc_deps_integrity.py:L1-L187

Code Quality

• No findings. The parser change is targeted and fail-closed for the previously reported top-level-comment truncation case. tests/test_doc_deps_integrity.py:L102-L141

Performance

• No findings. The new check is linear in docs/doc-deps.yaml plus a single diff_diff/**/*.py scan, which is appropriate for a docs-only CI job. tests/test_doc_deps_integrity.py:L67-L85, tests/test_doc_deps_integrity.py:L153-L168

Maintainability

• No findings. The source-tree skip avoids breaking installed-package test environments, and the agent_workflow.py mapping closes the only current unmapped public module in this area. tests/test_doc_deps_integrity.py:L37-L47, docs/doc-deps.yaml:L455-L460

Tech Debt

• Severity: P3
• Impact: None for approval. The .txt guide executable-snippet gap is explicitly tracked and re-scoped, so it is non-blocking under the deferred-work policy. TODO.md:L174
• Concrete fix: None required in this PR.

Security

• No findings. The PR adds read-only repo integrity checks and CI invocation only; it does not expand trust of PR-controlled code or secrets handling. .github/workflows/docs-tests.yml:L65-L80, tests/test_doc_deps_integrity.py:L37-L47

Documentation/Tests

• No findings. The prior review’s parser robustness concern is addressed by skipping top-level comments inside groups: and by asserting structural/flat group-member parity; the workflow also now executes the integrity test under the required import environment. tests/test_doc_deps_integrity.py:L102-L141, tests/test_doc_deps_integrity.py:L171-L186, .github/workflows/docs-tests.yml:L69-L80

Validation note: pytest is not installed in this sandbox (/usr/bin/python: No module named pytest), so I could not run tests/test_doc_deps_integrity.py directly. I did reproduce its scan logic against the checkout with a direct Python script and observed 0 unmapped public modules, 0 broken group primaries, and 0 missing referenced paths.