Follow-up #2090: log GitNexus skip evidence for docs drift scan

[stranske]

Source: Issue #2090

Closes #2090

Automated Status Summary

Scope

Operational documentation drifts faster than humans remember to update it. The 2026-05-13 weekly cycle confirmed three drift instances in source-of-truth docs (README.md model versions, docs/ci/WORKFLOWS.md autofix self-contradiction, docs/ops/REPO_REVIEW_PROCESS.md pointing at the pre-Phase-4 entry point). Each was found ad-hoc by a round-1 reviewer agent looking at a different design slice. There is no recurring mechanism that catches drift between weekly review cycles — drift just accumulates until someone happens to read the affected doc and notice.

CLAUDE.md flags the failure mode explicitly ("agents read one doc, treat its signal as system ground truth, take the wrong action"). The damage compounds because agents read these docs as authoritative input to their own reasoning.

A per-PR CI gate is overkill — slight staleness is tolerable. A per-doc-instance issue tree is wrong — docs have mutual references and need coordinated fixes. The right tool is a weekly scan that surfaces drift in the existing desktop reminder, with one bundled remediation issue per repo per cycle when drift is found.

Tasks

• Add config/source_of_truth_docs.yml with one entry per repo listing the source-of-truth doc paths. For stranske/Workflows, seed it with the ~13 docs CLAUDE.md cites (README.md, docs/INTEGRATION_GUIDE.md, docs/ops/REPO_REVIEW_PROCESS.md, docs/keepalive/GoalsAndPlumbing.md, docs/AGENTS_POLICY.md, docs/LABELS.md, docs/keepalive/Agents.md, docs/ci/WORKFLOWS.md, docs/MODEL_MANAGEMENT.md, docs/WORKFLOW_GUIDE.md, docs/ops/REPO_REVIEW_ROUND2_PROTOCOL.md, docs/ops/REPO_REVIEW_ROUND1_SCHEMA.md, AGENTS.md).
• Write scripts/repo_review_docs_drift_scan.py. For each (repo, doc) in the config: invoke claude with a focused prompt that reads the doc + relevant implementation files (selected by rg queries against terms the doc mentions) and emits a JSON list of drift instances. Each instance has {doc_path, claim, authoritative_source, classification} where classification ∈ {stale, contradictory, accurate-no-drift}.
• In the per-doc prompt, instruct claude: (a) use rg for command/identifier/filename existence checks; (b) call gitnexus context|impact for behavioral claims about call paths when the local GitNexus map is current; (c) skip behavioral checks gracefully when .gitnexus/meta.json reports stale.
• Add coordinator wiring: new step 5b. docs-drift scan after 5. backlog-scan and before 6. notify. Same retry/timeout pattern as backlog-scan; failures are non-fatal.
• Update scripts/repo_review_notify.py: load <output_dir>/docs-drift-scan.json if present; render a "Doc drift detected" section in the desktop reminder, grouped by repo, with one gh issue create snippet per affected repo to file a holistic remediation issue.
• Document the new step in the cron's automation TOML prompt under "Operational notes" with the same depth as the existing backlog-scan note.
• Seed the scanner with the 3 drifts confirmed this cycle (README.md model versions, WORKFLOWS.md autofix contradiction, REPO_REVIEW_PROCESS.md Phase-4 entry point) as integration-test fixtures so the scanner's logic is exercised against real cases.

Acceptance criteria

• python scripts/repo_review_docs_drift_scan.py --registry config/repo_review_registry.json --out /tmp/test-drift.json runs on Workflows-steward and returns at least the 3 known drifts plus a clean (accurate-no-drift) classification for at least 5 other source-of-truth docs.
• pytest tests/scripts/test_repo_review_docs_drift_scan.py exercises the classifier logic against the seeded fixture drifts and verifies they classify as stale / contradictory not accurate-no-drift.
• A full cron dry-run (with --skip-preflight --skip-auto-archive) produces a <output_dir>/docs-drift-scan.json AND a desktop reminder file whose "Doc drift detected" section lists the seeded drifts.
• The desktop reminder's "Doc drift" section gives one gh issue create snippet per repo with non-empty drift, NOT one per doc.
• When .gitnexus/meta.json is stale or missing, the scanner completes with the call-graph checks gracefully skipped (logged but not fatal). Verify with rg 'skipping behavioral check' <output_dir>/logs/coordinator/docs-drift-scan.log.
• The cron's automation.toml "Operational notes" section documents the new step, where its output lives, and the expected runtime (~5–10 min across 9 repos).

[github-actions]

Runner dispatch state for codex on PR #2097. Do not edit.

[agents-workflows-bot]

🤖 Keepalive Loop Status

PR #2097 | Agent: Codex | Iteration 5+7 🚀 extended

Current State

Metric	Value
Iteration progress	[##########] 5/5 5 base + 7 extended = 12 total
Action	run (agent-run-skipped)
Agent status	✅ ALL TASKS COMPLETE
Gate	success
Tasks	13/13 complete
Timeout	45 min (default)
Timeout usage	4m elapsed (9%, 41m remaining)
Keepalive	✅ enabled
Autofix	❌ disabled

Last Codex Run

Result	Value
Status	⏭️ Skipped
Reason	agent-run-skipped

To retry:

• Add the agent:retry label, OR
• Wait for conditions to resolve (e.g., Gate success, labels present)

🔍 Failure Classification

| Error type | infrastructure |
| Error category | transient |
| Suggested recovery | Capture logs and context; retry once and escalate if the issue persists. |

[agents-workflows-bot]

Keepalive Work Log (click to expand)

#	Time (UTC)	Agent	Action	Result	Files	Tasks	Progress	Commit	Gate
0	2026-05-14 03:04:27	Codex	wait (gate-cancelled-transient-transient)	skipped	—	0	0/13	—	cancelled
1	2026-05-14 03:08:20	Codex	run (bypass-rate-limit-gate)	success	2 file(s)	+2	2/13	943223b	cancelled
2	2026-05-14 03:12:01	Codex	run (ready)	success	2 file(s)	+1	3/13	328485e	success
3	2026-05-14 03:16:15	Codex	run (ready)	success	2 file(s)	+1	4/13	a870922	success
3	2026-05-14 03:20:30	Codex	run (agent-run-failed)	failure	2 file(s)	0	4/13	—	success
3	2026-05-14 03:21:12	Codex	wait (gate-not-success)	skipped	—	0	4/13	—	failure
4	2026-05-14 03:32:14	Codex	run (ready)	success	2 file(s)	+2	6/13	6e603d2	success
5	2026-05-14 03:42:05	Codex	run (ready)	success	4 file(s)	+1	7/13	0399cb8	success
6	2026-05-14 03:53:33	Codex	run (ready-extended)	success	4 file(s)	0	7/13	5265b25	success
7	2026-05-14 04:05:20	Codex	fix (fix-lint)	success	2 file(s)	0	7/13	f0bc9db	failure
8	2026-05-14 04:09:08	Codex	run (ready-extended)	success	2 file(s)	+2	9/13	6f95d47	success
8	2026-05-14 04:09:59	Codex	wait (gate-not-success)	skipped	—	0	9/13	—	failure
9	2026-05-14 04:18:31	Codex	run (bypass-rate-limit-gate)	success	2 file(s)	+3	12/13	6e116e4	cancelled
10	2026-05-14 04:25:30	Codex	run (ready-extended)	success	2 file(s)	+1	13/13	93cf6e8	success
11	2026-05-14 04:31:24	Claude	run (verify-acceptance)	success	—	0	13/13	—	success
12	2026-05-14 04:34:14	Codex	run (scope-violation)	success	2 file(s)	0	13/13	—	success
12	2026-05-14 04:56:25	Codex	wait (gate-not-success)	skipped	—	0	13/13	—	—
12	2026-05-14 04:59:48	Codex	run (agent-run-skipped)	skipped	—	0	13/13	—	success

[Copilot]

Pull request overview

Adds explicit logging when the docs-drift scanner encounters a missing or stale GitNexus map, satisfying the acceptance criterion that requires skipping behavioral check evidence in the coordinator log. A regression test verifies the message is emitted via stdout (which the coordinator captures into its log file).

Changes:

• Emit a [docs-drift-scan] ... skipping behavioral check ... line from scan_doc when GitNexus status is missing or stale.
• Add a unit test exercising the new log emission with a fake invoker.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
scripts/repo_review_docs_drift_scan.py	Print skip-evidence line before doc-existence check when GitNexus map is missing/stale.
tests/scripts/test_repo_review_docs_drift_scan.py	New test asserting the skip message appears on stdout for a missing GitNexus map.

[agents-workflows-bot]

✅ Codex Completion Checkpoint

Iteration: 9
Commit: 93cf6e8
Recorded: 2026-05-14T04:25:02.055Z

Tasks Completed

• Add config/source_of_truth_docs.yml with one entry per repo listing the source-of-truth doc paths. For stranske/Workflows, seed it with the ~13 docs CLAUDE.md cites (README.md, docs/INTEGRATION_GUIDE.md, docs/ops/REPO_REVIEW_PROCESS.md, docs/keepalive/GoalsAndPlumbing.md, docs/AGENTS_POLICY.md, docs/LABELS.md, docs/keepalive/Agents.md, docs/ci/WORKFLOWS.md, docs/MODEL_MANAGEMENT.md, docs/WORKFLOW_GUIDE.md, docs/ops/REPO_REVIEW_ROUND2_PROTOCOL.md, docs/ops/REPO_REVIEW_ROUND1_SCHEMA.md, AGENTS.md).
• Write scripts/repo_review_docs_drift_scan.py. For each (repo, doc) in the config: invoke claude with a focused prompt that reads the doc + relevant implementation files (selected by rg queries against terms the doc mentions) and emits a JSON list of drift instances. Each instance has {doc_path, claim, authoritative_source, classification} where classification ∈ {stale, contradictory, accurate-no-drift}.
• In the per-doc prompt, instruct claude: (a) use rg for command/identifier/filename existence checks; (b) call gitnexus context|impact for behavioral claims about call paths when the local GitNexus map is current; (c) skip behavioral checks gracefully when .gitnexus/meta.json reports stale.
• Add coordinator wiring: new step 5b. docs-drift scan after 5. backlog-scan and before 6. notify. Same retry/timeout pattern as backlog-scan; failures are non-fatal.
• Update scripts/repo_review_notify.py: load <output_dir>/docs-drift-scan.json if present; render a "Doc drift detected" section in the desktop reminder, grouped by repo, with one gh issue create snippet per affected repo to file a holistic remediation issue.
• Document the new step in the cron's automation TOML prompt under "Operational notes" with the same depth as the existing backlog-scan note.
• Seed the scanner with the 3 drifts confirmed this cycle (README.md model versions, WORKFLOWS.md autofix contradiction, REPO_REVIEW_PROCESS.md Phase-4 entry point) as integration-test fixtures so the scanner's logic is exercised against real cases.

Acceptance Criteria Met

• python scripts/repo_review_docs_drift_scan.py --registry config/repo_review_registry.json --out /tmp/test-drift.json runs on Workflows-steward and returns at least the 3 known drifts plus a clean (accurate-no-drift) classification for at least 5 other source-of-truth docs.
• pytest tests/scripts/test_repo_review_docs_drift_scan.py exercises the classifier logic against the seeded fixture drifts and verifies they classify as stale / contradictory not accurate-no-drift.
• A full cron dry-run (with --skip-preflight --skip-auto-archive) produces a <output_dir>/docs-drift-scan.json AND a desktop reminder file whose "Doc drift detected" section lists the seeded drifts.
• The desktop reminder's "Doc drift" section gives one gh issue create snippet per repo with non-empty drift, NOT one per doc.
• The cron's automation.toml "Operational notes" section documents the new step, where its output lives, and the expected runtime (~5–10 min across 9 repos).

About this comment

This comment is automatically generated to track task completions.
The Automated Status Summary reads these checkboxes to update PR progress.
Do not edit this comment manually.

[stranske]

Runner dispatch state for autofix on PR #2097. Do not edit.

[github-actions]

Autofix updated these files:

• tests/scripts/test_repo_review_docs_drift_scan.py

[github-actions]

Runner dispatch state for claude on PR #2097. Do not edit.

[github-actions]

Provider Comparison Report

Provider Summary

Provider	Model	Verdict	Confidence	Summary
openai	gpt-5.4	PASS	90%	The merged PR appears to satisfy the documented acceptance criteria. It adds the source-of-truth docs config, extends the docs drift scanner, updates reminder rendering to include a per-repo 'Doc d...
anthropic	claude-sonnet-4-6	CONCERNS	72%	The PR addresses most of the acceptance criteria with meaningful additions to the scanner script, test suite (significantly expanded), notify.py, and source_of_truth_docs.yml. However, two structur...

📋 Full Provider Details (click to expand)

openai

• Model: gpt-5.4
• Verdict: PASS
• Confidence: 90%
• Scores:
  • Correctness: 8.0/10
  • Completeness: 9.0/10
  • Quality: 8.0/10
  • Testing: 8.0/10
  • Risks: 8.0/10
• Summary: The merged PR appears to satisfy the documented acceptance criteria. It adds the source-of-truth docs config, extends the docs drift scanner, updates reminder rendering to include a per-repo 'Doc drift detected' section with bundled gh issue snippets, documents the step in automation TOML, and seeds/uses real drift fixtures. Test coverage was materially expanded in both the scanner and coordinator areas, which supports the dry-run/output integration expectations and seeded stale/contradictory classifications. Quality looks solid overall, with focused changes and supporting fixtures/docs. The main caveat is that a few acceptance details (exact coordinator ordering and exact GitNexus skip logging behavior) are not fully inspectable from the provided diff summary alone, but nothing in the visible changes suggests a functional gap large enough to block acceptance.
• Concerns:
  • The verifier context mentions a new step '5b. docs-drift scan after backlog-scan and before notify', while the diff only shows a minimal coordinator change. Based on the changed tests and automation prompt update, the wiring appears implemented, but the summary does not expose the full coordinator logic, so ordering/non-fatal behavior is inferred rather than fully inspectable here.
  • The acceptance criterion about graceful GitNexus skip behavior is only partially visible from the diff summary; the scanner script was updated and tests expanded, but the exact log text/path behavior cannot be fully confirmed without the full file contents.

anthropic

• Model: claude-sonnet-4-6
• Verdict: CONCERNS
• Confidence: 72%
• Scores:
  • Correctness: 7.0/10
  • Completeness: 6.0/10
  • Quality: 7.0/10
  • Testing: 8.0/10
  • Risks: 8.0/10
• Summary: The PR addresses most of the acceptance criteria with meaningful additions to the scanner script, test suite (significantly expanded), notify.py, and source_of_truth_docs.yml. However, two structural concerns stand out: (1) the coordinator.py change is only +1/-1, which is almost certainly insufficient to implement a full retry/timeout-wired step 5b as required by the AC; (2) the automation.toml documentation addition is only 2 lines, likely falling short of the 'same depth as backlog-scan' requirement. The test suite expansion (+188/-59 in the drift scan tests, +123 new coordinator tests) is a positive signal, but the coordinator test additions may be testing existing behavior rather than the new step wiring. The GitNexus graceful-skip logging requirement (AC5) and the per-repo (not per-doc) grouping requirement (AC4) cannot be fully confirmed from the diff summary alone. Overall, the implementation is partially complete with meaningful gaps in the coordinator wiring and TOML documentation depth.
• Concerns:
  • config/repo_review_automation.toml only adds 2 lines, which likely does not meet the AC requirement to document the new step 'with the same depth as the existing backlog-scan note' including output location and expected runtime (~5-10 min across 9 repos)
  • scripts/repo_review_coordinator.py has only a +1/-1 change, which is insufficient to implement 'new step 5b. docs-drift scan' with the same retry/timeout pattern as backlog-scan; the AC explicitly requires retry/timeout wiring comparable to the existing backlog-scan step
  • AC5 requires verifying graceful GitNexus skip via log output ('skipping behavioral check'); cannot confirm the exact log string is present in the implementation without seeing the full diff of repo_review_docs_drift_scan.py
  • The desktop reminder grouping logic (one gh issue create snippet per repo, not per doc) is implemented in only 21 lines of notify.py changes - correctness of the grouping logic cannot be fully confirmed without seeing the full diff
  • docs/reports/doc-drift-audit/stranske__Workflows-doc-drift.md is a new file not mentioned in acceptance criteria; its presence is benign but suggests scope creep or manual artifact committed to the repo
  • AC1 requires the scanner to return 'at least 5 clean (accurate-no-drift) classifications' in addition to the 3 known drifts; the seeded fixture covers the 3 drifts but it is unclear if the test suite verifies the clean-classification count

Agreement

• Correctness: scores within 1 point (avg 7.5/10, range 7.0-8.0)
• Quality: scores within 1 point (avg 7.5/10, range 7.0-8.0)
• Testing: scores within 1 point (avg 8.0/10, range 8.0-8.0)
• Risks: scores within 1 point (avg 8.0/10, range 8.0-8.0)

Disagreement

Dimension	openai	anthropic
Verdict	PASS	CONCERNS
Completeness	9.0/10	6.0/10

Unique Insights

• openai: The verifier context mentions a new step '5b. docs-drift scan after backlog-scan and before notify', while the diff only shows a minimal coordinator change. Based on the changed tests and automation prompt update, the wiring appears implemented, but the summary does not expose the full coordinator logic, so ordering/non-fatal behavior is inferred rather than fully inspectable here.; The acceptance criterion about graceful GitNexus skip behavior is only partially visible from the diff summary; the scanner script was updated and tests expanded, but the exact log text/path behavior cannot be fully confirmed without the full file contents.
• anthropic: config/repo_review_automation.toml only adds 2 lines, which likely does not meet the AC requirement to document the new step 'with the same depth as the existing backlog-scan note' including output location and expected runtime (~5-10 min across 9 repos); scripts/repo_review_coordinator.py has only a +1/-1 change, which is insufficient to implement 'new step 5b. docs-drift scan' with the same retry/timeout pattern as backlog-scan; the AC explicitly requires retry/timeout wiring comparable to the existing backlog-scan step; AC5 requires verifying graceful GitNexus skip via log output ('skipping behavioral check'); cannot confirm the exact log string is present in the implementation without seeing the full diff of repo_review_docs_drift_scan.py; The desktop reminder grouping logic (one gh issue create snippet per repo, not per doc) is implemented in only 21 lines of notify.py changes - correctness of the grouping logic cannot be fully confirmed without seeing the full diff; docs/reports/doc-drift-audit/stranske__Workflows-doc-drift.md is a new file not mentioned in acceptance criteria; its presence is benign but suggests scope creep or manual artifact committed to the repo; AC1 requires the scanner to return 'at least 5 clean (accurate-no-drift) classifications' in addition to the 3 known drifts; the seeded fixture covers the 3 drifts but it is unclear if the test suite verifies the clean-classification count

🔍 LangSmith Traces

• openai
• anthropic