feat: docs-check gate (recurring doc-vs-reality CI check)#178
Merged
Conversation
…tion
Introduces rivet-core/src/doc_check.rs: an invariant engine that scans
README.md, CHANGELOG.md, AGENTS.md/CLAUDE.md and every *.md under docs/
to catch doc-code drift.
Ships 8 invariants (MVP):
- SubcommandReferences — "rivet <word>" must be a real subcommand
- EmbedTokenReferences — "{{name:...}}" must be a registered embed
- VersionConsistency — workspace version == vscode/npm package.json
and > every version string mentioned in prose
- ArtifactCounts — "N <noun>" needs {{stats}} or AUDIT marker
- SchemaReferences — schemas/foo.yaml must exist
- SoftGateHonesty — "enforced"/"wired into CI" claims must not match
a job carrying continue-on-error: true
- ConfigExampleFreshness — fenced yaml/yml blocks must parse
- ArtifactIdValidity — REQ-NNN style IDs must resolve in the store
Opt-out for roadmap docs:
<!-- rivet-docs-check: design-doc-aspirational-ok -->
Files under docs/plans/ or docs/design/ are auto-detected as design docs.
21 unit tests in-module covering fixture-doc-flagged and fixture-doc-
clean paths for each invariant plus line/code-fence helpers.
Refs: REQ-054
Wires the doc_check engine into the CLI as a `check` subtopic of the existing `docs` command. Usage: rivet docs check # text output, exit 1 on failure rivet docs check --format json rivet docs check --fix # apply auto-fixes (version numbers) The handler pulls the known-subcommand list from clap::CommandFactory at runtime so it stays in sync with the actual CLI. Known embed kinds are currently hard-coded to match rivet-core/src/embed.rs. Loads the project store when available; invariants that need it (like ArtifactIdValidity) silently skip when the project fails to load so the check can still run against docs-only repos. Also reads .github/workflows/ci.yml (when present) to drive the SoftGateHonesty invariant. JSON output is machine-readable with `status`, `violation_count`, `by_invariant`, and a `violations[]` array, ready for CI annotations. Refs: REQ-054
Adds a new 'docs-check' job in both .github/workflows/ci.yml and .github/workflows/release.yml that runs 'rivet docs check' and fails the build if any invariant is violated. The release workflow gates 'create-release' on this job so a tag cannot produce a GitHub Release while documentation claims are stale. Deliberately uses neither continue-on-error nor an allow-list: doc drift should fail the build the same way clippy errors do. Budget is <1 minute on a warm rust-cache. Trace: skip
Observations running the gate against the current tree:
- "rivet never touches" / "rivet models itself" matched SubcommandReferences.
Fix: require the match to be inside inline backticks or after a shell-
prompt marker ("$ ").
- Document front-matter IDs (AUDIT-001, SRS-001, ROAD-001, …) tripped
ArtifactIdValidity. Fix: collect front-matter IDs up front and exclude
them from body checks.
- Anchor-style hex hashes (YAML-654FF0, STPA-654FF0) tripped the same
invariant. Fix: recognize the hex-hash suffix pattern.
- DO-178C / UTF-8 / NOPE-999 were flagged. Fix: extend non-artifact
prefix skip-list.
- Design docs under docs/plans/ mention future rivet versions (v0.5.0,
v1.0.0) as part of roadmaps. Fix: skip VersionConsistency entirely
for design docs, plus skip versions inside inline backticks (third-
party version pins like `kani-version: '0.50.0'`).
- CHANGELOG and audit-report.md are large historical snapshots; tagging
every count line with AUDIT was noisy. Fix: add a file-level
`<!-- AUDIT-FILE: verified YYYY-MM-DD -->` marker that suppresses
ArtifactCounts for the entire document.
Also recognize `rivet import` unconditionally (it exists behind the
`wasm` feature gate but the CI build doesn't enable it) so that docs
describing the full surface don't break the release gate.
Three new unit tests: subcommand_references_skip_plain_prose,
artifact_id_validity_ignores_frontmatter_ids,
artifact_id_validity_skips_hex_anchor_hashes. All 24 tests pass.
Refs: REQ-054
Fixes the violations that `rivet docs check` reports against main: - vscode-rivet/package.json: bumped 0.3.0 -> 0.4.0 to match the workspace. Without this the extension version lags every CLI release (this has happened every release so far). - README.md: the commands table listed `rivet import` for "ReqIF, sphinx-needs JSON". The real command for that path is `rivet import-results`; `rivet import` is the WASM adapter entry point. Also updated the dogfood line from the stale "447 artifacts" to the actual current count (709) with an AUDIT marker pointing future readers at `rivet stats`. - CHANGELOG.md: added a file-level AUDIT-FILE marker. Every count in the changelog is a release-time snapshot and should not be required to track current state. - AGENTS.md: added an AUDIT marker after the retroactive traceability table (a frozen historical record of the v0.0.x -> v0.3 commit map). - docs/audit-report.md: added a file-level AUDIT-FILE marker for the same reason as CHANGELOG. Result: `rivet docs check` passes (31 files, 0 violations). Refs: REQ-054
avrabe
force-pushed
the
feat/v041-docs-check-gate
branch
from
a55228e to
49b22f4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a recurring
rivet docs checkgate that asserts documentationclaims match reality before every PR merge and every release tag. This
complements the one-time doc-reality audit — we catch future drift the
same way clippy catches lint errors.
rivet-core/src/doc_check.rswith 8 invariants(SubcommandReferences, EmbedTokenReferences, VersionConsistency,
ArtifactCounts, SchemaReferences, SoftGateHonesty,
ConfigExampleFreshness, ArtifactIdValidity). 24 unit tests cover
flagged/clean paths per invariant plus false-positive heuristics.
rivet docs check [--fix] [--format json]that prints
file:line [invariant] claim -- realityfor eachviolation and exits non-zero if any fire.
--fixauto-correctsversion numbers in ancillary
package.jsonfiles.docs-checkjob in.github/workflows/ci.ymlwithNO
continue-on-error. Release workflow gatescreate-releaseon the same job.
gate finds on main today — vscode-rivet 0.3.0 -> 0.4.0, README.md
stale command table and artifact count, AGENTS.md/CHANGELOG.md/
audit-report.md file-level AUDIT markers for historical snapshots.
Before/after counts on main:
Design-doc opt-out
Roadmap / planning docs can opt out of existence-based invariants
(SubcommandReferences, EmbedTokenReferences, ArtifactIdValidity,
VersionConsistency) with either:
<!-- rivet-docs-check: design-doc-aspirational-ok -->docs/plans/ordocs/design/(auto-detected)They remain subject to SchemaReferences, SoftGateHonesty, and
ConfigExampleFreshness, which are always true-false regardless of
aspiration.
Release-time gate
A release tag that contains stale doc claims cannot ship.
Test plan
cargo test -p rivet-core --lib doc_check— 24 tests passrivet docs checkon current main — PASS (31 files, 0 violations)rivet docs check --format json— machine-readable outputdocs-checkjob greencreate-releasewhen docs-check failsTrailer
Refs: REQ-054