Skip to content

docs-check: --coverage gate has false-positives + soft-warn pattern broken #248

@avrabe

Description

@avrabe

Summary

12-persona dogfood pass against rivet 0.7.0 found two issues with the new rivet docs check --coverage gate (#241):

B5 — Coverage rule 3 (parent-mapping) too generous

The gate has 4 priority rules to mark a subcommand path "covered":

  1. Exact slug match
  2. Parent-walk to next-shorter path
  3. Manual umbrella mapping (COVERAGE_TOPIC_MAP)
  4. Allow-list

Rule 3 (parent mapping) is too loose: it marks batch, query, stamp, lsp as "covered" via cli — but a grep of the cli topic body shows none of those four subcommands have a usage block. The gate is reporting "59% covered" but the real number is lower because rule 3 silently passes children whose parent's topic doesn't actually mention them.

Proposed fix: rule 3 should require the child subcommand's name to appear in the parent topic's body. Even better: require a minimum (one usage block + one example) per subcommand path.

B6 — --coverage without --strict always exits 0 → broken soft-warn pattern

rivet docs check --coverage (without --strict) lists uncovered paths but exits 0. This makes the "warn-only-then-promote-to-strict" rollout pattern impossible inside one CI matrix without external glue. A DevOps engineer wanting:

  • "Warn on uncovered now"
  • "Fail on uncovered after 2 weeks"
  • "Land both as workflow files; flip a repo variable to switch"

…has to wrap the command in shell to capture the report and grep -q '✗' themselves, OR use two separate workflow files.

Proposed fix: make the warn-vs-strict semantics explicit. Three modes:

  • --coverage → just print the report, exit 0
  • --coverage --warn-only → print + emit ::warning:: GitHub annotations on uncovered, exit 0
  • --coverage --strict → print, exit 1 if anything's uncovered

Trigger

Post-0.7.0 12-persona dogfood. Tech writer (B5) + DevOps (B6) independently flagged.

Acceptance

  • Rule 3 tightened: require child name to appear in parent's body
  • --warn-only flag added; emits GitHub annotations for CI
  • Default --coverage is silent-print (exit 0)
  • Test fixtures cover all three modes
  • rivet docs docs-coverage topic updated to document the three modes

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions