Detector: Docs Drift

Detector: Docs Drift

Detects stale references, dependency drift, and optionally doc-code semantic drift in documentation.

Property	Value
Name	docs-drift
Tier	LLM_ASSISTED
Languages	All (scans markdown documentation)
External tool	None
LLM required	Optional (for doc-code drift sub-check)
Confidence	0.80–0.95 (deterministic), 0.65 (LLM)

What it detects

Three categories of documentation issues:

1. Stale references (deterministic)

Markdown links and inline backtick paths that point to files or anchors that no longer exist.

• Resolves paths relative to the document and repo root
• Suffix matching against all repo files
• Skips external URLs, code blocks, strikethrough text, template patterns

2. Dependency drift (deterministic)

pip install / npm install commands in documentation that reference packages not in the project's actual dependency list.

3. Doc-code drift (LLM, optional)

Compares documentation code blocks (Python imports, CLI commands) against the actual source code to detect semantic inconsistencies. Requires a model provider.

Severity

Sub-check	Severity	Confidence
Stale markdown link	MEDIUM	0.95
Stale inline path	LOW–MEDIUM	0.80
Dependency drift	MEDIUM	0.90
Doc-code drift (LLM)	From LLM	0.65

Example finding

[DOCS-DRIFT] docs/contributing.md:45 — Stale reference
  Link target: #asgi-transport → anchor not found in document
  Severity: MEDIUM, Confidence: 0.95

Observed accuracy

Repo	Findings	TP Rate	Notes
pip-tools	2	~0%	2 FP (edge-case inline paths)
httpx	1	100%	Real stale anchor
shadcn-ui/ui	8	~25%	2 TP, 6 FP (user-project paths)

FP-reduction features

• Template path detection (path/to/your/example)
• Example context detection ("e.g.", "for example", "such as")
• Fenced code block tracking (skip paths inside code blocks)
• Shell comment stripping in pip install commands
• Non-path npm import filtering

Known limitations

• Inline path resolution can produce FPs for template-like patterns in component libraries
• CHANGELOG feature descriptions without explicit "example" phrases may false-positive
• Doc-code drift sub-check not yet validated at scale (requires LLM)