QA and Audit Guide

QA and Audit Guide

How to run the auditor, what its sixteen check families actually verify, how to interpret and fix each error class, and how to tune ban.json / terms.json without training yourself to ignore the output. Includes the self-test discipline that exists because three successive checker bugs produced hundreds of phantom errors in real runs. Verified against qa/qa_wiki.py at commit 7e17a11; all behaviours reproduced in this wiki run (T1, and the live audit of this very wiki).

Contents

• The two-command discipline
• What the audit checks
• The scrubbed view (why correct prose stopped failing)
• Fixing each error class
• Reading the report
• The step after zero errors: re-run what you documented
• See also

---

The two-command discipline

# once per session — proves the checker can fail
python3 qa/qa_wiki.py --self-test

# the real audit — iterate to zero errors
python3 qa/qa_wiki.py workspace/wiki/OWNER.REPO.wiki \
    --terms workspace/terms.json --ban workspace/ban.json --strict

Important

Run --self-test first, every session. The checker's own history is the reason: version one reported 381 link errors — all false (it compared Page.md against stems without the extension). Version two reported 12 anchor errors — 11 false (the slugger dropped backticks). A third bug audited every page against the previous page's text and produced 235 phantom errors. A checker that cannot fail is worse than no checker, because a clean run then means nothing. The current self-test feeds three deliberately broken fixtures and requires 22 specific detections (verified 22/22, T1).

The mirror rule (PITFALLS D2): when the auditor reports a mass failure, verify one case by hand before believing it; when it reports zero failures, verify it can fail (that is what --self-test is for).

What the audit checks

Sixteen families, grouped by the defect class each exists to catch:

Family	Checks	Error or warning
Links	[[Page]] targets exist; [[Page#anchor]] anchors exist; relative .md links resolve; dead link in _Sidebar/_Footer	error (with a hint when the hyphenated filename exists: "name files with SPACES")
In-page anchors	every ](#…) matches a heading under GitHub's slug algorithm	error
Coverage	_Sidebar.md links every page; _Footer.md exists (absence warns)	error / warning
Structure	exactly one H1; lead paragraph of ≥40 prose characters right after the H1; a ## See also section	error (lead = warning)
Alerts	every > [!X] uses one of NOTE, TIP, IMPORTANT, WARNING, CAUTION	error
Fences	nesting: a 3-backtick block containing a 3-backtick opener with an info string is the early-close bug; unterminated fences	error
Prose leaks	[[...]] outside code — GitHub turns it into a link to a non-existent page	error
Mermaid	diagram type recognised (16 kinds); brackets/quotes balanced (sequence diagrams exempt from bracket balance); quoted labels stripped before the bracket check; cylinder [(...)] shape exempt; bare & in edge labels; unquoted labels containing special characters; kind-aware subgraph/end vs alt/opt/loop/par/end balancing	error (& inside quoted labels = warning)
Shell	every bash/sh/shell/console/zsh block passes bash -n (text passed on stdin, never executed; pure-output blocks skipped)	error
Python	every python block passes compile() (REPL transcripts skipped)	error
Filler	20 marketing-filler regexes (matching game-chang…, seamless, cutting-edge, empower, delve, blazing-fast, "in today's fast-paced", …) on the scrubbed prose view	error (exemptable)
Emoji	any emoji outside code blocks	warning — confirm each is data (a documented character mapping or a verbatim UI glyph), not decoration
Banned claims	--ban phrases, case-insensitive substring on the scrubbed view	error, with negation downgrade to warning and exemptions to info
Terminology	--terms avoid patterns on the scrubbed view; canonical regexes counted	error (avoid) / stats (canonical)
Stats	per-page words, H1 count, anchors, alerts, mermaid, code blocks, table rows; totals; terminology frequencies	report
JSON out	--json FILE writes errors, warnings, infos and the full stats structure	artefact

Exit codes: 0 clean or non-strict · 1 with --strict and at least one error · 2 usage error or failed self-test. Warnings never fail --strict — errors are contract, warnings are judgement calls.

The scrubbed view (why correct prose stopped failing)

Ban, avoid and filler matching does not run on raw text. Each page is reduced to a scrubbed view first:

1. fenced-code interiors blanked (fence-length aware),
2. inline code spans blanked,
3. [[wiki-links]] blanked,
4. ](#anchor) targets blanked,
5. filenames matching \S+\.(py|md|php|js|ts|json|sh|css|ya?ml|toml|ini|tmpl|ps1|…) blanked.

Blanking (not deleting) preserves character offsets, so the reported context excerpt stays accurate. This removed an entire false-positive class documented in PITFALLS C8: a GitHub anchor slug (#mdscribe-extensions — the slugger drops the dot), a probe filename, and quoted code firing terminology rules.

On top of the scrubbed view:

• Negation downgrade — a negating or hedging word (no, not, never, without, lacks, false, myth, claims, asserts, allegedly, reportedly, "the docs say", …) within ~90 characters before the hit, in the same sentence, turns a ban ERROR into a WARNING. A sentence boundary resets the scope, so an earlier clause cannot hedge a later assertion. It is never a suppression: "There is no GPG verification" must stay visible to a human.
• Exemptions — {phrase, file?, quote?, reason} objects in ban.json/terms.json; a matched exemption reports as INFO, never silently dropped, so the list cannot become a rubbish drawer.

Fixing each error class

Error message shape	Fix
[[X]] → no such page (did you mean the file "X.md"? name files with SPACES)	rename the file to spaces, or fix the link text; never create hyphenated filenames to match
[[Page#frag]] → anchor not found	run qa/fix_toc.py and re-copy the computed slug; never hand-write anchors
in-page link #x → no such heading	the heading changed; rebuild Contents with fix_toc.py
_Sidebar.md does not link every page	add the missing [[Page]] entries (the scaffold does this at init; manual page additions must update the sidebar)
expected exactly one H1, found N	demote duplicates to ##
missing a "## See also" section	add one with cross-links — it is the navigation contract of every page
invalid GitHub Alert type [!X]	use one of the five; a made-up type renders as a plain blockquote with visible [!X] text
fence nesting: … closed early …	wrap the outer example in four backticks (PITFALLS C7 — this bug shipped once and rendered the rest of a page as garbage)
mermaid#N: unbalanced [] / odd number of double quotes	fix the diagram; labels with special characters go in double quotes
mermaid#N: bare '&' in edge label	write & — GitHub's mermaid renderer requires the entity
bash block #N fails bash -n	fix the snippet; if the block is a transcript or output, re-tag it text or console-output style so the checker skips it honestly
python block #N does not compile	fix the snippet; REPL transcripts with >>> are skipped automatically
marketing filler 'x'	cut the phrase; if it is domain terminology (e.g. "unlock a hard lock", "PK magic bytes"), record a fluff_exemptions entry with the reason
banned claim 'x' — reason	the claim is false or unverifiable: delete or rewrite the sentence; if the hit is a refutation, paraphrase instead of quoting (C8); if legitimate, add an exemption and expect INFO
avoided term 'x'	use the canonical term; if a verbatim subject string or filename collides with the pattern, the subject wins — narrow the pattern and log it in _tuning_log

A real tuning case from this wiki's own run: the avoid pattern for whitelist fired on prose describing the auditor's closed set of alert types — a different concept from ban/terms exemption lists. Per doctrine, the prose was paraphrased, the pattern kept (narrow and correct for its concept), and the decision logged in terms.json's _tuning_log.

Reading the report

file                       words  H1  anch  alert  mmd  code  tbl
Security Considerations.md  1994   1    13      3    0     0   44
...
TOTAL  pages=22 (20 content)  words=12,495  alerts=7  mermaid=4  tables=308
       code=24  wiki-links=155
checked: 14 shell blocks, 2 python blocks
terminology: run×73, subject×70, the kit×34, battery×33, evidence×33, ...

ERRORS (0):
WARNINGS (0):
EXEMPTED (N) — recorded, not enforced:

The per-page table is your progress dashboard (copy the numbers into PROGRESS.json's qa block at the phase boundary). The terminology line is your consistency check: a canonical term used 90 times next to a synonym used twice is a pattern to fix in prose, not in the checker. --json gives the same data to scripts.

The step after zero errors: re-run what you documented

The audit proves the wiki is well-formed; it cannot prove the wiki is true. Truth comes from Phase 4 step 3: re-run every command the wiki documents, in a clean sandbox, from a fresh clone — qa/verify_commands.py is the worked template (fresh clone → documented server start → documented curls and recipes → evidence JSON). In the reference run this step caught two factual errors that no structural checker could see: a 403 page's <title> differing from its <h1>, and a README-transcribed claim about a force-flag propagating between update phases that the code did not implement.

Budget honestly: the checker is fast; fixing what it finds is not. The reference run spent ~12% of total effort on QA and delivery combined.

---

See also

• Tool Reference — qa_wiki.py and fix_toc.py flags, internals and exit codes
• Configuration Reference — ban.json / terms.json schemas in full
• Writing Pages Guide — writing so the audit stays quiet
• Troubleshooting — auditor errors you cannot explain
• Publishing and Delivery — the QA gate that blocks publication