Tool Reference

Tool Reference

Complete reference for the fourteen tools of the kit at commit 7e17a11: what each does, every command-line flag, exit codes, self-tests, and the internals that matter when something breaks. Line numbers refer to the files in the subject repository. Every behaviour statement was verified by execution during this wiki run (test IDs T0–T18 map to logs in the run's research/evidence/ directory).

Contents

• Overview
• setup/bootstrap.sh
• setup/check_runtime.py
• setup/snapshot.sh
• scan/repo_scan.py
• lib/req.py
• lib/serve.py
• lib/session.py
• lib/evidence.py
• probe/webapp_probe.py
• scaffold/new_page.py
• qa/qa_wiki.py
• qa/fix_toc.py
• qa/verify_commands.py
• publish/publish.sh
• See also

---

Overview

Tool	Phase	Lines	Role	Self-test	Verified this run
setup/bootstrap.sh	1	155	Environment + capability/egress report	—	T0
setup/check_runtime.py	1	144	Runtime-floor gate (exit 2 on mismatch)	—	T12
setup/snapshot.sh	all	10	Git-snapshot the workspace	—	T13
scan/repo_scan.py	1	433	Whole-repo inventory → JSON + brief	—	T0b
lib/req.py	1	235	Stdlib HTTP client; status/headers/body as data	CLI	T11, T18
lib/serve.py	1	325	ServedApp: staged instance on a verified-free port	—	T10, T17
lib/session.py	1	259	Cookie-jar client + CSRF extraction	live 8/8	T9
lib/evidence.py	1, 4	134	Fallback-safe evidence writes	3/3	T3
probe/webapp_probe.py	1	457	Declarative batteries → evidence	—	T14, T16
scaffold/new_page.py	3	353+	QA-clean wiki skeleton or single page	via CI	T8
qa/qa_wiki.py	4	802	The audit (16 check families)	22 detections	T1
qa/fix_toc.py	3	211	Block-scoped TOC rebuilder	8/8	T2
qa/verify_commands.py	4	203	Phase-4 command harness template	—	read-only (T-shape)
publish/publish.sh	4	231	QA gate → clone → commit → bundle → hand-over	—	T15

Supporting documents (not executable): setup/PITFALLS.md (31 traps), SKILL.md, AGENTS.md, PROMPT.md, PUBLISHING.md, templates/* (five files), .github/workflows/selftest.yml, examples/ (two reference runs).

setup/bootstrap.sh

Idempotent environment setup. Installs, in order: apt-get update (the step everyone forgets — a fresh sandbox has empty package lists), then git, curl, jq, python3, unzip, zip, ripgrep, shellcheck, then php-cli php-mbstring php-curl. Prints a capability report (versions of 10 tools + 8 Python stdlib modules) and a network-egress check against github.com, raw.githubusercontent.com, api.github.com.

bash setup/bootstrap.sh                    # base environment
bash setup/bootstrap.sh --with-node        # + nodejs/npm (mermaid-cli, JS subjects)
bash setup/bootstrap.sh --with-php83       # prints the Sury recipe when PHP < 8.3
bash setup/bootstrap.sh --with-composer    # installs composer via getcomposer.org
bash setup/bootstrap.sh --subject repo/SUBJECT   # + runtime-floor gate; exits non-zero on mismatch

Flag	Effect
--with-node	installs nodejs + npm
--with-php83	if installed PHP < 8.3, prints the Sury APT recipe (PITFALLS B10) and the honest-alternative note ("verified on PHP 8.2.34, documented target 8.3+" is a useful statement, not a defect)
--with-composer	downloads and installs composer to /usr/local/bin
--subject DIR	runs setup/check_runtime.py DIR and exits non-zero on mismatch — "probing a subject the runtime cannot boot produces silently wrong findings"
-h, --help	prints the header docblock

Measured (T0, Debian-class sandbox, root): installed PHP 8.2.34 with curl/json/mbstring/openssl modules, curl 7.88.1, jq 1.6, rg 13.0.0; egress github.com → 200, api.github.com → 200, raw.githubusercontent.com → 301. Runtime ~58 s dominated by apt.

setup/check_runtime.py

Compares the subject's declared runtime floor with the installed runtime. Floor sources, in order: composer.json → require.php, package.json → engines.node, go.mod → go X.Y. Also reports dependency state (vendor/autoload.php, node_modules) because composer install without unzip prints a usage dump that reads like a wrong-flag error (PITFALLS B9).

python3 setup/check_runtime.py repo/SUBJECT          # human report
python3 setup/check_runtime.py repo/SUBJECT --quiet  # exit code only

Exit codes: 0 fine or nothing declared · 2 mismatch (prints a boxed banner: "It will NOT BOOT, so every probe would measure a corpse") · 1 bad argument. Verified (T12): no-manifest subject → 0; stub declaring php: >=8.3 under PHP 8.2.34 → 2 + banner; stub >=8.1 → 0. This reproduces the exact behaviour the CHANGELOG claims for 1.2.0.

setup/snapshot.sh

Ten lines that make accidents cheap: git init -b snapshot (if needed), git add -A, commit as wiki-agent, print the last commit. Run at every phase boundary and before any bulk edit.

bash setup/snapshot.sh workspace "phase 1 done"

Verified (T13): two successive snapshots produced two commits on branch snapshot; a no-change run prints "nothing to snapshot" and exits 0.

scan/repo_scan.py

Phase 1 in one command: walks the tree (skipping 30 VCS/build/cache directory names, files over 4 MB, binary extensions) and extracts:

Extract	Detail
git	HEAD, branch, commit count, first/last dates, remote, tag count (with an explicit "NO TAGS: versioning is by file marker" note at zero), commit-kind classification (fix/feat/other)
license	file, type (9 recognisable licences), copyright line, extra attribution clauses
dependencies	17 manifest names across 8 ecosystems — or the notable absence ("NO MANIFEST — zero declared dependencies")
tree	every file (size, lines, sha, extension), grouped counts, 25 largest files ("where the logic lives")
entry points	conventional names (index.*, main.*, app.*, …), script dirs (bin/, scripts/, cmd/, public/, www/, htdocs/), and any server-side script at the repo root
symbols	functions/methods/classes with line numbers for PHP, JS, TS, Python, Go, Ruby, Java, C#, PowerShell, shell (SYMBOL_PATTERNS, scan/repo_scan.py:71)
config surface	define(), const, env vars (3 idioms), PHP superglobals, INI-style keys (CONFIG_PATTERNS, line 105)
network	every external URL grouped by host — the supply-chain/privacy fact source
security signals	11 regex classes: output-escaping, raw-html-passthrough, path-validation, exec-shell, sql, crypto, tls-options, auth, security-headers, file-upload, signature-verify (line 122)
versions	Version: X.Y.Z-style markers anywhere in the tree

python3 scan/repo_scan.py repo/SUBJECT --brief -o research/scan.json   # brief + JSON
python3 scan/repo_scan.py repo/SUBJECT -o research/scan.json           # JSON only
python3 scan/repo_scan.py repo/SUBJECT --max-files 10000 --brief       # big repos

Flags: positional root; -o/--out FILE; --brief (human two-minute summary; also implied when -o is absent); --max-files N (default 4000). Exit 2 when root is not a directory. The --brief output on the subject itself (T0b) correctly reported: 8 commits in one day, no tags, MIT, no manifest, 44 files / 5,523 code lines, and the entry point setup/bootstrap.sh.

lib/req.py

Dependency-free HTTP client. Design contract: a 4xx/5xx is data, not an exception — HTTPError is caught and returned, redirects are NOT followed by default (a 302 is part of the contract you are documenting), and transport failures return the status string "ERR" with the message as the body. Response is a (status, headers, body) tuple with named accessors (.text, .json(), .header(name) case-insensitive, .ok).

python3 lib/req.py GET  http://127.0.0.1:8080/md.php
python3 lib/req.py GET  http://127.0.0.1:8080/page.html -o page.html
python3 lib/req.py POST http://127.0.0.1:8080/md.php -d md_preview=1 -d 'content=# Hi'
python3 lib/req.py HEAD http://127.0.0.1:8080/md.php --headers-only
python3 lib/req.py GET  http://127.0.0.1:8080/x.php -H 'X-API-Key: mdv_abc'

Helpers: get, post_form (urlencoded), post_multipart (manual boundary builder; files={field: (filename, bytes[, ctype])}). The docstring carries two hard-won warnings: never name a local file http.py (it shadows the stdlib and every urllib call dies with a misleading error — PITFALLS A2), and decode bodies with errors="replace" because hostile payloads can produce invalid UTF-8. Verified (T11/T18): 200 with headers, 404 returned as data, "ERR" on a dead port, CLI --headers-only output.

lib/serve.py

ServedApp — the reason PITFALLS A1 exists. A context manager that stages a clean copy of the subject in a temp dir (probe-app-*), picks a verified-free port, starts the server in its own process group, waits for the socket, fires one HTTP request to prove the interpreter is not dying on every request, and on exit kills the process group and deletes the staging dir (unless keep=True).

import sys; sys.path.insert(0, 'toolkit/lib')
from serve import ServedApp

with ServedApp('repo/SUBJECT', copy=['md.php', 'css', 'js']) as app:
    app.write('md.md', '# Hello\n')          # stage a fixture
    app.set_ini({'DISABLE_UPLOAD': 'false'}) # flip a config flag (.md.ini merge)
    r = app.get('/md.php')
    print(r.status, r.header('Content-Type'))
    print(app.ls())                           # what did the app write to disk?

Constructor parameters: source, copy (names to stage; None = everything except VCS dirs), serve_cmd (template with {host} {port} {root}; default php -S {host}:{port} -t {root}), port (None/0 = auto; a busy preferred port raises rather than silently drifting — lib/serve.py:58), host (default 127.0.0.1), keep, quiet. Methods: write/mkdir/set_ini/ls/read (fixtures), get/post/upload/request (via req.py), base_url/url, server_log(tail).

CLI mode runs in the foreground (Ctrl-C to stop) and exits 1 with a log dump if the server dies:

python3 lib/serve.py repo/SUBJECT --serve-cmd 'python3 -m http.server {port} -d {root}' --check /index.html

Verified (T10/T17): non-PHP serve_cmd staging, ready in 0.15 s, GET 200 with body marker, clean teardown; CLI foreground behaviour as documented.

lib/session.py

SessionClient for stateful surfaces: cookie jar (because dict(resp.headers) collapses multiple Set-Cookie headers — the trap that forces a real jar), no auto-redirect (a login 302 is contract), and CSRF plumbing that puts the token in the body field and a header, never in the URL (the MD.Scribe subject rejected query-string tokens outright). csrf_from_html(html, extra_patterns=()) extracts tokens across six printed shapes (JSON "csrf":"…", JS literal csrf: "…", _token: "…", hidden inputs, and friends).

from session import SessionClient, csrf_from_html
c = SessionClient(app.base_url)
home = c.get('/')
tok = csrf_from_html(home.text)
r = c.post_form('/login', {'user': 'admin', 'pass': 'x'}, token=tok)
assert r.status == 302          # observable redirect, not followed
r = c.post_form('/admin/action', {'do': 'x'}, token=c.csrf or tok)

--self-test spins up a 12-line PHP session app and exercises the whole lifecycle. Verified (T9) 8/8 on PHP 8.2.34: token extracted · action-before-login 401 · login-without-token 403 · wrong password 401 · login 302 observable · authenticated action ok · session survived regeneration · token stable within session.

lib/evidence.py

One function, one job: write_evidence(dest, payload) must not lose completed probe results to a filesystem hiccup. Attempts, in order: the absolute dest, cwd/dest (when relative), then <kit>/workspace-fallback/<name>; dict/list → JSON, bytes → raw, anything else → str. It never raises for a write failure — it returns the path actually used, or None plus a loud diagnosis on stderr ("Do NOT let this be the only copy"). diagnose(dest) renders a one-line why-unwritable report for logs. Exists because of PITFALLS B11 (a field run lost its final evidence file while parent.is_dir() printed False in-process and the shell wrote the same directory fine a second later).

--self-test verified (T3) 3/3: nested JSON write, text write, fallback on an unwritable destination. The unwritable fixture is a path whose parent is a regular file — the 1.2.1 fix after /proc/definitely-not-writable turned out to be creatable on Windows (C:\proc\…), making the self-test fail on every Windows machine while CI stayed green.

probe/webapp_probe.py

The battery runner. Two mutually exclusive targets: --repo DIR (stage a fresh ServedApp copy — recommended) or --base-url URL (probe something already running). Then --entry PATH, --suite NAME, --copy (staging list), --serve-cmd, -o JSON, --md MARKDOWN, --keep.

Batteries (methods on Probe): battery_headers (10 security headers present/MISSING, 4 leak headers, cookies) · battery_traversal (20 payloads through {PAYLOAD} templates; body-marker denial classification; leak markers) · battery_injection (26 payloads through post/get/raw sinks; verdicts) · battery_flags (flip INI keys via the staged app, re-request, compare) · battery_disk (what the app wrote). Every probe.run(...) records group, name, method, path, status, ms, bytes, all headers, a 180-char excerpt, and matches_expectation against expect=.

Registered suites: generic (11 reachability paths incl. /.git/config, /.env; 5 HTTP methods; header battery) and php-mdviewer (modes incl. viewer/browser/download/clipboard-preview, traversal, injection, flag gating, machine-API with key-shape check mdv_[0-9a-f]{48}, 16-filename upload matrix, wrong-field-name and dir matrices, disk side-effects). A suite is just a function (Probe, entry) -> dict added to SUITES — see Extending the Kit.

Verified (T14): the full php-mdviewer battery against MD.Viewer 2.18.0 completed in 0.46 s wall time; traversal 17/20 denied with 0 leaks; injection 19/26 REFLECTED-VERBATIM; two expectation mismatches flagged with the tool's verbatim ❌ marker and reported as drift findings (the tool doing its job). Verified (T16): generic suite, 16 requests, including the observation that md.php answers 200 to GET/POST/PUT/DELETE/PATCH/OPTIONS under php -S.

Warning

Batteries fire attack payloads. Point them only at staged local copies of software you are authorized to test. See Security Considerations.

scaffold/new_page.py

Creates wiki skeletons that pass qa_wiki.py --strict unfilled (QA-clean by construction, verified across three page lists including the degenerate single-page case — T8). Two modes:

# whole skeleton: Home.md, _Sidebar.md, _Footer.md + one stub per page
python3 scaffold/new_page.py wiki/OWNER.REPO.wiki --init \
    --project "SUBJECT" --repo-url https://github.com/OWNER/REPO \
    --version 1.2.1 --commit abc1234 --license MIT \
    --progress workspace/PROGRESS.json

# single page
python3 scaffold/new_page.py wiki/OWNER.REPO.wiki "Security Hardening Guide" \
    --sections "Threat model,Findings,Hardening" --pages "Home,FAQ"

Flag	Effect
--init	full skeleton instead of one page
--pages "A,B,C"	page list (Home auto-prepended)
--project, --repo-url, --version, --license, --commit	footer/badge/facts metadata
--progress FILE	take page tree and pages[].group from PROGRESS.json (preferred)
--groups "Label:A,B|Label2:C"	explicit sidebar grouping
--sections "X,Y"	H2 stubs for single-page mode
--force	overwrite existing files (default: skip with a note)

Grouping precedence: --progress > --groups > keyword heuristic (install/usage/operating/overview/reference). Pages named in a grouping but missing from the list are created anyway (sidebar consistency). _Footer.md links are derived from the actual page list — never hardcoded (the D7 lesson: one hardcoded dead link in the footer is a site-wide error because the footer renders on every page).

Note

Found and fixed during this wiki run: the explicit-grouping branch (--progress/--groups) did not write _Footer.md at all — only the heuristic branch did — and CI could not catch it because the scaffold round-trip uses the heuristic path and a missing footer is a warning, not an error. The fix (footer written from the derived link list in both branches) is recorded in the run's REPORT.md as a maintenance contribution, per the README's "commit the improvement back" doctrine.

qa/qa_wiki.py

The audit. Sixteen check families in one command:

python3 qa/qa_wiki.py --self-test                     # once per session, non-negotiable
python3 qa/qa_wiki.py wiki/OWNER.REPO.wiki --strict \
    --terms workspace/terms.json --ban workspace/ban.json --json qa.json

Flag	Effect
wiki_dir	directory of .md pages (positional)
--strict	exit 1 on any error (warnings do not fail)
--json FILE	full result as JSON (errors, warnings, infos, stats incl. per-page table)
--terms FILE	canonical-term counts + banned-synonym errors
--ban FILE	banned-claim errors with reasons, exemptions, negation downgrade
--quiet	suppress the report tables
--self-test	run the three-fixture detection suite

Checks: wiki-link targets and #anchors (with a "did you mean the file with hyphens? name files with SPACES" hint), in-page anchors via GitHub's slug algorithm (github_slug, line 173 — never hand-write anchors), _Sidebar/_Footer coverage (sidebar must link every page; dead links in either are errors), one H1 per page, lead paragraph (≥40 chars of prose), ## See also presence, the closed set of five alert types, fence nesting (detects the early-close bug where a 3-backtick example inside a 3-backtick block swallows the rest of the page), [[...]] leaking into prose, mermaid (16 recognised diagram types, bracket/quote balance with quoted-label and cylinder exemptions, bare & in edge labels = error, kind-aware subgraph/end vs alt/opt/loop/end), bash -n on every shell block, compile() on every Python block, 20 marketing-filler regexes, decorative emoji (warning), banned claims, terminology.

Ban/avoid/fluff matching runs on a scrubbed view: fenced code, inline code, wiki-links, anchor targets and filenames are blanked (offset-preserving) before matching; a negation or hedge within ~90 characters in the same sentence downgrades a ban hit to a WARNING (never a suppression); exemptions entries report as INFO so they cannot become a rubbish drawer. The self-test (T1: 22/22) covers all three fixture families: a deliberately broken page (9 detections), the ban meta-key regression (_comment must never fire; exactly one real phrase fires), and the context regression (negation, anchor slug, inline code, exemptions).

Exit codes: 0 clean (or non-strict), 1 with --strict and errors, 2 usage/self-test failure.

qa/fix_toc.py

Rebuilds each page's ## Contents list from its actual headings using github_slug imported from qa_wiki.py (with a standalone fallback slugger — and the file deliberately avoids literal triple-backtick sequences, built as chr(96)*3, because a documentation aggregator once truncated it to 2,356 of 6,999 bytes).

python3 qa/fix_toc.py wiki/OWNER.REPO.wiki --bak       # write <file>.bak first
python3 qa/fix_toc.py wiki/OWNER.REPO.wiki --dry-run
python3 qa/fix_toc.py --self-test

The replacement is block-scoped: TOC_LIST_RE matches ## Contents plus the contiguous list lines only. That is the whole point — the first implementation replaced everything up to the next ---/## and ate a page's lead paragraph, badges and an alert block on a page without a --- separator (PITFALLS D8). The self-test fixture is that accident: lead, badge and alert must survive, stale entries must vanish, new anchors must appear, and a second run must be a no-op (verified T2: 8/8 including idempotence). Skips _-prefixed chrome files. Exit 0 normally, 1 on self-test failure.

qa/verify_commands.py

Phase 4's "re-run what you documented" harness, shipped as a worked template from the MD.Scribe run rather than a generic runner: it fresh-clones from GitHub (not the workspace copy — you verify what a reader would get), starts the documented server on a free port, then runs the documented curl/recipe set (404 page, API search, CSP header, cookie-jar CSRF flow, one-liner API-key extraction, manifest re-hash) through ok(name, cond, detail) into an evidence JSON. In its originating run it scored 19/19 and caught real manifest drift (a .gitignore byte-count mismatch and composer-generated files that never hash equal to a fresh install). Adapt the command list to your subject; the shape (fresh clone → documented commands → evidence) is the reusable part.

publish/publish.sh

Seven numbered steps, no prompts, no credentials needed:

0. Pre-flight — absolutise the wiki dir (a relative path silently breaks after the later cd), count .md files, note (never block on) hyphenated filenames.
1. QA gate — qa_wiki.py --strict; on failure: "Fix the errors above, then re-run."
2. Clone https://github.com/OWNER/REPO.wiki.git — reports the default branch (GitHub wikis use master) and existing history; on clone failure (uninitialised wiki) falls back to git init -b master with origin configured, so the hand-over is self-describing even after snapshots strip .git/config (PITFALLS A3).
3. Copy + stage — prints git status --short.
4. Commit — identity from GIT_AUTHOR_NAME/GIT_AUTHOR_EMAIL (default wiki-bot); the message embeds file and word counts; subject overridable via WIKI_COMMIT_SUBJECT.
5. Bundle — git bundle create … master (fallback --all) then git bundle verify with output printed.
6. PUBLISH.md — the written hand-over: clone path, branch, commit SHA, remote, bundle path, file count, push instructions, bundle-apply instructions (with the B8 warning: cloning a bundle directly needs -b master), the never-force-push rule, and a post-publish checklist (sidebar renders, wiki-links resolve, mermaid renders, alerts render).
7. Push — only with --push; failure prints the expected reason (missing credentials) and the two manual recipes.

bash publish/publish.sh workspace/wiki/OWNER.REPO.wiki OWNER/REPO          # hand-over
bash publish/publish.sh workspace/wiki/OWNER.REPO.wiki OWNER/REPO --push   # attempt push

Environment overrides: PUBLISH_OUT (clone dir), PUBLISH_BUNDLE (bundle path). Verified end-to-end (T15) against an uninitialised wiki remote: QA gate 0 errors → fallback init → commit a59eaa2 → bundle verified "okay" → PUBLISH.md written.

---

See also

• Configuration Reference — the same surface organised by flag and file, with JSON schemas
• Architecture — how the tools compose into the four phases
• The Four Phase Workflow — when to run which tool
• Troubleshooting — what tool failures mean
• Extending the Kit — adding suites and tools