Extending the Kit

Extending the Kit

The kit is designed to be extended during a run — the README's maintenance note says it plainly: "If you improve one during a run — and you probably will, since each new subject surfaces a new edge case — commit the improvement back. The toolkit is part of the delivery, not scaffolding to be thrown away." This page covers the four extension points (suites, serve commands, QA inputs, tools), the contracts each must respect, and the CI discipline that keeps extensions honest. A live example from this wiki run — finding and fixing a scaffold defect — closes the page.

Contents

• Extension point 1 — probe suites
• Extension point 2 — serve commands and non-server subjects
• Extension point 3 — QA inputs (ban/terms) without touching code
• Extension point 4 — new tools and the CI contract
• Worked example — a defect found and fixed during this wiki run
• Contributing back
• See also

---

Extension point 1 — probe suites

A suite is a plain function registered in one dict (probe/webapp_probe.py, SUITES):

def suite_my_subject(probe, entry):
    """Anything with a URL works; batteries are transport-level."""
    out = {}
    probe.run("landing", entry, expect=200, group="modes")
    out["traversal"] = probe.battery_traversal(
        ["/download?path={PAYLOAD}"],
        deny_markers=["invalid path", "not found"],   # BODY markers, not statuses
    )
    out["injection"] = probe.battery_injection(
        [{"name": "comment", "path": "/comment", "field": "body", "via": "post"}],
    )
    out["headers"] = probe.battery_headers([entry, "/admin"])
    out["files_written"] = probe.battery_disk()
    return out

SUITES["my-subject"] = suite_my_subject

Contracts to respect:

1. Return a dict; it lands in the evidence JSON under extra. probe.results accumulates every run() record automatically.
2. Assert on body markers, not status codes. A front controller answers 200 for almost any path (the router falls through to the home page); run 2's first "successful" probe was exactly that — a correct status and a meaningless result. The check(name, resp, marker=…, status=…) shape from examples/run2-mdscribe/ is the model.
3. Config mutation needs the staged app. battery_flags and app.set_ini(...) require --repo mode (a ServedApp); against --base-url the battery returns an explicit error record instead of guessing.
4. Ordered flows do not belong in batteries. Batteries are declarative and order-independent; a setup → login → actions → logout sequence uses lib/session.py and a hand-written driver (the run-2 battery is 1,571 lines of exactly that).
5. Write evidence through lib/evidence.write_evidence(), never a bare Path(...).write_text(...) as the last line of a long probe — filesystem hiccups at the worst moment are documented (PITFALLS B11), and the fallback directory exists for that.

For subjects with two deployment layouts, stage two instances in one driver — some endpoints are physically unreachable in one layout:

with ServedApp(repo, copy=['public', 'src'], serve_cmd='php -S {host}:{port} -t {root}/public router.php') as A, \
     ServedApp(repo, copy=['api.php', 'src'], serve_cmd='php -S {host}:{port} -t {root}') as B:
    pass  # probe A's contract and B's contract separately; never merge the evidence

Extension point 2 — serve commands and non-server subjects

ServedApp needs only a command template with {host}, {port}, {root}:

Subject	--serve-cmd
PHP (default)	php -S {host}:{port} -t {root}
Node	node {root}/server.js (read host/port from env you export)
Python WSGI dev	python3 -m wsgiref.simple_server ... or your app's dev command
Static site	python3 -m http.server {port} -d {root} (verified, T10)
Go/Rust binary	build first, then {root}/bin/app --listen {host}:{port}

CLI-shaped subjects skip serving entirely: write a suite that shells out and captures stdout/stderr/exit codes into the same evidence shape — declarative cases → JSON → Markdown tables. Library-shaped subjects: execute every documented recipe in a clean venv; the recipes are the probe surface. The reusable part is the shape, not the transport.

Extension point 3 — QA inputs (ban/terms) without touching code

Most "extension" during a run is authoring, not programming:

• New banned claims — every verified falsehood or unverifiable claim goes into sources.md §D first (with the reason), then into ban.json as an affirmative phrase. Prefer the shortest phrase that is unambiguously wrong; banning "syntax highlighting" would also hit the sentence that says there is none — ban the affirmative form instead.
• New canonical terms and synonyms — from the domain-notes glossary into terms.json. The synonym list is the valuable half; keep patterns narrow (a specific wrong phrase beats a common noun — two shipped patterns had to be removed after firing on correct prose in a 73k-word wiki, PITFALLS D6).
• Exemptions over mangling — when a legitimate hit (a verbatim subject string, a domain term) collides with a pattern, record {phrase, file?, quote?, reason}; it reports as INFO forever after, visible but not failing. The subject's verbatim strings always win over terminology purity; log the decision in _tuning_log.

Extension point 4 — new tools and the CI contract

Adding a tool to the kit carries obligations, all enforced by .github/workflows/selftest.yml:

1. --help must work — the workflow runs it on every listed tool and prints a NOTE for any .py in the tree that is neither exercised nor excused. Six tools once shipped invisible to CI; that is the reason the list is policed.
2. Self-tests for anything with failure modes — the doctrine: a checker that cannot fail is worse than no checker. qa_wiki.py (22 detections), fix_toc.py (8, including a fixture that is the accident it once caused), evidence.py (3, with a cross-platform unwritable path), session.py (8, live PHP) are the models. A self-test fixture should reproduce the worst case you actually hit.
3. Syntax gates — ast.parse for Python, bash -n for shell, json.load for every tracked .json (a facts file once shipped as Markdown with a fence inside; the glob that missed it was templates-only).
4. Fail loudly — every tool in the reference table is marked "fails loudly? yes": non-zero exits, printed diagnoses, no silent no-ops. An all-meta ban.json warns instead of passing silently, because a silent no-op lets you believe bans are enforced.
5. Destructive operations need guards — the bulk-edit contract from fix_toc.py: replace only the block you named, assert a word-count sanity bound, support --bak, ship a self-test reproducing the accident that motivated the tool.
6. Docs updates — README tool table, SKILL.md files table, AGENTS.md tool reference, and the CI --help list move together; the workflow has a documented-paths step that fails when docs reference files that do not exist.

Template rule (PITFALLS D7): never hardcode a page name in a template. Derive links from the list actually being created, and test the scaffold round-trip against at least three page lists including the degenerate one-page case — a hardcoded footer link became a site-wide error once, because the footer renders on every page.

A subtle content rule for tools that ship code inside docs: fix_toc.py's standalone fallback builds its fence string as chr(96)*3 instead of a literal, because a documentation aggregator embedding the source into Markdown truncated the file at its first literal fence (2,356 of 6,999 bytes survived). If your tool's source will be quoted anywhere, assume naive triple-backtick embedding.

Worked example — a defect found and fixed during this wiki run

While scaffolding this wiki with --progress PROGRESS.json (the explicit-grouping branch of new_page.py), the run produced 21 files where 22 were announced: _Footer.md was missing. Reading the source confirmed it — the explicit branch wrote _Sidebar.md and the stubs and returned, while only the heuristic branch wrote the footer. CI could not catch it: the scaffold round-trip step uses the heuristic path, and a missing _Footer.md is a warning, not an error, so --strict stayed green.

The fix followed the kit's own doctrine: derive the footer's links from the actual page list (never hardcode), write it in both branches, and re-verify — round-trips green on all three CI page lists, the 22-file skeleton audited at 0 errors, and the finding recorded in the run's REPORT.md as a maintenance contribution with a suggested CI addition (extend the round-trip to a --progress-driven page list). That loop — use the tool, notice the anomaly, read the source, fix minimally, re-verify, record — is the intended lifecycle of the kit itself.

Contributing back

The repository has no CONTRIBUTING file at 7e17a11; the operative instructions are the README maintenance note and the CHANGELOG's style (every release lists what was verified and how, including negative controls). A useful contribution therefore arrives as: the defect or edge case, the reproducing test, the minimal fix, and the verification story — the same four-part shape the CHANGELOG entries use. Keep run workspaces out of the tree (.gitignore and the CI workspace-refusal step exist because a 184-file run directory nearly shipped).

---

See also

• Tool Reference — the internals you extend
• Research and Probing Guide — suite-authoring context and field rules
• QA and Audit Guide — tuning ban/terms in practice
• Architecture — layering rules new tools must respect
• Changelog and Versioning — how extensions became releases