Skip to content

Test Harness

Jump to bottom
Chris Sweet edited this page May 31, 2026 · 1 revision

type: concept up: "Home_llm-wiki-memory-template" extends: "Lesson-Validation-Methodology" status: in-review tags: [testing, ci, structural-validation, harness, in-pr]

Test Harness

A bash test harness that exercises both the template's shipping scripts and the proposed MVP additions to them. Lives at scripts/test-mvp/ in the fork chrissweet/llm-wiki-memory-template on branch add-test-harness, currently open as PR #10 against the template's main branch (addresses issue #5).

The harness implements the structural half of Lesson-Validation-Methodology: it does not validate live agent behavior, only that the template's scripts and the MVP's proposed scripts install correctly, behave correctly on artificial input, and remain idempotent.

Status

  • Open in PR #10, not yet merged into main.
  • CI green on both ubuntu-latest and macos-latest.
  • Total assertions: 91 (14 smoke + 77 e2e across 6 stages).
  • See Implementation-Status for current classification.

What the harness does

Two modes of operation, one harness:

Mode Target What it catches
Smoke A real template clone (local or cloned over the network) Regressions in the shipping scripts themselves: bash syntax, instantiate.sh substitutions, init-wiki.sh idempotence, namespaced nav files
E2e A simulated derivative with the MVP patch.sh applied Behaviour of the MVP's proposed additions: identity, fetch/pull, collision guard, pre-write awareness, namespacing lint

Smoke runs against the template as it is. E2e runs against the template as the MVP proposes it should be, with the proposed change layered on by patch.sh per stage.

Architecture

scripts/test-mvp/
├── README.md
├── run.sh              # discovers stages, runs in category order, reports
├── lib/
│   ├── assert.sh       # assert / assert_eq / assert_ne / assert_contains / assert_not_contains / skip
│   ├── sandbox.sh      # mktemp dir + cleanup trap
│   └── template.sh     # clone_template (real) + init_derivative (simulated)
└── tests/
    ├── smoke/template-bootstrap/        { patch.sh, assertions.sh }
    └── e2e/
        ├── stage1-identity/
        ├── stage2-status-lint/
        ├── stage3-fetch-pull/
        ├── stage4-collision-guard/
        ├── stage5-pre-write-awareness/
        └── stage6-unified-vault/

Each stage is a directory containing two scripts. patch.sh sets up the scenario (writes files, runs scripts, simulates other-machine state); assertions.sh then runs the checks. run.sh discovers stages by directory and invokes them in category order (smoke, unit, integration, e2e, regression, then any other category alphabetically).

patch.sh runs in a subshell; assertions.sh is sourced so its assert_* calls update the shared PASS, FAIL, SKIP, FAILED_TESTS globals.

What it tests today

Smoke: template-bootstrap (14 assertions)

Clones the template, runs instantiate.sh, asserts the result has:

  • Bash syntax valid for every shipping script
  • CLAUDE.md generated from CLAUDE.md.template with placeholders substituted
  • Wiki sub-repo created at wiki/<repo>.wiki/
  • Namespaced navigation files present (Home_<repo>.md, index_<repo>.md, log_<repo>.md, SCHEMA_<repo>.md)
  • init-wiki.sh is idempotent (running again is a no-op)

E2e (77 assertions across 6 stages)

Stage Assertions What it tests Scope status in trimmed MVP
stage1-identity 12 .claude/agent-id file; SessionStart hook installs a commit-msg trailer injector; Co-Authored-By and Agent-Instance trailers appear on commits; instance ID is stable on one path and varies across paths; custom handle overrides the computed default Co-Authored-By kept; Agent-Instance assertions deferred
stage2-status-lint 14 SCHEMA gets status: lifecycle field (canonical / contribution / reconciled / declined / superseded), curator observation fields, directional inverses; wiki-lint-check.sh enforces these per status; all five statuses lint orthogonally; curator fields are independent Deferred (no reconciler consumer yet)
stage3-fetch-pull 10 SessionStart hook fetches the wiki sub-repo; auto-pulls on fast-forward; reports incoming commits; defers (no auto-merge) on divergence Kept
stage4-collision-guard 15 Pre-push hook installs in the wiki sub-repo; clean push succeeds; rebaseable push triggers fetch + rebase, aborts with a "Re-run git push" message; conflicting push aborts with a BLOCKED message and recovery steps Kept
stage5-pre-write-awareness 13 PreToolUse hook on wiki Edit / Write: silent when up-to-date, reports incoming changes with page names + authors when behind, silent on non-wiki paths, always exits 0 (non-blocking), defensive on missing event fields Deferred (awareness without safety; SessionStart + pre-push cover the safety case)
stage6-unified-vault 13 SCHEMA documents the namespacing mandate; wiki-lint-namespacing.sh accepts suffixed nav pages, flags unsuffixed index.md / log.md / SCHEMA.md, allows Home.md as a small redirect bridge, flags Home.md with substantive content; two projects' nav pages coexist in a unified vault Kept (namespacing portion only; simplified patch.sh)

Each stage's patch.sh is idempotent: re-running on a sandbox where it has already been applied is a no-op.

Parts missing from the related feature report

The MVP feature report (the upcoming PR description for the MVP itself) describes the proposed template changes. The harness covers most of it, but the alignment is not perfect:

The harness covers more than the trimmed MVP plans to ship. Three areas of coverage exercise features currently slated for deferral in the minimal-change pass:

  • stage2-status-lint: tests the status lifecycle and status-aware lint. The trimmed MVP defers this because there is no reconciler consumer yet to read the status field.
  • stage5-pre-write-awareness: tests the PreToolUse mid-session awareness hook. The trimmed MVP defers this because SessionStart auto-pull + pre-push collision guard already cover the safety case; the awareness hook is "early warning" only.
  • Agent-Instance trailer assertions inside stage1-identity: the trimmed MVP keeps Co-Authored-By but drops Agent-Instance as niche multi-machine noise.

When the MVP is opened upstream, these stages either trim (Option A: delete or simplify, with v2 test specs preserved in Omniscient_2.wiki Future- pages so they can be regenerated) or stay (Option B: ship the v2-ready harness alongside a v1-scope MVP). Option A is the current plan.

The harness does NOT cover one feature the MVP plans to add. The ALWAYS_FILES updates to both sync scripts (scripts/update-from-template.sh and scripts/check-template-version.sh) are mandatory for the MVP's three new files (.claude/agent-id.template, .claude/hooks/session-start.sh, scripts/wiki-lint-namespacing.sh) to actually propagate to existing derived projects. Sync-Flow documents this footgun and PR #1 hit exactly this bug. The harness should grow a smoke-test addition that diffs the template tree against both sync scripts: any file under .claude/, wiki/agents/, or scripts/ that exists but is not listed in the appropriate ALWAYS_FILES / CLAUDE_FILES / CURSOR_FILES array fails the smoke test. That would have caught PR #1 automatically. Filed as a follow-up; not yet implemented in the harness.

GitHub Action

The CI workflow at .github/workflows/test-harness.yml:

name: test-harness

on:
  push:
    branches: ["**"]
  pull_request:

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - name: Install jq (Linux)
        if: matrix.os == 'ubuntu-latest'
        run: sudo apt-get update -qq && sudo apt-get install -y jq
      - name: Verify bash and jq
        run: bash --version | head -1 && jq --version && git --version
      - name: Run test harness (smoke + e2e)
        env:
          MVP_TEMPLATE_LOCAL: ${{ github.workspace }}
        run: ./scripts/test-mvp/run.sh
      - name: Run smoke category only (re-run for clean per-category output)
        if: always()
        env:
          MVP_TEMPLATE_LOCAL: ${{ github.workspace }}
        run: ./scripts/test-mvp/run.sh --category=smoke

Key design choices:

  • Matrix on ubuntu-latest + macos-latest. macOS in CI is load-bearing. macOS default bash is 3.2 (mapfile is unavailable; set -u interacts badly with empty arrays); mktemp returns paths under /var/folders/* that resolve via symlink to /private/var/folders/*. Both bit us during harness development. Linux-only CI would have missed both classes of bug. Run both, fail-fast: false, so a Linux-only or macOS-only regression is visible without masking the other runner.
  • jq installed on Linux only. macOS runners have jq preinstalled via Homebrew on the GitHub runner image.
  • MVP_TEMPLATE_LOCAL=${{ github.workspace }}. Uses the checked-out template directly for smoke; no network clone needed. Significant speedup; also avoids fork/branch authentication ambiguity in cross-fork PRs.
  • Two passes (full, then smoke-only). The second pass re-runs smoke alone for clean per-category log output. if: always() ensures it runs even if the full pass had failing assertions, so the smoke verdict is visible regardless.

How to extend

Add a new stage:

  1. Pick a category. Existing: smoke, unit, integration, e2e, regression. Default to e2e for full-sandbox scenarios; regression for tests that pin a fixed bug; smoke for fast structural checks against real template state.
  2. Create the directory: tests/<category>/<test-name>/. Use kebab-case for the test name.
  3. Write patch.sh: idempotent setup. Takes the sandbox path as $1. Can source lib/template.sh and lib/sandbox.sh for helpers (each patch.sh runs in a subshell, so sourced state does not leak). 
    #!/usr/bin/env bash
set -euo pipefail
SANDBOX="$1"
cd "$SANDBOX"
# set up your scenario: write files, run scripts, simulate other-machine state
  4. Write assertions.sh: runs the checks. Sourced by run.sh (so it shares the PASS / FAIL / SKIP / FAILED_TESTS globals). Use the helpers from lib/assert.sh: 
    SANDBOX="$1"
cd "$SANDBOX"
assert "feature X is wired up" "[ -f path/to/file ]"
assert_eq "value matches" "expected" "$(cat some-file)"
assert_contains "config has the right line" "config.json" "key: value"
assert_not_contains "no leaked secret" "result.log" "PRIVATE_KEY"
skip "behavioral check" "requires live Claude Code session"
  5. Run locally: ./scripts/test-mvp/run.sh <test-name> runs just that stage. --no-cleanup keeps the sandbox dir for inspection.
  6. No CI workflow change needed. The existing run.sh invocation auto-discovers the new directory.

Extend the GitHub Action itself if you need a third matrix runner (e.g. windows-latest with WSL), an additional environment variable, or a separate job for a heavyweight stage that should not block the fast checks.

What can't be tested here

  • Live agent behavior. The harness verifies that scripts and hooks install correctly and behave correctly on artificial input, but cannot verify that a real Claude Code session invokes the SessionStart hook at the right moment or that the agent reacts to a PostToolUse reminder as designed. That requires either an LLM in the loop or an agent-compatible test harness that does not exist publicly today. See Lesson-Validation-Methodology for the structural-vs-behavioral split. The MVP spec's verification plan handles the behavioral half in two separate phases (B: Omniscient_2 dry-run; C: cross-machine).
  • Cross-machine race conditions. Simulating two machines by copying the derivative to a second path covers identity variance and stable-per-path ID generation, but not concurrent commits with real network round-trips. That belongs in a different test infrastructure.
  • Network failure modes. Smoke tests gracefully skip if neither MVP_TEMPLATE_LOCAL nor the network repo is reachable, but the harness does not exercise partial-network-failure paths.

See also

Clone this wiki locally