feat: ilo test subcommand for -- run: / -- out: assertions

[danieljohnmorris]

Summary

Pending #52: a user-facing ilo test subcommand that runs the same -- run: / -- out: / -- err: annotation format tests/examples_engines.rs already uses.

Manifesto framing: every .ilo example in the repo already carries embedded assertions and the integration harness runs them. Surfacing that same format as ilo test means end-user programs and agent-written test suites can assert behaviour the same way examples do, with zero new syntax to learn. Token win: one annotation pattern covers in-repo examples, agent docs, and user tests.

Repro

Before: an agent writing tests for its own ilo program had no in-language test runner. The closest thing was hand-rolled bash scripts piping ilo <file> into diff, or copying the Rust integration harness verbatim.

After:

$ cat tests/sum.ilo
-- run: m
-- out: 3
m>n;+1 2

$ ilo test tests/sum.ilo
PASS  tests/sum.ilo::m (line 1)

1 passed, 0 failed

ilo test examples/ runs every example in the repo as a smoke test (~ same set the integration harness covers).

What's in the diff

• feat: ilo test subcommand - src/cli/test_runner.rs (new) implements the file/directory walker, annotation parser (parse_cases / parse_engine_skips), engine selector (vm / jit / all), and subprocess fan-out. Cmd::Test(TestArgs) added to src/cli/args.rs; dispatch wired in src/main.rs. Path defaults to examples/ when omitted.
• test: end-to-end coverage for ilo test - 7 subprocess tests pin pass / fail / -- err: / dir recursion / --engine all / missing-path / no-annotations paths. Spawns the ilo binary so the test exercises real CLI parsing + the runner's own subprocess fan-out.
• doc: sync ilo test surface across SPEC, ai.txt, skill - SPEC.md CLI invocation line and **ilo test** paragraph next to ilo check. ai.txt agent-spec entry inline. skills/ilo/ilo-agent.md Testing section with the three canonical invocations.

Test plan

• cargo test --release --features cranelift --test cli_test_cmd - 7 passing
• cargo fmt && cargo clippy --release --features cranelift --all-targets -- -D warnings - clean
• Full suite passes except pre-existing regression_reserved_names_doc::spec_reserved_short_names_match_builtin_registry (b64/hex SPEC drift, unrelated to this PR)
• Manually verified ilo test examples/ walks the examples dir and reports per-file PASS/FAIL

Follow-ups

• Could add a --json output mode in a follow-up if agents want a machine-parseable summary instead of the human PASS/FAIL lines.

[codecov]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
4734	2	4732	0

View the top 2 failed test(s) by shortest run time

ilo::examples::examples

Stack Traces | 7.85s run time

thread 'examples' (42138) panicked at tests/examples.rs:158:9:
9/1167 example test(s) failed:

crypto-primitives.ilo (line 42): `ilo .../ilo/examples/crypto-primitives.ilo sha-empty`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43203) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo (line 44): `ilo .../ilo/examples/crypto-primitives.ilo sha-abc`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43209) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo (line 46): `ilo .../ilo/examples/crypto-primitives.ilo hmac-rfc2`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43215) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo (line 48): `ilo .../ilo/examples/crypto-primitives.ilo b64-roundtrip`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43221) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo (line 50): `ilo .../ilo/examples/crypto-primitives.ilo b64-padded`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43227) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo (line 52): `ilo .../ilo/examples/crypto-primitives.ilo hex-abc`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43233) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo (line 54): `ilo .../ilo/examples/crypto-primitives.ilo ct-eq-yes`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43239) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo (line 56): `ilo .../ilo/examples/crypto-primitives.ilo ct-eq-no`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43244) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo (line 58): `ilo .../ilo/examples/crypto-primitives.ilo ct-eq-len`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43250) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

ilo::examples_engines::examples_all_engines

Stack Traces | 7.88s run time

thread 'examples_all_engines' (42150) panicked at tests/examples_engines.rs:235:9:
9/1153 multi-engine example test(s) failed:

crypto-primitives.ilo [vm] (line 42): `ilo .../ilo/examples/crypto-primitives.ilo --vm sha-empty`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43412) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo [vm] (line 44): `ilo .../ilo/examples/crypto-primitives.ilo --vm sha-abc`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43416) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo [vm] (line 46): `ilo .../ilo/examples/crypto-primitives.ilo --vm hmac-rfc2`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43420) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo [vm] (line 48): `ilo .../ilo/examples/crypto-primitives.ilo --vm b64-roundtrip`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43423) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo [vm] (line 50): `ilo .../ilo/examples/crypto-primitives.ilo --vm b64-padded`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43428) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo [vm] (line 52): `ilo .../ilo/examples/crypto-primitives.ilo --vm hex-abc`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43432) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo [vm] (line 54): `ilo .../ilo/examples/crypto-primitives.ilo --vm ct-eq-yes`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43436) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo [vm] (line 56): `ilo .../ilo/examples/crypto-primitives.ilo --vm ct-eq-no`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43440) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

crypto-primitives.ilo [vm] (line 58): `ilo .../ilo/examples/crypto-primitives.ilo --vm ct-eq-len`
  FAILED (exit exit status: 101)
  stderr: thread 'main' (43444) panicked at src/vm/mod.rs:1877:13:
auto-unwrap on a non-Result tree-bridge builtin slipped past verify
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}