test(mutate): Add mutation-testing harness and strengthen cobs/retry

Mutation testing measures test strength (would a test notice if a line were wrong?), complementing coverage (reach) and fuzzing (memory/UB).

Harness (tools/mutation-testing/mutate.py, wired as `ci.sh --mutate`):
- libclang AST finds mutation sites, so a comparison `<` is mutated but a
template `<` is not, and strings/comments are untouched; un-typecheckable mutants are excluded from the score rather than miscounted as survivors.
- Each mutant builds against a shadow include tree (a symlink mirror of
include/ with only the mutated header materialised, used as the sole include root) so angle and quote includes resolve to the same file and #pragma once still dedupes. A flat -I overlay -I include lets quoted includes slip past to the real header and double-include it.
- doctest_main.o is cached; covering test TUs run fastest-first and
short-circuit at the first kill, keeping the common case cheap.
- On-demand audit, NOT a per-commit gate: minutes long and scores are
noisy (equivalent mutants survive legitimately and need triage). Self-skips when python3 / clang.cindex / a compiler is absent.

Durable output is the assertions the audit drove into the suite:
- retry.hpp 66.7% -> 97.2%: TransactionTiming defaults; in-loop
early-return timing record (the timing tests used max_retries=0, which skips the loop entirely); timeout budget measured relative to a non-zero start clock; inclusive timeout boundary; the timeout_ms==1 edge -- for both retry_loop and retry_transaction. The lone remaining survivor is an equivalent mutant (the wait is zero-length at the gap boundary).
- cobs.hpp 75.9% -> 79.3%: an exact-length sweep for cobs_encoded_length
(block-code accounting), a test for the previously-untested scratch-buffer encode() overload, and an incremental-flush assertion. Remaining survivors are equivalent mutants or ASan-territory buffer bounds, not missing assertions.

Full host and Arduino test suites pass.

Author: m-mcgowan

ci.sh

@@ -16,6 +16,10 @@ ROOT="$(cd "$(dirname "$0")" && pwd)"
 #   ./ci.sh --coverage       Build with coverage instrumentation and generate report
 #   ./ci.sh --integrations   Build and run JSON backend integration tests (host)
 #   ./ci.sh --fuzz           Fuzz the JSON/JSONB parsers under ASan/UBSan
+#   ./ci.sh --mutate         Mutation-test the correctness-critical headers
+#                            (test-strength audit; on-demand, NOT a gate).
+#                            Extra args pass through, e.g. --mutate --all or
+#                            --mutate --target jsonb
 #   ./ci.sh --hil            Hardware-in-the-loop sweep — upload + run all
 #                            test groups on $PIO_LABGRID_DEVICE for the
 #                            `serial` and `i2c` envs in tests/integration/firmware
@@ -1239,6 +1243,32 @@ run_fuzz() {
     echo "Parser fuzzing passed (corpus + ${iters}x3 iterations)."
 }
 
+# ── Mutation testing ──────────────────────────────────────────────────────
+# Deliberately corrupt one operator/literal at a time in the correctness-
+# critical headers, rebuild the covering host tests, and check a test fails
+# ("kills" the mutant). A surviving mutant = a weak/missing assertion (or a
+# legitimate equivalent mutant). This measures test *strength*, complementing
+# coverage (reach) and fuzzing (memory/UB on bad input).
+#
+# On-demand audit, NOT a per-commit gate: it is minutes long and scores are
+# noisy (equivalent mutants survive and need human triage). It exits 0 even
+# with survivors; the durable output is the assertions the audit drives into
+# the suite. Self-skips when python3 / clang.cindex / a compiler is absent.
+# Pass-through args ($@) reach the harness (e.g. --all, --target NAME).
+run_mutate() {
+    ci_stage "Mutation testing (test strength)"
+    if ! command -v python3 >/dev/null 2>&1; then
+        echo "  no python3 found — skipping mutation testing."
+        return 0
+    fi
+    if ! python3 -c "import clang.cindex" >/dev/null 2>&1; then
+        echo "  clang.cindex not installed — skipping mutation testing."
+        echo "  enable with:  python3 -m pip install libclang"
+        return 0
+    fi
+    python3 "$ROOT/tools/mutation-testing/mutate.py" "$@"
+}
+
 run_full() {
     # Host CI + hardware sweep. The hardware step is self-skipping when
     # $PIO_LABGRID_DEVICE isn't set, so this stays safe in headless
@@ -1271,6 +1301,10 @@ case "${1:-}" in
     --fuzz)
         run_and_log run_fuzz
         ;;
+    --mutate)
+        shift
+        run_and_log run_mutate "$@"
+        ;;
     --hil)
         run_and_log run_integration_hardware
         ;;

tests/test_cobs.cpp

@@ -212,3 +212,72 @@ TEST_CASE("COBS: encoder does not modify source") {
 
     REQUIRE(memcmp(data, copy, sizeof(data)) == 0);
 }
+
+// ---------------------------------------------------------------------------
+// Strength tests surfaced by mutation testing (tools/mutation-testing).
+// ---------------------------------------------------------------------------
+
+TEST_CASE("COBS: cobs_encoded_length equals actual encoded length (sweep)") {
+    // cobs_encoded_length feeds the JSON handshake `cobs` field, so it must
+    // equal the number of bytes the encoder actually emits (excluding EOP) for
+    // every length and zero pattern — especially the block-boundary code
+    // accounting around multiples of 254. A coarse spot-check leaves the
+    // code-reset arithmetic unconstrained; this sweep pins it.
+    auto check = [](const std::vector<uint8_t>& d) {
+        size_t predicted = note::cobs_encoded_length(d.data(), d.size());
+        size_t actual = stream_encode(d.data(), d.size()).size();
+        CAPTURE(d.size());
+        CHECK(predicted == actual);
+    };
+    for (size_t len = 0; len <= 600; ++len) {
+        std::vector<uint8_t> d(len);
+        // Dense non-zero data — forces 254-byte block splits (code reaches 0xFF).
+        for (size_t i = 0; i < len; ++i) d[i] = static_cast<uint8_t>((i % 255) + 1);
+        check(d);
+        // Periodic zeros — forces zero-driven block boundaries.
+        for (size_t i = 0; i < len; ++i)
+            d[i] = static_cast<uint8_t>((i % 7 == 0) ? 0 : (i % 251) + 1);
+        check(d);
+    }
+}
+
+TEST_CASE("COBS: scratch-buffer encode overload matches the stack-buffer overload") {
+    // The encode(src, len, scratch, flush) overload (caller-provided buffer) is
+    // a separate code path from the stack-buffer encode(); it must produce the
+    // identical encoding and still round-trip.
+    std::vector<uint8_t> data(600);
+    for (size_t i = 0; i < data.size(); ++i)
+        data[i] = static_cast<uint8_t>((i % 5 == 0) ? 0 : (i * 7 + 3) % 256);
+
+    note::CobsEncoder enc;
+    uint8_t scratch[NOTE_COBS_BLOCK_SIZE];
+    std::vector<uint8_t> via_scratch;
+    enc.encode(data.data(), data.size(), note::byte_span(scratch, sizeof(scratch)),
+               [&](const uint8_t* b, size_t n) { via_scratch.insert(via_scratch.end(), b, b + n); });
+
+    CHECK(via_scratch == stream_encode(data.data(), data.size()));
+
+    auto decoded = stream_decode(via_scratch.data(), via_scratch.size());
+    REQUIRE(decoded.size() == data.size());
+    CHECK(memcmp(decoded.data(), data.data(), data.size()) == 0);
+}
+
+TEST_CASE("COBS: decoder emits incrementally for large payloads") {
+    // The decoder bounds its fixed working buffer by flushing as blocks fill,
+    // rather than buffering the whole payload before emitting once.
+    std::vector<uint8_t> data(2000);
+    for (size_t i = 0; i < data.size(); ++i) data[i] = static_cast<uint8_t>((i * 5 + 1) % 256);
+    auto encoded = ref_encode(data.data(), data.size());
+
+    note::CobsDecoder decoder;
+    int sink_calls = 0;
+    std::vector<uint8_t> decoded;
+    auto sink = [&](const uint8_t* b, size_t n) { ++sink_calls; decoded.insert(decoded.end(), b, b + n); };
+    decoder.feed(encoded.data(), encoded.size(), sink);
+    uint8_t term = note::cobs_eop;
+    decoder.feed(&term, 1, sink);
+
+    CHECK(sink_calls > 1);
+    REQUIRE(decoded.size() == data.size());
+    CHECK(memcmp(decoded.data(), data.data(), data.size()) == 0);
+}

tests/test_retry.cpp

@@ -542,6 +542,125 @@ TEST_CASE("retry_loop: records timing on failure") {
     CHECK(timing.last_transaction_end_ms == 200);
 }
 
+// ---------------------------------------------------------------------------
+// Timing defaults and timeout-boundary edges
+//
+// These pin behaviour that the matrix above leaves unconstrained — surfaced by
+// mutation testing (tools/mutation-testing): the default gap, the in-loop
+// early-return timing record, the relative-to-start timeout math, and the
+// inclusive timeout boundary all survived mutation without a covering assert.
+// ---------------------------------------------------------------------------
+
+TEST_CASE("retry: TransactionTiming has the documented defaults") {
+    TransactionTiming t;
+    CHECK(t.min_gap_ms == 2);              // default minimum inter-transaction gap
+    CHECK(t.last_transaction_end_ms == 0);
+    CHECK_FALSE(t.has_previous);
+}
+
+TEST_CASE("retry_loop: records timing on in-loop early returns") {
+    // Distinct from the max_retries==0 fall-through: with retries available the
+    // success and non-retryable branches return from *inside* the loop, and
+    // must still record timing so the next transaction's gap is enforced.
+    SUBCASE("first attempt already succeeded") {
+        LoopTestOps ops; ops.now_ms = 77;
+        auto t = ops.ops();
+        TransactionTiming timing;
+        RetryPolicy policy{.max_retries = 5};
+        AttemptTracker tracker;
+        bool ok = retry_loop(true, Error{}, AttemptTracker::attempt, &tracker,
+                             t, timing, Safety::ReadOnly, policy);
+        CHECK(ok);
+        CHECK(timing.has_previous);
+        CHECK(timing.last_transaction_end_ms == 77);
+    }
+    SUBCASE("first attempt failed non-retryably") {
+        LoopTestOps ops; ops.now_ms = 88;
+        auto t = ops.ops();
+        TransactionTiming timing;
+        RetryPolicy policy{.max_retries = 5};
+        AttemptTracker tracker;
+        bool ok = retry_loop(false, Error::Notecard, AttemptTracker::attempt, &tracker,
+                             t, timing, Safety::ReadOnly, policy);
+        CHECK_FALSE(ok);
+        CHECK(timing.has_previous);
+        CHECK(timing.last_transaction_end_ms == 88);
+    }
+}
+
+TEST_CASE("retry_loop: timeout budget is measured relative to start (non-zero clock)") {
+    LoopTestOps ops; ops.now_ms = 1000;   // non-zero start: catches millis()+start vs millis()-start
+    auto t = ops.ops();
+    TransactionTiming timing;
+    RetryPolicy policy{.max_retries = 100, .retry_delay_ms = 100, .timeout_ms = 250};
+    AttemptTracker tracker;
+    retry_loop(false, Error::SendFailed, AttemptTracker::attempt, &tracker,
+               t, timing, Safety::ReadOnly, policy);
+    CHECK(tracker.calls == 3);             // same budget math as the zero-start case
+}
+
+TEST_CASE("retry_loop: timeout boundary is inclusive (elapsed == timeout stops)") {
+    LoopTestOps ops;
+    auto t = ops.ops();
+    TransactionTiming timing;
+    RetryPolicy policy{.max_retries = 100, .retry_delay_ms = 100, .timeout_ms = 300};
+    AttemptTracker tracker;
+    retry_loop(false, Error::SendFailed, AttemptTracker::attempt, &tracker,
+               t, timing, Safety::ReadOnly, policy);
+    // 100ms steps, budget exactly 300: stop when elapsed first REACHES 300 (>=),
+    // not only when it exceeds (>). 3 retries, not 4.
+    CHECK(tracker.calls == 3);
+}
+
+TEST_CASE("retry_loop: timeout_ms == 1 still enforces the budget (boundary)") {
+    LoopTestOps ops;
+    auto t = ops.ops();
+    TransactionTiming timing;
+    RetryPolicy policy{.max_retries = 5, .retry_delay_ms = 100, .timeout_ms = 1};
+    AttemptTracker tracker;
+    retry_loop(false, Error::SendFailed, AttemptTracker::attempt, &tracker,
+               t, timing, Safety::ReadOnly, policy);
+    // timeout_ms==1 (the > 0 guard, not > 1): after one 100ms delay the budget
+    // is blown, so only one retry attempt runs.
+    CHECK(tracker.calls == 1);
+}
+
+TEST_CASE("retry: timeout budget is measured relative to start (non-zero clock)") {
+    MockClock clock; clock.now_ms = 1000;
+    TransactionTiming timing;
+    RetryPolicy policy{.max_retries = 100, .retry_delay_ms = 100, .timeout_ms = 250};
+    int attempts = 0;
+    retry_transaction<TestResult>(
+        clock, timing, Safety::ReadOnly, policy,
+        [&]() -> TestResult { ++attempts; return send_failed(); },
+        [&]() { clock.reset(); });
+    CHECK(attempts == 4);                  // 1 initial + 3 retries within budget
+}
+
+TEST_CASE("retry: timeout boundary is inclusive (elapsed == timeout stops)") {
+    MockClock clock;
+    TransactionTiming timing;
+    RetryPolicy policy{.max_retries = 100, .retry_delay_ms = 100, .timeout_ms = 300};
+    int attempts = 0;
+    retry_transaction<TestResult>(
+        clock, timing, Safety::ReadOnly, policy,
+        [&]() -> TestResult { ++attempts; return send_failed(); },
+        [&]() { clock.reset(); });
+    CHECK(attempts == 4);                  // 4th retry blocked at elapsed == 300
+}
+
+TEST_CASE("retry: timeout_ms == 1 still enforces the budget (boundary)") {
+    MockClock clock;
+    TransactionTiming timing;
+    RetryPolicy policy{.max_retries = 5, .retry_delay_ms = 100, .timeout_ms = 1};
+    int attempts = 0;
+    retry_transaction<TestResult>(
+        clock, timing, Safety::ReadOnly, policy,
+        [&]() -> TestResult { ++attempts; return send_failed(); },
+        [&]() { clock.reset(); });
+    CHECK(attempts == 2);                  // initial + one retry, then 1ms budget blown
+}
+
 // ---------------------------------------------------------------------------
 // Error to_string coverage
 // ---------------------------------------------------------------------------

tools/mutation-testing/README.md

@@ -0,0 +1,95 @@
+# Mutation testing
+
+Coverage tells you a line *executed*. It can't tell you whether a test would
+*notice* if that line were wrong. Mutation testing measures the difference —
+test **strength**, not test reach.
+
+The harness deliberately corrupts one operator or literal at a time in the
+correctness-critical headers, rebuilds the covering host tests, and runs them:
+
+- the mutant is **killed** if a test fails — good, an assertion caught the change;
+- it **survived** if every test still passes — a weak/missing assertion, *or* an
+  equivalent mutant that changes no observable behaviour.
+
+A surviving mutant on a critical path is the signal. **The durable deliverable
+is the assertion it drives** — a normal, permanent unit test added to the suite
+that would have caught the corruption. The mutants are ephemeral (applied to a
+throwaway overlay, never to the tree); the assertions live forever. This mirrors
+fuzzing, whose durable artifact is its committed corpus.
+
+## Running it
+
+```sh
+./ci.sh --mutate                 # PoC set (cobs + retry); on-demand audit
+./ci.sh --mutate --all           # every configured target
+./ci.sh --mutate --target jsonb  # one target
+python3 tools/mutation-testing/mutate.py --list-targets
+```
+
+Requires a C++ compiler (`c++` by default; override with `CXX`) and the
+`clang.cindex` Python bindings (`python3 -m pip install libclang`). The harness
+self-skips with a clear message if either is missing, so `./ci.sh --mutate` is
+safe to invoke anywhere.
+
+**This is an on-demand audit, not a per-commit gate.** It is minutes long, and
+scores are inherently noisy because equivalent mutants survive legitimately and
+need human triage — a hard gate would false-fail. It exits 0 even with
+survivors; run it after touching the parsers, framing, or retry logic, then
+triage the survivor list.
+
+## How it works
+
+```mermaid
+flowchart LR
+    A[libclang AST<br/>find mutation sites] --> B[apply ONE mutation<br/>to a shadow header]
+    B --> C[recompile covering<br/>test TUs]
+    C --> D[link vs cached<br/>doctest_main.o]
+    D --> E[run with timeout]
+    E -->|test fails| K[killed]
+    E -->|all pass| S[survived]
+    C -->|won't compile| U[uncompilable<br/>excluded from score]
+```
+
+- **Sites come from the AST**, not regex: a comparison `<` is mutated, a template
+  `Foo<int>` `<` is not, and strings/comments are never touched. Operators:
+  relational/equality/logical/arithmetic swaps, integer-literal `n±1` boundaries,
+  bool-literal flips, drop unary `!`, and standalone statement deletion.
+- **Shadow include tree.** Each mutant gets a private mirror of `include/` built
+  from symlinks, with only the mutated header materialised as a real file, used
+  as the *sole* include root. This makes both `<note/link/cobs.hpp>` and quoted
+  `"link/cobs.hpp"` includes resolve to the mutated copy uniformly — a flat
+  `-I overlay -I include` would let quote-includes slip past to the real header
+  and double-include it (`#pragma once` dedupes by file identity).
+- **Cached objects.** `doctest_main.o` + `alloc_counter.o` are compiled once; a
+  mutation only changes a header, so per mutant we recompile just the covering
+  test TU(s).
+- **Short-circuit.** Test TUs run fastest-first and stop at the first failure, so
+  a mutant killed by the quick unit test never pays the heavier TUs' compile.
+- A process pool runs ~(cores − 2) workers; a per-mutant run timeout guards
+  mutations that induce an infinite loop.
+
+## Targets
+
+`TARGETS` in `mutate.py` maps each header to the test TU(s) that exercise it.
+The mapping must list **every** TU that directly asserts the target's
+behaviour, not just the obvious unit test — a mutant is only killed if a
+*rebuilt* test fails. (e.g. `cobs_encoded_length` is asserted in
+`test_binary_execute.cpp`, not `test_cobs.cpp`; omitting it produces false
+survivors.) Verify a new target's mapping against host coverage
+(`./ci.sh --coverage`).
+
+## Triage
+
+For each survivor, decide:
+
+- **Real gap** → add a permanent assertion to the covering test that pins the
+  behaviour the mutant broke. Re-run; the survivor should become killed.
+- **Equivalent mutant** → the corruption changes no observable behaviour, so no
+  test can or should catch it. Common cases here: a default member initialiser
+  overwritten by a constructor's `reset()`, a `<`↔`<=` inside a `min`-style
+  ternary that picks the same value at equality, or a redundant early flush.
+  These are expected and left alone.
+
+Some real defects only surface under a sanitizer (e.g. a deleted bounds-flush
+that overflows a fixed buffer with adversarial input). Those belong to the
+ASan/UBSan fuzz lane (`./ci.sh --fuzz`), not to a deterministic assertion.