Skip to content

ci: run chat-delegate and web-container tests, and guard the gap class - #615

Merged
sanity merged 2 commits into
mainfrom
ci/chat-delegate-tests
Aug 9, 2026
Merged

ci: run chat-delegate and web-container tests, and guard the gap class#615
sanity merged 2 commits into
mainfrom
ci/chat-delegate-tests

Conversation

@sanity

@sanity sanity commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Problem

The chat-delegate crate has 40 unit tests that have never executed in CI
(the issue estimated ~39; the exact count is 40 — 13 in handlers.rs, 15 in
subscription/tests.rs, 12 in versioning.rs).

Makefile.toml defines test-chat-delegate and rolls it into cargo make test, but no workflow ever invoked it. build.yml has explicit steps for
room-contract, river-core, riverctl and river-ui; the delegate was
simply never added. The only workflows mentioning chat-delegate are
check-cli-wasm.yml and check-delegate-migration.yml, and neither runs
cargo test — the latter only does a WASM byte-hash comparison.

A test that no CI job runs is indistinguishable from a test that passes. That
is bad anywhere, and worse here: this is the crate where River's secret storage
and its delegate-re-key migration path live, and the delegate is re-keyed
roughly weekly (#612). A silent regression there destroys user identity.

Found while fixing it, same bug, different crate: web-container-contract
(6 lib + 2 integration tests) and web-container-tool (1 test) were also
never run by any workflow. That includes
test_tool_and_contract_compatibility, which pins that the signing tool and
the verifying contract agree on the parameter encoding.

Verified before changing anything, per the issue's request: the premise held in
full. No subset of the delegate's tests was running.

Approach

Three changes to .github/workflows/build.yml:

  1. cargo test -p chat-delegate — a step alongside the existing per-crate
    ones.
  2. cargo test -p web-container-contract -p web-container-tool — the same
    gap, found while fixing the first.
  3. scripts/check-ci-test-coverage.sh — a guard that fails CI when any
    [workspace] member has no cargo test -p <name> step in build.yml.

On the toolchain question the issue raised: no --target x86_64-unknown-linux-gnu pin is needed, unlike the Makefile.toml task. The
pin there pairs with --target-dir target/native so a developer's local host
build cannot clobber the wasm artifacts the UI pulls in via include_bytes!.
CI has no such conflict — cargo make build has already produced the wasm, and
every existing test step likewise builds for the host into the same
CARGO_TARGET_DIR. The crate's default target is wasm only by convention of
how it is built, not by a [build] target setting, so a bare host cargo test
is correct. room-contract, also a wasm cdylib, is already wired exactly this
way.

On the guard (point 3), which is the part not strictly asked for. The
per-crate steps in build.yml carry comments recording four previous instances
of this same gap: river-ui's unit tests, riverctl's unit tests, the bulk of
river-core's lib tests, and the nine common/tests files that an allowlist of
--test <name> steps silently skipped. This issue is the fifth and sixth. Each
previous fix added the missing step, which fixes the instance and not the
class. The script reads [workspace] members directly and resolves each
member's package name from its own Cargo.toml, so adding a crate without
wiring its tests now fails CI. It is plain bash with no jq/python
dependency. Happy to drop it if reviewers consider it out of scope.

Testing

No test rot: all 40 delegate tests passed unmodified on first run, as did all
9 web-container tests. Nothing was deleted, skipped, or #[ignore]d — the
before/after count of passing tests is 40 → 40 and 9 → 9. What changed is that
they now gate merges.

Mutation evidence — the new delegate step can actually fail

Expectation stated before running: inverting one assertion in each of the three
test modules should produce exactly three failures and a non-zero exit, proving
all three modules genuinely execute rather than just one.

Mutations applied (handlers.rs:832 assert_eq!(result.len(), 1)99;
versioning.rs:164 assert_eq!(gen, 13)14; subscription/tests.rs:192
assert_eq!(m1, m2)assert_ne!), then the exact command CI runs:

$ cargo test -p chat-delegate
thread 'handlers::tests::test_list_request' panicked at delegates/chat-delegate/src/handlers.rs:832:9:
assertion `left == right` failed
  left: 1
 right: 99

thread 'subscription::tests::does_not_rotate_when_member_set_unchanged' panicked at delegates/chat-delegate/src/subscription/tests.rs:192:5:
assertion `left != right` failed
  left: {MemberId(CUS7CJMI), MemberId(4FVVXL7J)}
 right: {MemberId(CUS7CJMI), MemberId(4FVVXL7J)}

thread 'versioning::tests::encode_decode_round_trip' panicked at delegates/chat-delegate/src/versioning.rs:164:9:
assertion `left == right` failed
  left: 13
 right: 14

failures:
    handlers::tests::test_list_request
    subscription::tests::does_not_rotate_when_member_set_unchanged
    versioning::tests::encode_decode_round_trip

test result: FAILED. 37 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out
error: test failed, to rerun pass `-p chat-delegate --lib`
$ echo $?   # 101

All mutations reverted; the tree is clean against the baseline.

Mutation evidence — the guard can actually fail

Removing the cargo test -p chat-delegate line from build.yml:

$ ./scripts/check-ci-test-coverage.sh
ok:      river-core (common)
ok:      river-ui (ui)
ok:      riverctl (cli)
ok:      room-contract (contracts/room-contract)
ok:      web-container-contract (contracts/web-container-contract)
ok:      web-container-tool (contracts/web-container-contract/web-container-tool)
MISSING: chat-delegate (delegates/chat-delegate) has no 'cargo test -p chat-delegate' step in build.yml
$ echo $?   # 1

Unmutated, it reports all 7 members ok and exits 0.

Scope caveat — what these 40 tests do NOT cover

Worth recording so the new green checkmark is not over-read. On native targets
DelegateCtx's set_secret is a no-op and get_secret always returns None
(freenet-stdlib's stub; see the cfg at handlers.rs:190). So these tests
cover dispatch and pure-value logic, not storage round-trips.

Measured, rather than assumed: I removed the set_key_index(...) call from
handle_store_request — a genuine #612-class regression that would stop every
room from migrating — and all 40 delegate tests still passed. The
source-scrape pin added by #613 caught it:

$ cargo test -p river-ui --bins only_the_indexed_delegate_paths
test components::app::chat_delegate::tests::only_the_indexed_delegate_paths_register_keys_for_migration ... FAILED
panicked at ui/src/components/app/chat_delegate.rs:1494:13:
fn handle_store_request( must add its key to the delegate's key_index — that index is what the
migration probe's ListRequest enumerates, so a write that skips it is a write that never migrates
(freenet/river#612)
test result: FAILED. 0 passed; 1 failed

Follow-up, deliberately not done here (issue #614 point 6): that pin's own
comment says to move it into the delegate crate "once they are [wired into
CI]". This PR makes that possible. But the evidence above shows it must stay a
source-scrape when it moves — the behavioral version is still not
expressible against a stub ctx that never stores anything. Moving it also needs
care about self-matching: as an in-file pin its needles would appear in its own
assertion strings, so the existing body-slicing helper has to stay. Clean
separate PR.

Closes #614
Refs #612, #613, freenet/freenet-core#2776

[AI-assisted - Claude]

sanity added 2 commits August 9, 2026 14:27
The chat-delegate's 40 unit tests never ran in CI. Makefile.toml defines
test-chat-delegate and rolls it into `cargo make test`, but no workflow
invoked it, so the tests gated only a developer's local run. Same for
web-container-contract + web-container-tool (9 tests).

Adds a cargo test step for each, plus scripts/check-ci-test-coverage.sh,
which fails CI when any [workspace] member has no `cargo test -p <name>`
step in build.yml. That converts a recurring per-crate oversight into a
CI failure.

Closes #614
Refs #612, freenet/freenet-core#2776

[AI-assisted - Claude]
… the real migration gate

Three review fixes:

1. The guard's header cited five prior incidents as the class it fixes, but
   two of them (river-core's bulk lib tests, the nine common/tests allowlist
   files) happened while river-core DID have -p test steps — a configuration
   the guard passes. It checks presence, not adequacy. Says so now, since a
   guard overclaiming its coverage is the failure mode this PR is about.

2. `\s` is GNU-only, so the script failed spuriously on macOS/BSD. Uses
   [[:space:]] throughout.

3. The build.yml caveat said what the delegate tests don't cover but not
   where that coverage lives. Now points at the #613 pin in
   `cargo test -p river-ui --bins`, and cites handlers.rs:191-198 (191 is
   the cfg arm; 190 was the prose comment).

Also parses `members = [...]` by accumulating to the closing bracket, so a
single-line reformat is handled instead of running to EOF and reporting
"member '2' has no Cargo.toml" (it had picked up `resolver = "2"`).

[AI-assisted - Claude]
@sanity

sanity commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Review fixes pushed in 7af7aef8. All three addressed, plus both optionals.

1 (SHOULD-FIX) — the guard's header overclaimed. Fair catch, and the
recursion is not lost on me. It now states the limit explicitly:

SCOPE — this catches the ZERO-STEP class only [...] Two of those five
incidents (river-core's bulk lib tests, the nine common/tests files behind a
--test <name> allowlist) happened while river-core DID have
cargo test -p river-core --test X steps — a configuration this script
passes. It checks that a member has SOME test step, not that the step is
ADEQUATE [...] Guarding that class needs a catch-all step instead — see the
cargo test -p river-core --tests comment in build.yml.

2 — GNU-only \s. Replaced with [[:space:]] at both sites. The only
remaining \s in the file is inside the comment explaining why it is not used.

3 — where the coverage actually lives. The build.yml caveat now names the
gate to trust instead:

The migration behaviour is gated elsewhere: the #613 source-scrape pin
only_the_indexed_delegate_paths_register_keys_for_migration, which runs in
the cargo test -p river-ui --bins step above and does catch that deletion.
Trust that step, not this one, for #612.

It also records the measurement rather than just the conclusion (deleting
set_key_index(...) leaves all 40 delegate tests passing).

Both optionals, taken. The cfg citation is now handlers.rs:191-198 (190
was the prose comment). And members = [...] is parsed by accumulating to the
closing bracket rather than by line range, so the single-line reformat is
handled correctly instead of running to EOF and picking up resolver = "2".

Re-verified after the changes — the four guard mutations you ran, against
the revised script:

Mutation Result
Remove the cargo test -p chat-delegate step exit=1, 1 MISSING
Add a dummy delegates/brand-new-crate member exit=1, ERROR: workspace member 'delegates/brand-new-crate' has no Cargo.toml
Replace every step with cargo test --workspace exit=1, all 7 MISSING
Reformat members onto a single line exit=0, parses correctly — previously the confusing failure

The fourth now passes rather than failing closed, which is the intended
improvement: a reformat is a legitimate edit and the guard should keep working
across it. It still fails closed on every mutation that actually removes
coverage. shellcheck clean; YAML re-validated (3 jobs, 25 build steps).

One process note worth recording: two of my git checkout -- restores during
that mutation re-run silently reverted fix 3, which was still uncommitted.
Caught it by re-grepping for the new text rather than trusting the edit had
stuck. Same trap as the archive-pin loss noted in
.claude/rules/direct-messages.md — mutation testing and uncommitted work do
not mix.

[AI-assisted - Claude]

@sanity
sanity merged commit 4b4916b into main Aug 9, 2026
6 checks passed
@sanity
sanity deleted the ci/chat-delegate-tests branch August 9, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(ci): chat-delegate unit tests never run in CI

1 participant