fix(ui): stop a legacy delegate generation from deleting or dropping rooms - #593
Conversation
72fcc48 to
1271688
Compare
Review record — two blind reviewers, four rounds, design changed twiceThe change that opened this PR is not the change in it. Recording why, because the two redesigns were both driven by findings I could not have reached alone. Round 1 — the guard was rank-blind (both reviewers, independently)My first fix guarded on "is the room in the map right now". Both reviewers showed that misses the likely interleaving: probes dispatch oldest-generation-first, so the oldest generation usually answers while the map is still empty, its stale tombstone lands unopposed, and the newest generation's Root cause one level down: a legacy generation's absences are older evidence exactly as its presences are, and only presences were ranked. Tombstones are now ranked observations too. Round 2 — the redesign introduced a new bugRanking tombstones created a path that never existed: a room can come back out of Fixed by extracting the survival rule as Round 3 — an in-session leave could be undone by a source that agreed with it
Notably, one reviewer had cleared this case as safe, then probed it and reported it as real, unprompted. Round 4 — two tests that could not prove their own claims
The sufficiency argument, in the codeThe filter answers ahead of a deferred merge, so an individual answer can be wrong either way. It is still sufficient, and the reason is now a comment rather than folklore: a tombstone rank only ever rises or is removed, and removal happens solely in the restore branch — which only a response carrying Mutation coverageEvery fix has a test that fails when the fix is removed: the eviction rank check; a newer Three of those were green on the first attempt and needed the test rewritten, not the fix — including one where my replacement was also vacuous. 829 river-ui tests, full workspace green. Deliberately not in this PRCollapsing [AI-assisted - Claude] |
1271688 to
d4c0eea
Compare
Round 5 — a third rank-blind spot, and this one loses private-room historyFound by the data-loss reviewer at The non-adopt branch merged
|
d4c0eea to
a0c0cd6
Compare
Two ways the legacy-migration merge loses rooms permanently. Both live on main; both found reviewing #587, which would have widened the first. ## A legacy generation's tombstone deletes a live room (#590) `migrate_legacy_per_room` pushes whatever slot it parses into `slots`, including `RoomSlot::Tombstone`; `reconstruct_rooms` turns those into `removed_rooms`; and `hydrate_loaded_rooms` passes the whole `Rooms` to `merge_from_source`, which unioned every incoming tombstone and then evicted the map against the combined set. So an OLDER generation's tombstone deleted a room the CURRENT delegate held `Present`, and `do_save_rooms_to_delegate`'s tombstone pass CAS-wrote that tombstone over the current slot. `self_sk` cannot be re-derived from the network. No failure is required: leave a room under generation G, take a WASM bump so G becomes legacy, rejoin the room, and any later load whose current-delegate index is empty fires the fan-out and destroys it. ### Root cause, and why the first fix was not enough The merge treated every responding generation as a PEER. `source_rank` existed but was consulted only for `self_sk` conflicts, so nothing constrained what an old snapshot could delete. The first attempt guarded on "is the room in the map right now", which review showed misses the LIKELY interleaving: probes dispatch oldest-generation-first, so the oldest generation usually answers while the map is still empty. Its stale tombstone lands unopposed, the newest generation's `Present` is then skipped by the tombstone check, and the re-save tombstones the room anyway. It also regressed the mirror case: between two legacy generations nothing asked WHICH source had put the room there, so a newer generation's leave was dropped. The actual rule is that a legacy generation's ABSENCES are older evidence exactly as its presences are, and only presences were ranked. Tombstones are now ranked observations too (`MergeRanks` carries both maps): - a tombstone evicts only if it outranks the copy currently held; - a `Present` from a strictly NEWER source clears an older generation's tombstone and restores the room; - an `Authoritative` source — the current delegate, or a deliberate in-session action — outranks every generation, so its removals are unchanged; - a room present with no recorded rank was created or imported in-session and nothing loaded may override it. The justification this replaces — the receiver's tombstone set is authoritative "because legacy delegates predate the tombstone field" (#247) — had outlived its premise: `legacy_delegates.toml` gains an entry on every WASM bump, so recent legacy generations carry per-room tombstones routinely. ## One room's merge failure drops all the rest (#591) Two bare `?` on the fallible per-room `ChatRoomStateV1::merge` returned from the whole function, so every room later in `other.map`'s arbitrary `HashMap` order was never inserted — and a room absent from the map is exactly where the #527 rank check does not run, so the next generation's copy was adopted unranked. Now per-room isolated: failures accumulate, the loop continues, and an aggregate is returned. Because `Err` now means "one or more rooms failed" rather than "nothing merged", `hydrate_loaded_rooms` runs `repopulate_secrets_from_state` and the actions_state rebuild regardless — skipping them left the rooms that DID merge rendering "[Encrypted message - secret vN not available]" until reload. ## Tests Six behavioural tests: a newer generation's `Present` overrides an older generation's tombstone (the probe-order case); an older generation's tombstone cannot evict a newer generation's room; a newer generation's tombstone still removes; a legacy tombstone cannot evict a room the live set holds; it still applies where the live set has none; an authoritative tombstone still removes. #591's test drives REAL merge failures (a configuration signed by a non-owner) and asserts BOTH are reported — deliberately order-independent, since `other.map` is a `HashMap` and any "a later room survived" assertion is a coin flip. An earlier version of that test asserted exactly that and passed under the bug. Mutation-checked, each turning a test red: the eviction rank check disabled; a newer `Present` unable to override an older tombstone; tombstones never recorded; the authority hard-coded at the hydrate call site; the per-room merge aborting. Closes #590 Closes #591 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DHubk7vg1mSjBQaoLVzs2h
a0c0cd6 to
6522528
Compare
The drain that carries a ranked resurrection from the merge to the save path had only a source-scan pin, and the pin's message claimed more than its assertion tested: `drain < mark` is textual order only, so calling `mark_room_rejoined` INSIDE the ranks closure satisfied it. That version re-locks a non-reentrant Mutex — on single-threaded WASM it hangs the tab rather than failing a test. - Strengthen the pin: require the collect-out-of-closure shape and place `mark_room_rejoined` after the closure has closed. The deadlock shape now fails it (verified by mutation; it passed before). - Add a runtime test of the boundary itself: merge an older generation's tombstone and a newer generation's Present through the SHARED registry, drain as the call site does, and assert the delegate write overwrites a stored Tombstone with Present. Dropping `ranks.resurrected.insert` turns it red. A second drain must not see the key again, so iterating without draining is caught too. - Repair a doc-comment splice: the #527 wiring-pin block had been severed mid-sentence by a test inserted into the middle of it. Reported by a review lens that re-ran the carry-over end-to-end rather than reading the diff. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DHubk7vg1mSjBQaoLVzs2h
The drain pin asserted the drain's position relative to the MARKING that consumes the resurrection set, never relative to `merge_from_source`, which fills it. Hoisting the drain to the top of the function — next to the `tombstoned` computation, which also takes the ranks lock, so it is a natural place to consolidate the two acquisitions — preserves every shape the pin asserts while draining an EMPTY set. Nothing is marked, `reconcile_room_present` returns to AbortAdoptLeave, and #590 is silently restored. Verified: that mutation compiles and left the suite green before this commit, and fails on the anchor after it. Same failure shape as the two earlier gaps in this PR: the assertion described the code's shape rather than the dependency that makes it work, so a refactor preserving the shape killed the behaviour. Also record, at the drain, that this widens `REJOINED_THIS_SESSION` from "the user deliberately did something" to also mean "the ranks concluded this room should come back" — so a WRONG resurrection is now persisted rather than session-local. Accepted deliberately (a wrongly-resurrected room is one the user leaves again; #590 destroys self_sk, which is unrecoverable), but the blast radius of a bad rank decision grew and the code did not say so. Reported by the data-loss review lens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DHubk7vg1mSjBQaoLVzs2h
Rounds 6–8 — three more gaps, all in the tests rather than the fixThe production change has not moved since Round 6 — Round 7 — the drain pin claimed more than it tested (code-first lens). Round 8 — the rewrite dropped a property while improving the pin (data-loss lens). Also recorded at the drain: this widens What these rounds were actually aboutThree of the four gaps were assertions pinning a shape where the property that mattered was a dependency between two pieces of code: a literal string, a call site's presence, an ordering between two tokens. Each survives a refactor that preserves the shape. The tests that held up drive the behaviour and assert the outcome. Round 8 is a distinct failure mode and the more interesting one: the property was lost because the revision was good. Fixing the re-entrancy hole meant rewriting the assertion, and the rewrite silently dropped something nobody was tracking as separate. The diff looks like strengthening and the suite stays green. The guard is to treat replacing an assertion as needing the same care as deleting one — enumerate what the old one proved, then check each item is still proved by something. Every mutation cited above was run, not argued. Two crossed on the wire: the round-8 anchor was pushed before the finding arrived, and the hoist took three attempts before it compiled — reporting "caught" from a mutation that never built would have been the same class of false result this PR kept producing. Lenses: code-first and data-loss, both blind, eight rounds. [AI-assisted - Claude] |
`cargo make sign-webapp` incremented the counter during the publish of main @ 01aac14 (#593). Committing it so the next publish starts from a strictly higher number — the contract enforces monotonicity. Published: contract raAqMhMG7KUpXBU2SxgCQ3Vh4PYjttxdSWd9ftV7RLv, version 30000374 -> 30000375. Verified served by both local peers (:7509 and :7530, distinct processes) and by try.freenet.org: river-ui-dxhea72f231ce20c24b.js -> river-ui-dxhba19fed1bd326284.js. No WASM changed: all three committed contract/delegate binaries are byte-identical before and after the build, so no migration entry is needed and the delegate key is unchanged. Claude-Session: https://claude.ai/code/session_01DHubk7vg1mSjBQaoLVzs2h Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
#588 route 2. `reconcile_room_present`'s diverged-identity arm kept `local` unconditionally and never consulted rank, while the merge path (#527, #590) does. So an identity loaded from an older delegate generation could be CAS-written over the current delegate's slot, destroying `self_sk` — which cannot be re-derived from the network. Reachable today: #592's `get_key_index` turns an unparseable index into an EMPTY one, while `handle_get_request` reads secrets directly and never consults the index. A room therefore goes invisible to `ListRequest` while staying readable, which routes the loader to the legacy probe (its only guard is a localStorage flag that always reads false in the sandboxed iframe) and ends in this overwrite. No code path blocks it; whether the index actually corrupts in the field is unestablished, so this is cheap insurance against an unrecoverable loss rather than a response to an observed incident. ## Why not "keep the higher-ranked identity" There is only ONE rank in existence at that call site. `MergeRanks` is never persisted and `RoomSlot::Present` carries no provenance, so there is nothing to compare the stored slot against. The decidable question is whether the copy WE hold is legacy-sourced; if it is, the stored slot was written by the current delegate, which outranks every legacy generation by construction, so refusing is always right. Refusing costs `room_state` (re-derivable). Writing costs `self_sk` (not). ## The default IS the fix An ABSENT rank entry does not mean "unknown". `record_identity_source` deliberately records nothing for an already-present unranked room, which is exactly how an in-session create/import looks — so absent must read as SOURCE_RANK_AUTHORITATIVE. `unwrap_or(0)` would make every locally created room overwritable by any legacy copy, strictly worse than no guard. Extracted into `identity_rank_for_save` and pinned, because the first mutation run showed `unwrap_or(0)` SURVIVING the behavioural tests: they call `reconcile_room_present` directly and never exercised the caller's default. Mutations verified red: unwrap_or(0); `<=` for `<`; comparison reversed; compared against SOURCE_RANK_AUTHORITATIVE rather than current_delegate_source_rank(); `ranks.tombstone` for `ranks.identity` (compiles — both are HashMap<RoomKey, u32>); guard deleted; helper ignoring the registry. Preserved, each with a test that fails if the guard over-reaches: #414 in-session identities still win; #420 multi-tab last-writer-wins is untouched; a legacy-sourced copy with a MATCHING identity still merges and persists; Absent/Tombstone paths never consult rank. Residual, NOT closed: a slot written by an older River build that itself held a legacy-sourced identity violates the premise, and detecting that needs persisted provenance. This closes route 2; it does not make the branch correct in general. Also drops an unused import introduced by #593. Design and test matrix specified by an independent review lens before implementation. Refs #588, #592 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DHubk7vg1mSjBQaoLVzs2h
Two ways the legacy-migration merge loses rooms permanently. Both are live on
main. Found reviewing #587, which would have widened the first.1. A legacy generation's tombstone deletes a live room (#590)
migrate_legacy_per_roompushes whatever slot it parses intoslots, includingRoomSlot::Tombstone.reconstruct_roomsturns those intoremoved_rooms.hydrate_loaded_roomspasses the wholeRoomstomerge_from_source, which unions the incoming tombstones and immediately evicts:So an older generation's tombstone deletes a room the current delegate holds
Present, anddo_save_rooms_to_delegate's tombstone pass then CAS-writes that tombstone over the current slot.self_skcannot be re-derived from the network — the room is gone for good.No failure is required. Leave a room under generation G; a WASM bump makes G legacy; rejoin the room under the new generation; any later load whose current-delegate index is empty fires the fan-out and destroys it.
Root cause
The merge treated every responding generation as a peer. It isn't — a legacy generation is a strictly older snapshot.
source_rankwas already threaded through, but consulted only forself_skconflicts, so nothing constrained what an old snapshot could delete.merge_from_sourcenow takes aMergeAuthority:Authoritative— the current delegate, or a deliberate in-session action. Unchanged: its tombstones remove rooms.OlderSnapshot— a legacy generation. Additive only. It may contribute a tombstone for a room the live set does not hold, so "a room the user left stays left" survives; never for one it does.The asymmetry is deliberate and one-directional. Worst case a left room reappears and the user leaves it again; the alternative is unrecoverable.
The justification this replaces — the receiver's tombstone set is authoritative "because legacy delegates predate the tombstone field" (#247) — had outlived its premise.
legacy_delegates.tomlgains an entry on every WASM bump, so recent legacy generations carry per-room tombstones routinely. That comment was written when "legacy" meant only the pre-tombstone blob generations.2. One room's merge failure drops all the rest (#591)
Two bare
?on the fallible per-roomChatRoomStateV1::mergereturned from the whole function, so every room later inother.map's iteration order was never inserted.other.mapis aHashMap, so which rooms are lost is arbitrary and unstable between runs, and the caller only logs.Worse than the immediate loss: a room absent from the map is exactly where the #527 rank check does not run —
merge_from_sourcecompares ranks only when the room is already present — so the next legacy generation's copy is adopted unranked, andreconcile_room_present's diverged-identity branch CAS-writes that older identity over the current one.Now per-room isolated: failures accumulate, the loop continues, and an aggregate is returned so nothing is swallowed. Same shape as
do_save_rooms_to_delegate, which accumulates per-key errors for exactly this reason.Testing
Three behavioural tests for #590 — a legacy tombstone cannot evict a room the live set holds; it still applies to a room the live set lacks; an authoritative tombstone still removes — plus a pin for #591's loop shape.
Mutation-checked, each turning a test red:
Authoritativeat the hydrate call site;The #527 wiring pin is extended to require the authority be derived from
is_legacy_delegate, never hard-coded.824 river-ui tests, full workspace green.
Scope
UI-only — no delegate/contract WASM,
Cargo.tomlorCargo.lockchange, so no migration entry is required.This is the first of the root-cause fixes for the family in #586. It addresses the destruction class. Convergence (the unbounded hoard across delegate generations, which is what actually causes the "Migrating your rooms…" banner) is separate and follows.
Closes #590
Closes #591
[AI-assisted - Claude]