Room contract merge can exceed core's 5s execution budget on large rooms — contract stops converging network-wide

[sanity]

Problem

Production forensics on the nova gateway (freenet-core#4861) found a contract instance — behaviorally consistent with a River room (~154KB state, room-contract state/summary size profile) — whose every update_state merge exceeds freenet-core's 5-second WASM execution budget:

• WASM execution timed out with elapsed_ms consistently 5001–5008, 100–220 times/hour, sustained 36h+
• State frozen at 154,375 bytes: only 2 successful applies in 36h, 2,656 Merge rejected incoming state but local state is valid
• Identical behavior on two independent peers (v0.2.100 and v0.2.101), so contract-intrinsic
• Cost: ≈14–29% of one CPU core, continuously, on every peer hosting the contract (it is the only contract producing timeouts on the gateway — all-timeouts ≈ its timeouts every hour)

Contract instance key: Gsd2311PU4PFrFvptNToGjxswUKbMeouZF6sfjV3CVgQ. The room identification is a high-confidence hypothesis from behavior/sizes (the contract times out before it can emit identifying log lines); xqyyifd4… (confirmed room via its contract-emitted error) shows the same rejection signature at smaller cost.

freenet-core#4864 contains the core-side cost (enforced preemption + backoff), but the contract remains permanently unable to converge.

Ask

Profile room-contract update_state/validate on a ~150KB room state (many members/messages): what is O(·) in members × messages, and is per-delta signature re-verification over the whole state the driver? A merge that cannot complete within ~5s of CPU on commodity hardware means large rooms stop converging network-wide. Likely needs incremental validation (verify only the delta's additions) or a bounded merge.

Related: freenet/freenet-core#4861 (root-cause forensics), freenet/freenet-core#4864 (core containment).

[AI-assisted - Claude]

[sanity]

Issue Priority Assessment

Generated: 2026-07-24T12:44:58.837499214+00:00
Issue hash: da6b22ce

Value Assessment

No PR references issue #422 — it's unaddressed. Enough evidence gathered to write the assessment.

Assessment

Relevance: Still fully applicable to the current codebase. The room contract's update_state (contracts/room-contract/src/lib.rs:127) clones the entire chat_state on every UpdateData::State/Delta item processed in a batch (chat_state.clone() at lines ~155/162), and the top-level ChatRoomStateV1::post_apply_cleanup (common/src/room_state.rs:101) runs multiple full sweeps over members/bans/messages — including a sort_by_cached_key ban ordering pass and signature-sweep passes documented as running "AFTER enforcement" on the whole set — every single time a delta or state update is applied. While per-field apply_delta implementations (e.g. MessagesV1::apply_delta) are correctly delta-scoped and don't re-verify old signatures, the cleanup/merge machinery around them is not: it's O(members × bans) or worse per update, and a batch of N updates multiplies that by N full-state clones and cleanup passes. This is consistent with — and helps explain — the timeout behavior described in the issue. No PR or commit references #422, and nothing in the recent git log (delegate migration, DM features, WASM size work) touches this path, so the problem is unfixed.

User impact: Currently narrow but existential for affected rooms — this is a small number of large/active rooms (the forensics found essentially one instance on the nova gateway, plus a second smaller case), but for those rooms the impact is total: the contract stops converging network-wide, permanently, until either state shrinks or the contract logic changes. Every peer hosting that contract also burns 14-29% of a CPU core continuously, which is a resource-abuse vector as network usage grows — any actor (or organically-growing popular room) can push a room past this threshold and knock it out cross-network.

Strategic alignment: Directly threatens River's core promise — durable, converging, decentralized room state. It also interacts with the freenet-core hosting-invariants redesign (hosting-invariants.md): as rooms accumulate more subscribers/messages under demand-driven hosting, larger, more active rooms are exactly the ones expected to persist and gain hosts, making this a scaling cliff for the app's success cases, not just an edge case.

Blocking factor: Doesn't block current feature PRs, but it does block River's ability to safely scale room size/membership limits, and it undermines freenet-core's new preemption/backoff containment in #4864, which mitigates the symptom (CPU cost) but not the root cause (permanent non-convergence). Any future room-contract change that adds more per-update O(n) work (e.g. more cleanup passes) will compound this.

Urgency: Elevated but not "drop everything" — the core-side timeout containment (#4864) already prevents this from being a live-node stability emergency, and only one or two contract instances are currently confirmed affected. However, because the failure is silent (rooms just quietly stop converging, with only backend-forensics visibility) and the cost scales with room popularity, this should be picked up before the next round of scale/member-limit increases, and is a good candidate for near-term profiling work given the concrete repro case (Gsd2311PU4PFrFvptNToGjxswUKbMeouZF6sfjV3CVgQ) already exists.

Cost Assessment

Based on exploring the codebase, here's my cost assessment for issue #422:

Relevance. The issue is still fully actionable — nothing has moved this contract-state code toward addressing it. The ComposableState impls for MessagesV1, MembersV1, MemberInfoV1, BansV1, DirectMessagesV1, SecretsV1 in common/src/room_state/ are unchanged in structure since the design that produced the reported behavior, and there's a concrete, plausible O(n×m) culprit: MemberInfoV1::apply_delta (common/src/room_state/member_info.rs:246-345) calls parent_state.members.members_by_member_id() inside the per-delta-entry loop (line 289) rather than hoisting it out, and separately does an O(n) linear .find() scan (lines 305-308) per delta entry against the existing member_info list. For a 154KB room with hundreds of members and a delta touching many of them, this is quadratic — a strong match for "identical, deterministic 5s+ timeout on every merge, contract-intrinsic." Full-state signature reverification in verify() is O(n) but only fires at PUT/resync boundaries, not per-delta, so it's a secondary contributor at most, not the primary driver the issue speculates about.

Scope. Narrow. The fix is isolated to one or two functions in member_info.rs (hoist the HashMap build out of the loop; replace the linear .find() with an indexed lookup, possibly requiring member_info to be stored/accessed via a map rather than a Vec internally). The same quadratic-loop pattern should be checked in ban.rs, secret.rs, and direct_messages.rs apply_delta bodies (not yet fully inspected) since they likely share the same "lookup against parent state inside a delta loop" shape — that's the main scope uncertainty. This is a targeted patch to a handful of files, not a subsystem redesign.

Complexity. Low-to-moderate. The core change (loop-invariant hoisting, Vec→HashMap lookup) is mechanical Rust, not conceptually hard. The tricky part is preserving the exact serialization/determinism guarantees this codebase is strict about (per contract-summary-determinism.md and the backwards-compatibility signature rules) — any storage-shape change to member_info internals must not alter wire bytes or signature-covered byte layout, which constrains the fix to changing access patterns, not the stored Vec shape itself (to avoid touching signed bytes). That's a real but well-understood constraint given the existing rules already documented in this repo.

External dependencies. None — no third-party signups, approvals, or waiting on other people. This is pure code + profiling work, doable locally.

Risk. Low-moderate. apply_delta/verify/merge are consensus-critical (state authorization, backward compatibility, migration) — a mistake here could cause divergence or reject valid deltas, which the repo's own invariants treat very seriously (contract migration, summary determinism). But the actual fix (avoiding a redundant hashmap rebuild, replacing a linear scan) is low-risk if done carefully, since it doesn't change the logic of what's accepted, only the performance of accepting it. The main regression risk is subtle: getting an indexed-lookup refactor wrong could silently change which entry gets updated/overwritten in edge cases (duplicate member IDs, stale entries) — needs careful testing.

Testing effort. Low, and this is one of the report's strengths: no criterion/bench infra exists yet, but a straightforward unit test with a synthetic large state (e.g., 500 members, 5000 messages, ~150KB) using existing test helpers plus std::time::Instant timing assertions would both reproduce the slow path and pin the fix. This should be added in the same PR per the repo's testing philosophy (every behavioral/perf fix needs a regression test). Given no profiling was done before filing, the first real step should be reproducing the timeout locally with a synthetic large room state to confirm the member_info hypothesis before patching — that reproduction step is itself part of the cost, maybe half a day, with the actual fix + tests likely another half-day to day once the bottleneck is confirmed.

Overall: small-to-medium cost — a few hours of profiling/reproduction plus a targeted, well-scoped patch (likely under 100 lines changed) with straightforward test coverage, assuming the member_info hypothesis holds after profiling. If profiling reveals the bottleneck is elsewhere (e.g., in message.rs's larger dataset) the fix shape would be similar (loop-invariant hoisting / indexed lookups), so the cost estimate is fairly robust to that uncertainty.

---

Generated by issue-prioritizer