Skip to content

resync on broadcast lag and map mqdb conflicts to typed error - #1

Merged
fabracht merged 5 commits into
mainfrom
stitch-cas-lag-fixes
May 29, 2026
Merged

resync on broadcast lag and map mqdb conflicts to typed error#1
fabracht merged 5 commits into
mainfrom
stitch-cas-lag-fixes

Conversation

@fabracht

@fabracht fabracht commented May 29, 2026

Copy link
Copy Markdown
Contributor

Context

Follow-up to the chorale AWS scale test (10×200 tasks) which surfaced liveness: FAIL against the embedded mqdb backend. Root cause split across two crates:

  • mqdb side (already shipped in mqdb-agent 0.8.8 / mqdb-core 0.7.3): Database::update/delete CAS conflicts could stall single-process embedded callers and silently drop writes on the fjall backend. mqdb added bounded internal retry-on-conflict (32 attempts, field-level last-writer-wins) and an atomic fjall commit. No stitch change is required for the stall itself — this PR is the stitch-side hardening + the remaining liveness gap.
  • stitch side (this PR).

Changes

  • Typed conflict mapping (memory_store.rs): map mqdb Error::Conflict → stitch's existing Error::Conflict { entity, id } on the two CAS paths (update, delete), so the residual backstop conflicts (mqdb cap-exhaustion / vault path) are classifiable via is_conflict() instead of falling into an opaque Mqdb error.
  • Resync on broadcast lag (store.rs, config.rs): the internal mutation_loop previously dropped inbound remote deliveries on RecvError::Lagged and continued, causing local divergence + missed scheduler wakeups under high throughput (Finding 2). It now runs an authoritative sync_root_entity_list re-fetch when connected. Default event_channel_capacity raised 1024 → 4096 for high-write scopes.
  • No more blanket silent drops (store.rs): handle_remote_mutation replaced .unwrap_or_default() and the let _ = cache-mirror writes with warn-level logging on genuine failures; intentional LWW/scope drops still return Ok(false) quietly.
  • Regression test (tests/memory_store.rs): concurrent_updates_to_same_record_converge_without_conflict — 16 concurrent same-key updates on a multi-thread runtime must all return Ok, converge to a last-writer value, and preserve the untouched field (field-level merge).
  • Cargo.lock: mqdb-agent 0.8.8 / mqdb-core 0.7.3.

Notes

  • Resync is inline in mutation_loop (the background loops hold strong Arc<StoreInner> refs, so spawning would add an Arc cycle) — natural backpressure, nothing new to track.
  • sync_root_entity_list recovers root + child entities for the connected user, matching existing reconnect coverage. Top-level wildcard entities aren't re-fetched here — a pre-existing reconnect gap, not introduced by this PR.

Verification

  • cargo clippy -p stitch --all-targets clean
  • cargo test -p stitch green (91 tests, incl. the new concurrency test)
  • End-to-end chorale scale repro (liveness: PASS) needs the AWS Fargate harness and was not run locally.

@fabracht
fabracht merged commit 5169cce into main May 29, 2026
@fabracht
fabracht deleted the stitch-cas-lag-fixes branch May 29, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant