[Performance] App hangs when OpenApp / ReconnectApp / GetMissingOnyxMessages fire in rapid stampedes

[mountiny]

If you haven't already, check out our contributing guidelines.

Version Number: Multiple production versions observed (e.g. 9.3.54–9.3.64)
Reproducible in staging?: Unknown
Reproducible in production?: Yes
Email or phone of affected tester (no customers): N/A — reproduced via production telemetry, not a single test account
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL: https://github.com/Expensify/Expensify/issues/644858
Issue reported by: Internal engineering investigation

What performance issue do we need to solve?

The JS thread is blocked for 10+ seconds (dead clicks / fully unresponsive UI) when large OpenApp, ReconnectApp, and GetMissingOnyxMessages responses arrive in rapid succession. Sentry traces show this drives a storm of Onyx updates and deep equality work across subscribers.

Symptom tracking (aggregate): https://expensify.sentry.io/issues/7323151341/ (~14k events / 90 days)

What is the impact on end-users?

• App appears frozen after navigation, reconnect, or opening reports/search/settings
• Affects iOS, Android, Windows, and Mac (not platform-specific)
• Both new and established accounts

Benchmarks / evidence (from production traces)

Payload sizes observed: ~0.5 MB – 4 MB per response on sessions that should be lightweight.

Pattern A — Same-endpoint stampede

One session fires the same command many times in minutes:

• 40× ReconnectApp in ~10 min (iOS 9.3.54-7, scan flow, every 8–30s) — https://expensify.sentry.io/explore/traces/trace/6c175d912b7d4bf5b238d131fdcd10a7
• 21× ReconnectApp in ~5 min (iOS, route change) — https://expensify.sentry.io/explore/traces/trace/688bf8372e4744f3b9fc57afb30eda21
• 13× GetMissingOnyxMessages in ~28s (iOS, Report page) — https://expensify.sentry.io/explore/traces/trace/61d5b4e4d45f4fea979548f86352c912
• 20× ReconnectApp in ~10s (Windows, ~146 KB each, ~2 calls/sec) — https://expensify.sentry.io/explore/traces/trace/298128ff3d274b70860b2060c9fc3581
• 27× ReconnectApp in ~9s (Windows, /settings/profile/contact-methods) — https://expensify.sentry.io/explore/traces/trace/e3d86d00ed3148bf85d85162c4218e57

Pattern B — Byte-identical duplicate responses (≤3s apart)

Server returns the same payload twice; client processes duplicate Onyx data:

• 2× ReconnectApp, 516,004 / 516,004 bytes identical, 1s apart (Windows) — https://expensify.sentry.io/explore/traces/trace/61bc0c35ab24458fac3e76592dbcc28b
• 2× OpenApp, ~1 MB each, ~90s apart (iOS) — https://expensify.sentry.io/explore/traces/trace/44c8c38f52a941f7bf6cd4549fb1bbfd
• 4× byte-identical ~224 KB ReconnectApp in 2s on /search (Windows) — https://expensify.sentry.io/explore/traces/trace/9d20e02d47d047aeaad3f0afcc82fa1d

Pattern C — Cross-command overlap (≤10s)

• GetMissingOnyxMessages 475 KB → ReconnectApp 475 KB, 1s apart (Mac, Report) — https://expensify.sentry.io/explore/traces/trace/1d72808fd7c8414e8b86bdc38d349ed1
• ReconnectApp → GetMissingOnyxMessages, byte-identical 250,893 B, 1s apart (Windows, Home) — https://expensify.sentry.io/explore/traces/trace/da50a56c2ec4443685eada0f86e486e2

Example hang timeline (iOS HybridApp 9.3.64-31)

1. OpenApp on Home — https://expensify.sentry.io/explore/traces/trace/940b5545caec4d5ba1ac8ce1c0b8738b
2. ReconnectApp on Onboarding_Purpose ~32s later — https://expensify.sentry.io/explore/traces/trace/0df88e7f01bc4a1ea048f491b87f021d
3. APP HANG ~52s after open — https://expensify.sentry.io/issues/7323151341/events/8677a76e805d4948bd37c2412b90e3fd/

Proposed investigation / solution direction

Open questions (from internal triage):

1. Why are payloads so large? — Possible whole-collection responses; correlate server output.size in logs for OpenApp / ReconnectApp.
2. Why are commands fired repeatedly? — Network flapping vs uncoordinated client callers (e.g. subscribeToFullReconnect.ts and other call sites).
3. Sentry ↔ server correlation — Confirm requestID still appears in HTTP breadcrumbs for recent sessions.
4. Delegate sessions — Check overlap with known GetMissingOnyxMessages over-fetch in delegate mode (PR in review internally).
5. Mitigation — Client in-flight dedup by updateIDFrom/updateIDTo may help, but root cause (why duplicate/large calls happen) is priority.

Sentry traces above are shareable; customer emails and FullStory remain on the internal issue only.

Platforms

• iOS: App
• Android: App (inferred from cross-platform scope)
• Windows: Chrome
• MacOS: Chrome / Safari

Issue Owner

Current Issue Owner: @adhorodyski

[MelvinBot]

Issue Analysis

Root cause (client side): The hang is amplified by two missing dedup guards in App — but the trigger (why duplicate/large responses are produced at all) is server side and needs an internal backend investigation.

1. Stampede amplifier (Pattern A) — no in-flight dedup. ReconnectApp/OpenApp/GetMissingOnyxMessages have ~6 uncoordinated trigger paths (network reachability, app-foreground, Pusher multi-event, auth-screen mount, subscribeToFullReconnect, delegate/priority flows) with no throttle, debounce, or in-flight guard. The only dedup — writeWithNoDuplicatesConflictAction → RequestConflictUtils.ts:57-73 — matches only against the queued requests returned by PersistedRequests.getAll(), which excludes the in-flight ongoingRequest. So while one ReconnectApp is mid-flight, every new trigger pushes a fresh copy that fires the instant the in-flight one returns → N calls per stampede.

2. Duplicate-apply amplifier (Patterns B & C) — large payloads reapplied in full. OnyxUpdates.apply() has an isUpdateOld guard that normally skips reapplying onyxData when lastUpdateID <= lastUpdateIDAppliedToClient — but it is explicitly bypassed for OpenApp and full ReconnectApp at OnyxUpdates.ts:133-136 (these commands also skip gap-detection via the bypass set in SaveResponseInOnyx.ts:12-18). Result: a byte-identical 0.5–4 MB duplicate OpenApp/full-ReconnectApp payload is written to Onyx again in full, firing subscriber callbacks + deep-equality work across every subscriber — the 10 s+ JS-thread block.

Recommended fix (App-side mitigation): These two guards are exactly the "client in-flight dedup" the issue calls out. They're implementable in App and will collapse most of the storm, but they are a mitigation, not the root cause — they reduce the cost of duplicate/large responses without stopping them from being produced.

• In-flight dedup: make the duplicate-conflict matcher also consult ongoingRequest (or add an in-flight no-op guard in reconnectApp/openApp) so a second identical command attaches to the in-flight one instead of queueing a fresh copy.
• Duplicate-payload guard: for OpenApp/full-ReconnectApp, skip reapplying onyxData when lastUpdateID hasn't advanced past lastUpdateIDAppliedToClient (revisit the bypass at OnyxUpdates.ts:133-134) so byte-identical duplicates don't re-storm Onyx.

Needs internal backend investigation (outside the App repo): The priority questions from your triage — (1) why payloads are 0.5–4 MB (whole-collection responses?) and (2) why the server returns byte-identical duplicates ≤3 s apart — live in Auth/Web-Expensify and can't be diagnosed or fixed from App. These should be escalated to an internal engineer with backend access; correlating server output.size and requestID against the Sentry traces is the next step there.

Investigation details & evidence

In-flight dedup gap (Pattern A):

• reconnectApp() and openApp() have no in-flight flag/Promise guard; only conditionals are hasLoadedApp (init) and isUsingImportedState. getMissingOnyxUpdates() uses makeRequestWithSideEffects with no conflict checker at all.
• Trigger paths with no dedup: Reconnect.ts (network-restored and app-foreground), User.ts (Pusher multi-event), AuthScreensInitHandler.tsx (per-mount), subscribeToFullReconnect.ts.
• Partial flap protection exists only on the sustained-failure path: a 0–5 s RECONNECT_STAMPEDE_JITTER_MS jitter in NetworkState.ts:127-147, whose comment even asserts "Duplicate reconnectApp() calls are safe — SQ deduplicates them" — which is false for the in-flight case because of the getAll()/ongoingRequest gap above.
• The flow: first call → queue empty → push → processNextRequest moves it to ongoingRequest, leaving persistedRequests empty → next call sees empty queue → pushes another fresh copy → repeats.

Duplicate-apply gap (Patterns B & C):

• The isUpdateOld guard (OnyxUpdates.ts:132-159) strips onyxData for ordinary duplicates but exempts OpenApp and full ReconnectApp (no updateIDFrom). When not exempted-but-old, only successData/failureData are reapplied; when exempted, the full onyxData is handed to Onyx.update(...).
• There is no content-hash / requestID / "already applied" dedup anywhere in this pipeline — the only dedup key is lastUpdateID, and it's intentionally disabled for the two biggest-payload commands.
• GetMissingOnyxMessages / partial ReconnectApp duplicates can be caught by the guard, but only if the async lastUpdateIDAppliedToClient cache (OnyxUpdates.ts:17-23) has propagated before the duplicate is processed — a race when two land ≤3 s apart.

Alternative explanations considered: Could the hang be pure render cost from one large legitimate payload (no duplication)? Partly — a single 4 MB OpenApp is expensive regardless. But that doesn't explain the repeated traces showing byte-identical duplicates and 20–40× same-command bursts, which the two gaps above directly account for. The deep-equality "storm across subscribers" in the Sentry traces is consistent with full-payload reapplication, not a one-shot apply.

Confidence: High that both client gaps exist and amplify the hang (verified at file:line). Medium on attribution — whether these client amplifiers vs. the server over-fetch/duplication are the dominant driver needs the log correlation below.

Logs that would confirm dominance (internal): count, per affected session, [OnyxUpdateManager] Applying update type: https with lastUpdateID: <N> (OnyxUpdates.ts:130) for the same lastUpdateID within 3 s broken out by command; and [SequentialQueue] Conflict action determined with command: ReconnectApp and conflictType: push vs replace. Correlate against server ReconnectApp/OpenApp output.size and requestID.

---

Next Steps: Reply with @MelvinBot implement this to draft a PR for the two App-side mitigations (in-flight dedup + duplicate-payload guard), or @MelvinBot <feedback> to refine. The backend over-fetch/duplication questions (priority root cause) need an internal engineer with Auth/Web-Expensify access.