You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Noticed as an aside by the MailConcurrent end-to-end loop (node memo taking 0.16–2.13 s, 20–100× an inbox read, present with and without a wedged client so not#288), then measured and root-caused here on the shipped 0.1.64-beta2.
ProjectRegistry wires that to persistence.saveGraph(updatedGraph) — a synchronous disk write of the entire graph — and broadcast() runs on the GraphStore actor. So every graph mutation holds the actor across a full serialize-and-write to disk, exactly as it used to hold the actor across a socket write before #291.
#293 took the Mailroom out of the wire snapshot. It is still in the persisted one:
bytes
persisted project JSON
323,603
of which mailroom
271,256 (330 posts)
share
84%
wire graphChanged frame, for contrast
56,604 (mailroom = 68-byte digest)
So the broadcast got 77% smaller and the disk write did not move at all. Every memo, every post, every persisted state change re-serializes 271 KB of mail bodies that did not change.
Measured cost
graphcode node memo, three consecutive calls on an otherwise quiet daemon:
real 0.03
real 0.77
real 0.87
MailConcurrent saw up to 2.13 s under load. A socket-level openProject round trip on the same daemon is 0.003 s, so a memo costs 10–700× a read, and the variance points at the filesystem rather than at compute.
Every loop writes memos — it is how a pass leaves anything behind for the next one — so this is on a hot path for the whole graph, and it gets slower as the room fills.
Two independent fixes, either helps
Do not hold the actor across the write.onGraphChanged should hand the snapshot to a serial writer and return, the way OutboundChannel now takes frames. Coalescing consecutive saves would also collapse a burst of memos into one write.
Stop persisting the room inside the project graph, or persist it separately so a node change does not rewrite every post. Serve the Mailroom from a bounded mailbox request, not every broadcast (#288) #293 established that posts do not belong in the snapshot clients receive; the same argument applies to the file rewritten on every mutation.
The first is the direct analogue of #291 and fixes the stall regardless of payload size. The second is the analogue of #293 and removes 84% of the bytes.
Why it was not caught
Every measurement in the #288 series was taken at the socket. The disk half was never instrumented, and PR #303's diagnostics would time persistence as a phase — which is precisely how this would have surfaced on its own.
Noticed as an aside by the
MailConcurrentend-to-end loop (node memotaking 0.16–2.13 s, 20–100× an inbox read, present with and without a wedged client so not #288), then measured and root-caused here on the shipped 0.1.64-beta2.The shape is #288 again, one layer over
GraphStore.broadcast():ProjectRegistrywires that topersistence.saveGraph(updatedGraph)— a synchronous disk write of the entire graph — andbroadcast()runs on theGraphStoreactor. So every graph mutation holds the actor across a full serialize-and-write to disk, exactly as it used to hold the actor across a socket write before #291.The payload is what #293 was supposed to remove
#293 took the Mailroom out of the wire snapshot. It is still in the persisted one:
mailroomgraphChangedframe, for contrastSo the broadcast got 77% smaller and the disk write did not move at all. Every memo, every post, every persisted state change re-serializes 271 KB of mail bodies that did not change.
Measured cost
graphcode node memo, three consecutive calls on an otherwise quiet daemon:MailConcurrentsaw up to 2.13 s under load. A socket-levelopenProjectround trip on the same daemon is 0.003 s, so a memo costs 10–700× a read, and the variance points at the filesystem rather than at compute.Every loop writes memos — it is how a pass leaves anything behind for the next one — so this is on a hot path for the whole graph, and it gets slower as the room fills.
Two independent fixes, either helps
onGraphChangedshould hand the snapshot to a serial writer and return, the wayOutboundChannelnow takes frames. Coalescing consecutive saves would also collapse a burst of memos into one write.The first is the direct analogue of #291 and fixes the stall regardless of payload size. The second is the analogue of #293 and removes 84% of the bytes.
Why it was not caught
Every measurement in the #288 series was taken at the socket. The disk half was never instrumented, and PR #303's diagnostics would time
persistenceas a phase — which is precisely how this would have surfaced on its own.Related: #288, #291, #293, #289
🤖 Generated with Claude Code
https://claude.ai/code/session_01BP43ags4cn8fq2ZZdv85J9