Skip to content

Every graph change synchronously writes 271 KB of mail bodies to disk, holding the GraphStore actor #307

Description

@scgopi

Noticed as an aside by the MailConcurrent end-to-end loop (node memo taking 0.16–2.13 s, 20–100× an inbox read, present with and without a wedged client so not #288), then measured and root-caused here on the shipped 0.1.64-beta2.

The shape is #288 again, one layer over

GraphStore.broadcast():

private func broadcast() {
  onGraphChanged?(graph)   // ← synchronous
  notifyClients()
}

ProjectRegistry wires that to persistence.saveGraph(updatedGraph) — a synchronous disk write of the entire graph — and broadcast() runs on the GraphStore actor. So every graph mutation holds the actor across a full serialize-and-write to disk, exactly as it used to hold the actor across a socket write before #291.

The payload is what #293 was supposed to remove

#293 took the Mailroom out of the wire snapshot. It is still in the persisted one:

bytes
persisted project JSON 323,603
of which mailroom 271,256 (330 posts)
share 84%
wire graphChanged frame, for contrast 56,604 (mailroom = 68-byte digest)

So the broadcast got 77% smaller and the disk write did not move at all. Every memo, every post, every persisted state change re-serializes 271 KB of mail bodies that did not change.

Measured cost

graphcode node memo, three consecutive calls on an otherwise quiet daemon:

real 0.03
real 0.77
real 0.87

MailConcurrent saw up to 2.13 s under load. A socket-level openProject round trip on the same daemon is 0.003 s, so a memo costs 10–700× a read, and the variance points at the filesystem rather than at compute.

Every loop writes memos — it is how a pass leaves anything behind for the next one — so this is on a hot path for the whole graph, and it gets slower as the room fills.

Two independent fixes, either helps

  1. Do not hold the actor across the write. onGraphChanged should hand the snapshot to a serial writer and return, the way OutboundChannel now takes frames. Coalescing consecutive saves would also collapse a burst of memos into one write.
  2. Stop persisting the room inside the project graph, or persist it separately so a node change does not rewrite every post. Serve the Mailroom from a bounded mailbox request, not every broadcast (#288) #293 established that posts do not belong in the snapshot clients receive; the same argument applies to the file rewritten on every mutation.

The first is the direct analogue of #291 and fixes the stall regardless of payload size. The second is the analogue of #293 and removes 84% of the bytes.

Why it was not caught

Every measurement in the #288 series was taken at the socket. The disk half was never instrumented, and PR #303's diagnostics would time persistence as a phase — which is precisely how this would have surfaced on its own.

Related: #288, #291, #293, #289

🤖 Generated with Claude Code

https://claude.ai/code/session_01BP43ags4cn8fq2ZZdv85J9

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions