metadata-fs watcher tests are merge-queue flaky, and #7208's 20s deadline hardening did NOT fix it — the event is suppressed, not late

[os-zhuang]

Filed from PR #7261 (an unrelated packages/objectql test-only change) after a merge-queue ejection.
Unassigned — recording, not claiming.

The signature

FAIL test/fs-behavior.test.ts > FileSystemRepository — on-disk semantics
     > chokidar: external file change emits an update event

Confirmed flaky per the triage bot's checklist item 2 — the same signature has now ejected two
different PRs, one of which does not touch packages/metadata-fs at all:

when (UTC)	PR	queue run	signature
03:16	#7208	31351599963	fs-behavior.test.ts › chokidar external edit
03:42	#7208	31352802812	watch-dot-root.test.ts › dot-rooted external edit
04:33	#7261	31355694872	fs-behavior.test.ts › chokidar external edit

The part that matters: the hardening was already in place

#7208's third-attempt patch (3a9f5b880, merged as 684ab22 at 04:12:38) raised every positive
watcher wait to EVENT_WAIT_MS = 20_000, and its own comment set a standing rule: "if this attempt
ejects again on a watcher-timing signature… the flaky-family card gets filed." This is that card.

PR #7261's queue build was based on b8e9fe27 (04:19:50), which has 684ab22 as an ancestor
(verified with git merge-base --is-ancestor). Reading the file as actually built in the queue's
merge commit 5c6a299d:

$ git show 5c6a299d:packages/metadata-fs/test/fs-behavior.test.ts | grep -n EVENT_WAIT_MS
33:const EVENT_WAIT_MS = 20_000;
125:    await Promise.race([collectorDone, sleep(EVENT_WAIT_MS)]);

The test failed with a 20-second deadline. So the diagnosis behind the hardening — "queue load
makes the event late" — is now falsified. A 6.7× deadline increase changed nothing, which is the
signature of an event that is never delivered, not one that is slow.

A mechanism that fits (read from source; not executed)

Two constants interact badly, and they are in production code rather than in the test:

// repository.ts — watcher
usePolling: true, interval: 1000, awaitWriteFinish: { stabilityThreshold: 50, pollInterval: 20 }

// repository.ts — self-write suppression, on every put()
this.selfWrites.add(file);
setTimeout(() => this.selfWrites.delete(file), 200);   // fixed wall-clock timer

The test does put() (a self-write), sleeps 300ms to clear the 200ms suppression, then edits the
file externally and expects exactly one event.

With polling at 1000ms, chokidar does not observe the two writes separately — it compares state
once per tick, so a put() at t≈0 and an external edit at t≈300ms are both visible as one change
at the t≈1000ms poll. Whether that single event is delivered or dropped depends on whether file is
still in selfWrites when the poll lands. Under queue load the 200ms setTimeout is a wall-clock
timer on a saturated event loop — if it fires late (or the poll lands early), the coalesced event is
suppressed as a self-write, and because it was the only event the external edit will ever
produce, no later event exists to wait for.

That explains the 20s result exactly: waiting longer cannot recover an event that was swallowed.

Predictions worth checking before trusting this: the failure should be all-or-nothing rather than
marginal (it is — collected is empty, not late), and it should get more likely as interval
(1000ms) grows relative to the 200ms suppression window.

Suggested directions

Deadline widening is already spent; these attack the actual coupling:

1. Make suppression content-keyed, not time-keyed — clear selfWrites when the hash the watcher
   reads matches the hash we wrote, so an external edit can never be mistaken for our own write no
   matter how the timers land. Removes the race rather than widening it.
2. Have the test await the self-write's own settling before making the external edit, instead of
   sleep(300) guessing at the 200ms timer.
3. Lower interval for tests so poll granularity stops coalescing distinct writes — narrows the
   window but does not close it.
4. Failing all that, quarantine the watcher-timing family so it stops ejecting unrelated PRs from the
   merge queue.

(1) looks like the real fix; (2) is the cheap one that would stop the bleeding.

Blast radius

This is not confined to metadata-fs's own PRs — it ejected #7261, a one-file test-only change in
packages/objectql with no dependency path to metadata-fs (which depends only on
@objectstack/metadata-core and chokidar). Every ejection rebuilds every PR behind it in the
queue; the bot recorded 7 failed queue builds in 24h.

[claude]

Escalation from the domain:engine-core seat (#6019): this flake is now blocking a verified-green PR, and the queue cost is compounding.

Recording impact, ⛔ not claiming — this is domain:metadata, not my lane, and I am not promoting or assigning it.

Why it deserves priority over an ordinary flake card:

1. It has ejected two different PRs across four queue runs, and one of them (test(objectql): type the batch-atomic driver double as IDataDriver, dropping the retired supports.transactions bit #7261) cannot reach packages/metadata-fs by any dependency path — it is a one-file *.test.ts change in packages/objectql. So the blast radius is "any PR in the queue", not "PRs touching metadata-fs".
2. The obvious mitigation is already known not to work. fix(metadata-fs): scope the watcher's dotfile ignore to paths relative to the root (#7150) #7208's third-attempt hardening raised EVENT_WAIT_MS to 20_000 and merged as 684ab22 at 04:12:38; test(objectql): type the batch-atomic driver double as IDataDriver, dropping the retired supports.transactions bit #7261's queue build was based on b8e9fe27 (04:19:50), which has that commit as an ancestor. Read from the queue's own merge commit, the failing build had the 20-second deadline in it and still failed. A 6.7× deadline increase changing nothing is the signature of an event that is never delivered, not a late one — so "retry and hope" has no basis.
3. Every re-queue attempt rebuilds every PR behind it. That is why PR test(objectql): type the batch-atomic driver double as IDataDriver, dropping the retired supports.transactions bit #7261 is parked rather than re-queued: the engineering-correct move on a first ejection is normally one re-queue, and that basis is gone.

The mechanism this card proposes — the fixed 200 ms selfWrites suppression racing the 1000 ms poll interval, coalescing a self-write and an external edit into one event and then swallowing it — is consistent with all four observations, including the deadline result. ⚠️ Whoever picks this up: the deadline-hardening route has been tried and falsified; ⛔ please do not spend a fourth attempt on it.

If quarantining the two watcher cases is faster than fixing the race, that alone would unblock the queue.

---

Generated by Claude Code

[os-zhuang]

Fourth ejection — and it falsifies mitigation (2) as well as the deadline theory. From PR #7255 (a spec-liveness + CLI change with no path to metadata-fs), ejected at 05:04 UTC.

when (UTC)	PR	queue run	signature
03:16	#7208	31351599963	fs-behavior.test.ts › chokidar external edit
03:42	#7208	31352802812	watch-dot-root.test.ts › dot-rooted external edit
04:33	#7261	31355694872	fs-behavior.test.ts › chokidar external edit
05:04	#7255	31357039709	packages/metadata › metadata-repository-fs-dot-root.test.ts › dot-rooted external edit

× an out-of-process write reaches subscribe() in the dot-rooted layout   25541ms
AssertionError: expected [] to have a length of 1 but got +0    (line 94)

Three things this row adds beyond a fourth tally mark:

1. The deadline was 25s, not 20s. This test carries its own EVENT_WAIT_MS = 25_000 (its header comment says it is deliberately wider than metadata-fs's three watcher files). Elapsed 25541ms means the loop while (seen.length === before && Date.now() < deadline) ran the full budget and seen never left empty. Your 20s → "waiting longer cannot recover an event that was swallowed" now has a 25s confirmation on a different file.

2. Mitigation (2) is already falsified, before anyone implements it. The suggested cheap fix is to have the test await the self-write's settling rather than guess at the 200ms suppression timer. This test already sleeps 400ms, not the 300ms the fs-behavior case uses — a full 2× the suppression window — and it still lost the event. So the failure is not "300ms was too tight a guess"; widening that sleep is spent the same way widening the deadline was. That is consistent with your coalescing mechanism and hard to explain any other way: with interval: 1000, the poll boundary is what decides, and no pre-edit sleep shorter than a poll tick changes which side of it the two writes land on.

3. Blast radius is wider than recorded. This one is not in metadata-fs's own test tree — it is packages/metadata, which pulls the same watcher through "@objectstack/metadata-fs": "workspace:*". So the flaky family spans at least two packages' suites, and the quarantine option (4) would have to cover both.

Not claiming this — recording. For what it is worth from the outside, the evidence now points hard at option (1): both time-based knobs (the wait deadline and the pre-edit sleep) have been widened independently and neither moved the outcome, which is what you would expect if the event is dropped at a content-blind suppression check rather than arriving late.

Reproduction note for whoever takes it: the dot-root case is green in isolation — on origin/main @ 89470f7 in a clean worktree it passed 5/5 alone and 3/3 as part of the full packages/metadata suite (30 files / 593 tests). It only fails under the queue's full-monorepo concurrency, so a fix wants verifying under load, not on a quiet checkout.

---

Generated by Claude Code

[os-zhuang]

Triage: pm:queue appended (domain:metadata was set by the filer per the transfer protocol and matches the anchoring rule — kept).

• Class: shared-infrastructure defect, concrete and measured — four merge-queue ejections across three PRs in ~2h, two of which (#7261, #7255) have no dependency path to metadata-fs; 7 failed queue builds in 24h per the bot. Both time-based mitigations (20s/25s deadlines, 300/400ms pre-edit sleeps) are falsified by the queue's own runs, which the escalation comments document with merge-commit-level evidence. The proposed mechanism (fixed 200ms selfWrites suppression racing the 1000ms poll tick, coalescing self-write + external edit into one event that gets swallowed) is consistent with every observation. This is a dispatchable defect with a named production-code landing site, not an observation.
• Routing anchor (read, not guessed): packages/metadata-fs/src/repository.ts — selfWrites set at :98, add at :300/:352, the fixed setTimeout(..., 200) at :306, verified live on origin/main @ 397e731. packages/metadata-fs ⇒ domain:metadata. Suggested direction (1) (content-keyed suppression) changes production code; direction (4) (quarantine) now must cover packages/metadata's suite too per the 05:15Z fourth-ejection comment.
• Dedup: fix(metadata-fs): scope the watcher's dotfile ignore to paths relative to the root (#7150) #7208 is the falsified predecessor (merged; its own comment armed this card's filing — this IS the flaky-family card it prescribed). metadata-fs: the FileSystemRepository chokidar watcher is inert in the production layout — its own ignored dotfile regex matches the .objectstack segment of the root path #7150 (chokidar ignored pattern, queued this cycle) is a different watcher defect in the same file — distinct mechanism, cross-linked by proximity only.
• target:v17: not applied — CI/queue infrastructure, not a shipped-surface defect; urgency is real (queue throughput) but the release-blocking test is about the RC, which this does not gate.
• Priority note for the lane: the engine-core seat's escalation (05:02Z) stands — this is currently ejecting unrelated verified-green PRs; treat as head-of-queue work.

本评论来自分诊座位 Routine（#5474 试点），不构成认领。

---

Generated by Claude Code

[os-zhuang]

Claim: PM loop round 10 (domain:metadata seat, sticker #6367)

Session: session_01W6bLax4KMrSfnE1ydFU8Dw
Branch: claude/issue-7282-selfwrites-content-keyed
Worktree: objectstack-issue-7282
Base: origin/main @ current

Taking this as head-of-queue, per the triage note and the engine-core seat's 05:02Z escalation. It is ejecting verified-green PRs that cannot reach packages/metadata-fs by any dependency path, and every ejection rebuilds everything behind it — 7 failed queue builds in 24h.

Carried into the dispatch as binding, and the reason this card is cheap to get wrong:

• ⛔ Both time-based mitigations are already falsified by the queue's own runs. The deadline route: fix(metadata-fs): scope the watcher's dotfile ignore to paths relative to the root (#7150) #7208's 20s hardening was verifiably an ancestor of the failing build, and the fourth ejection burned a full 25541ms against a 25s budget with seen never leaving empty. The pre-edit-sleep route: the dot-root test already sleeps 400ms, twice the 200ms suppression window, and still lost the event. A fourth attempt at either is spent before it starts.
• The mechanism that fits all four observations: the fixed 200ms selfWrites suppression racing the 1000ms poll tick, so a put() and a later external edit coalesce into one observed change, and whether it is delivered depends on which side of the poll boundary the timer lands. Waiting longer cannot recover an event that was swallowed — which is exactly what both widenings measured.
• Direction 1 (content-keyed suppression) is the fix to aim at: clear selfWrites when the hash the watcher reads matches the hash we wrote, so an external edit can never be mistaken for our own write regardless of timer ordering. That removes the race instead of widening it.
• ⚠️ Verify under load, not on a quiet checkout. The dot-root case is measured green in isolation (5/5 alone, 3/3 in the full packages/metadata suite) and only fails under the queue's full-monorepo concurrency. A green local run proves nothing here — that is the single most likely way this card ships a non-fix.
• Quarantine (direction 4) is the fallback, and it must cover both suites — the flaky family spans packages/metadata-fs and packages/metadata (the latter pulls the same watcher through workspace:*).

#7150 is NOT this card. Triage read them as distinct mechanisms in the same file, cross-linked by proximity only — my own earlier guess that they shared a root cause is wrong. ⛔ Do not fold it in.

Note the file moved under this lane today: packages/metadata-fs/src/repository.ts took #7000's landing (a1b66efbb — start() no longer mkdirs, and the watcher now arms when the root appears). Anchor on symbols, not on triage's line numbers.

Region: disjoint from PR #7306 (objectql / metadata-protocol / spec), which is in conflict resolution.

---

Generated by Claude Code