fix: remove stale waiter in waitForState timeout handler - #330
Open
YunchuWang wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a leak in the in-memory testing backend where waitForState() timeout handlers failed to remove stale waiters due to an identity/reference mismatch, causing stateWaiters to grow over time and timers to remain untracked across reset().
Changes:
- Fix waiter cleanup on timeout by removing the exact
waiterobject (indexOf(waiter)) rather than comparing function references. - Track
waitForState()timeout timers inpendingTimers, and remove them on resolve/reject/timeout soreset()can reliably clear them. - Add regression tests to ensure timed-out waiters are removed (both single-waiter and multi-waiter scenarios).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| packages/durabletask-js/src/testing/in-memory-backend.ts | Fixes waitForState() timeout cleanup and ensures timeout timers are tracked/removed via pendingTimers. |
| packages/durabletask-js/test/in-memory-backend.spec.ts | Adds regression tests verifying stale waiter cleanup after timeouts (single and multiple waiters). |
The timeout handler in InMemoryOrchestrationBackend.waitForState() used findIndex with `w.resolve === resolve` to locate and remove the timed-out waiter. However, the waiter's resolve property is a wrapper function that calls clearTimeout before delegating to the original resolve, so the identity check always failed and the stale waiter was never removed. Fix: - Move waiter declaration before timer so the timeout callback can use indexOf(waiter) for correct object-identity lookup. - Track waitForState timers in pendingTimers so reset() cleans them up. - Remove timer from pendingTimers on resolve, reject, and timeout. - Delete the stateWaiters map entry when the last waiter is removed. Fixes #201 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
YunchuWang
force-pushed
the
copilot-finds/bug/fix-waitforstate-timeout-cleanup
branch
from
July 29, 2026 23:21
35e93a4 to
e2a2b38
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
packages/durabletask-js/test/in-memory-backend.spec.ts:978
- This test depends on real 50ms/60s timers. Using real timers can be flaky and (if an assertion throws before cleanup) can leave a 60s timer keeping the Jest process alive longer than necessary. Using fake timers makes the timeout deterministic and avoids long-lived real timers.
// Start a waiter with a long timeout (won't time out during the test)
const longWaitPromise = backend.waitForState(
instanceId,
() => false, // Never matches
60000, // 60 second timeout — won't fire
);
// Start a waiter with a very short timeout
await expect(
backend.waitForState(
instanceId,
() => false, // Never matches
50, // 50ms timeout
),
).rejects.toThrow("Timeout waiting for orchestration");
packages/durabletask-js/test/in-memory-backend.spec.ts:948
- The test relies on a real 50ms timeout. This can be flaky under CI load and can intermittently fail if timers don’t fire promptly. Prefer Jest fake timers and advance the clock deterministically so the timeout handler is exercised reliably.
This issue also appears on line 963 of the same file.
// Call waitForState with a predicate that will never match and a short timeout
await expect(
backend.waitForState(
instanceId,
() => false, // Will never match
50, // 50ms timeout
),
).rejects.toThrow("Timeout waiting for orchestration");
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #201.
The bug
waitForState()'s timeout handler tried to remove its own waiter withwaiters.findIndex((w) => w.resolve === resolve). Butw.resolveis a wrapper closure(
(result) => { clearTimeout(timer); resolve(result); }), not the Promise's originalresolve. Two different function objects, sofindIndexalways returned-1and thesplicewas dead code.Root cause is a circular dependency: the waiter needs
timer(toclearTimeout), and thetimer callback needs
waiter(to remove it). The original code declaredtimerfirst, soits callback had no
waiterin scope and fell back to comparingresolve— which nevermatches.
Verified against unpatched
main:No data corruption — resolving an already-rejected Promise is a no-op — but the waiter and
its closure leak, and
notifyWaiters()keeps evaluating dead predicates untilreset().The fix
Declare
waiterbefore the timer. The wrapper readstimeronly when invoked, by whichpoint it has been assigned, so the circular dependency dissolves and the timeout callback can
use object identity:
waiters.indexOf(waiter)instead of the brokenfindIndexreference comparisonpendingTimers, and remove it on resolve / reject / timeoutstateWaitersmap entry when its last waiter is removedValidation
The two new tests fail on unpatched
main:mainshould clean up stale waiters after waitForState timeoutExpected: false, Received: trueshould remove only the timed-out waiter when multiple waiters existThe crash is a real consequence: the assertion fails, so the test's own
reset()+.catch()cleanup never runs, leaving an unhandled
Backend was resetrejection.With the fix: build 0, lint 0, 1178/1178 tests across 67 suites.
Note for reviewers — rebase and conflict resolution
This branch was rebased onto current
main(it predatedexecutionIdonOrchestrationInstanceand no longer compiled against it).The rebase conflicted with the
timeoutMs === 0"wait indefinitely" feature that landed onmainafter this branch was cut. Taking either side wholesale would have been wrong — thisbranch created the timer unconditionally, which with
timeoutMs = 0would fire asetTimeout(…, 0)immediately and break that feature. The resolution keeps
main'sif (timeoutMs > 0)guard andthe
timer: … | undefinedtype, and applies the fix inside it.Both
timeoutMs = 0tests still pass:One correction to #201
The issue lists a fourth impact:
That is not accurate.
reset()iteratesstateWaitersand callswaiter.reject(...), whichis the wrapper that already calls
clearTimeout(timer):Adding the timer to
pendingTimersis defensive bookkeeping, not a leak fix. The other threeimpacts in the issue are accurate.