Skip to content

fix: count a timeout once, and let the failure rate see it - #57

Merged
xevrion merged 1 commit into
xevrion:mainfrom
yashksaini-coder:fix/timeout-attribution
Sep 6, 2026
Merged

fix: count a timeout once, and let the failure rate see it#57
xevrion merged 1 commit into
xevrion:mainfrom
yashksaini-coder:fix/timeout-attribution

Conversation

@yashksaini-coder

Copy link
Copy Markdown
Contributor

Closes #51. Both halves in one PR, as you asked, since the second is not verifiable without the
first.

Where I ended up, and where it differs from (a)

You said onTimeout is the single owner and the timeout arm in resolve should go. That is what
this does for the timeouts counter. totalFailed needed one more step, and I would rather show
you the measurements than argue it.

Dropping the arm literally, so resolve skips the client entirely on a timeout, fixes
client -> service and breaks client -> api -> db:

                       client.totalFailed   system.totalFailed
  before               1446                 1446
  arm dropped           530                 1446

The client is losing 100% of its traffic and reports a third of it. onTimeout credits whoever
gave up, which there is the api, so removing the resolve branch leaves nothing booking the root
at all.

Then chasing that turned up the same double one level down. engine.ts:2111 books the parent's
failure when a fan-out join completes with a failed child, and onTimeout had already booked the
same node, so the api came out at 2362 against a system total of 1446.

So the split that holds everywhere is:

  • onTimeout owns the timeouts counter, and books no totalFailed at all. Every node
    already has one path that books that when the request ends there: resolve for the root client,
    the join for everything below it. Both fire for every reason rather than only this one.
  • resolve owns the root's totalFailed, for every reason alike, and no longer adds a second
    timeout to the client's counter.

That is one rule each, which I think is the thing you were after. It also means the shed and
generic-error arms you flagged are untouched: client.totalFailed++ still runs for every reason,
so a root shed keeps its count and needed no splitting.

Measured

30 seconds per topology, node failures against what the system booked:

before after system
client -> service, client 2743 1494 1494
client -> api -> db, api 2362 1446 1446
client -> api -> db, client 1446 1446 1446
partial 26% loss, client 820 410 410

The timeout rate no longer outruns the load: 50 rps offered read 91 timeouts a second, now 46.

And the bug this started from, once the counting is right:

client timeout shown before shown after actually losing
40ms 0% 26% 26%

A middle node losing everything to its dependency reads 100% where it read 0%.

Attribution still lands where the waiting happened. In client -> api -> db the api holds the
timeouts and the client holds none, because the client never gave up, it was handed a failure.
The client's totalFailed still counts it, from resolve.

Tests

timeoutAttribution.test.ts, ten cases over the three topologies: no double at the root, the root
still booked when a node below it gave up, no node failing more often than the system did, a
client that cannot time out faster than it offers load, the counter landing on whoever waited, and
the failure rate tracking the traffic actually lost.

Seven of the ten fail against the old engine, checked by putting it back. The three that pass are
the ones that should: a client that gives up directly is still credited, and the bounds hold
either way.

921 tests pass. typecheck, lint and format:check clean, lint on the same 26 warnings as main.

Two rules, one owner each.

onTimeout owns the timeouts counter. Giving up is what that counter means,
and that hook is the only place that knows whose deadline elapsed. It no
longer books totalFailed at all: every node already has one path that books
it when the request ends there, resolve for the root client and the fan-out
join for everything below, and both fire for every reason rather than only
this one.

resolve owns the root's totalFailed, for every reason alike, and no longer
adds a second timeout to the client's counter.

Before this, a client calling its dependency directly was credited twice by
two paths for one dead request, and a node between the client and the slow
dependency was credited twice again by onTimeout and the join. Measured over
30s, per topology, node failures against what the system booked:

  client -> service     client 2743  ->  1494   system 1494
  client -> api -> db   api    2362  ->  1446   system 1446
  partial 26% loss      client  820  ->   410   system  410

The timeout rate no longer outruns the load: 50 rps offered read 91 timeouts
a second and now reads 46.

With the counting fixed, errorRate can include timeouts, which is the bug
this started from. It left them out of both halves, so the cell the canvas
labels "failing" was the failure rate among requests that did not time out,
and a caller losing a quarter of its traffic read 0%. That case now reads
26%, and a middle node losing everything to its dependency reads 100% where
it read 0%.

Attribution still lands where the waiting happened. In client -> api -> db
the api holds the timeouts and the client holds none, because the client
never gave up: it was handed a failure. The client's totalFailed still
counts it, from resolve.

timeoutAttribution.test.ts pins both rules across all three topologies:
no double at the root, the root still booked when a node below it gave up,
no node failing more often than the system did, a client that cannot time
out faster than it offers load, the counter landing on whoever waited, and
the failure rate tracking the traffic actually lost. Seven of the ten fail
against the old engine.

Closes xevrion#51.
@vercel

vercel Bot commented Sep 5, 2026

Copy link
Copy Markdown

@yashksaini-coder is attempting to deploy a commit to the whoarrryou's projects Team on Vercel.

A member of the Team first needs to authorize it.

@xevrion

xevrion commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Good catch on the chain case, you were right and my suggestion would have broken it. Checked the numbers, they hold. Thanks!

@xevrion
xevrion merged commit d1c9fee into xevrion:main Sep 6, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A node losing a quarter of its requests to timeouts reads 0% failing

2 participants