Skip to content

fabric status is broken in production: carry the reconnect total as u64 - #41

Merged
myobie merged 1 commit into
mainfrom
fix/telemetry-u128-breaks-status
Aug 4, 2026
Merged

fabric status is broken in production: carry the reconnect total as u64#41
myobie merged 1 commit into
mainfrom
fix/telemetry-u128-breaks-status

Conversation

@myobie

@myobie myobie commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Fixes a live regression I introduced in #36. fabric status fails on Silber right now with Error: u128 is not supported.

The bug

LatencySummary.total_micros was a u128. u64 microseconds overflows after roughly 584,000 years, so this is not a real downgrade — it is the type the wire can carry.

Read-only impact: only the operator command. The daemon, sessions, sync, shell, and exec are unaffected, and the telemetry file keeps recording correctly. cos independently confirmed it exits 1, not 0, so scripts fail loudly rather than reading a false green.

Why it reached production

Every check that should have caught it ran against empty state, which is the one case that works:

  • unit tests asserted on the struct
  • the integration test called the response builder directly and never serialized it
  • my hand check used a fresh node with no peers
  • rollout acceptance ran at install, before the first probe populated the map

The break is time-delayed by construction. Root's acceptance was honest and could not have caught it.

Why the obvious test does not catch it either

This is the part worth keeping. A plain serde_json round-trip of the telemetry snapshot passes while the command is broken — I wrote that test first and watched it pass. serde_json handles u128 in a normal struct.

ControlResponse is an internally tagged enum (#[serde(tag = "type")]), so serde routes it through its Content buffer, and that buffer has no u128 variant. The failure is on decode, in the client.

So the regression test:

  • serializes the real ControlResponse, not the inner struct
  • starts from a populated map, because empty is the case that always worked

Verified both directions: it fails on the u128 type with the exact production error — Error("u128 is not supported", line: 0, column: 0) — and passes on u64.

The plain-JSON test is kept but relabelled necessary but not sufficient, so nobody mistakes it for the guard.

Rollout note

A fixed client decodes correctly against an unfixed running daemon — the daemon serializes the value fine, only the client's decode failed. So installing the new binary repairs fabric status without requiring a daemon restart. Root's call.

Tests

197 lib tests green. Changed files pass rustfmt.

I shipped LatencySummary.total_micros as a u128 in PR 36. fabric status then
failed with "u128 is not supported" on any daemon that had probed a peer, which
is every daemon with a peer, a few minutes after start.

u64 microseconds overflows after roughly 584000 years, so this is not a real
downgrade. It is the type the wire can carry.

WHY IT REACHED PRODUCTION. Every check that should have caught it ran against
EMPTY state, which is the one case that works. The unit tests asserted on the
struct. The integration test called the response builder directly and never
serialized it. My own hand check used a fresh node with no peers. And the
rollout acceptance ran at install, before the first probe populated the map. The
break is time-delayed by construction, so a green acceptance could not have
caught it.

WHY THE OBVIOUS TEST DOES NOT CATCH IT EITHER, which is the part worth keeping.
A plain serde_json round-trip of the telemetry snapshot PASSES while the command
is broken. serde_json handles a u128 in a normal struct. ControlResponse is an
INTERNALLY TAGGED enum, so serde routes it through its Content buffer, and that
buffer has no u128 variant. The failure is on DECODE in the client.

So the regression test serializes the real ControlResponse, not the inner
struct, and it starts from a POPULATED map. I verified it fails on the u128
type with the exact production error, Error("u128 is not supported"), and passes
on u64. The plain-JSON test is kept and relabelled as necessary but not
sufficient, so nobody mistakes it for the guard.

A fixed client decodes correctly against an unfixed running daemon, because the
daemon serializes the value fine and only the client's decode failed.

197 lib tests green.
@myobie
myobie merged commit 5999384 into main Aug 4, 2026
2 checks passed
@myobie
myobie deleted the fix/telemetry-u128-breaks-status branch August 4, 2026 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant