fix(infra): pods list live by default, and a terminated pod is "Ended" not "Down" - #751
Merged
Conversation
…" not "Down"
The pods view painted most of an autoscaled fleet red. Two causes, both here.
`deriveHostStatus` returned "down" — styled `--severity-error` — for any
resource whose newest datapoint was older than 5 min. But `lastSeen` is
`max(TimeUnix)` over a whole window (12h by default), so "down" really meant
"this pod's series ended", which for a pod on an autoscaled fleet is the normal
end of its life: HPA scale-in, a rollout, a replaced Fargate task, a reclaimed
spot node. Continuity alone cannot prove a resource is down, so the state is now
`ended` and reads neutral. A real down signal needs an expectation — `k8s.pod.phase`,
or a workload's available-vs-desired replicas — and belongs beside these, not
instead of them. The status policy is shared, so nodes, hosts and workloads get
the same treatment.
The list also had no notion of who was still running: it returned the union of
everything that reported in the window, so dead pods inflated the denominator
and outranked live ones in the saturation sort. `PodLifecycle` ("live" | "ended"
| "all") now defaults to live, cutting at the same 300s-before-endTime threshold
the badge uses so the predicate and the badge cannot disagree. It rides with the
filters rather than the scope, so the count query narrows identically. The old
`stale` scope was the same idea under a worse name and is now the `ended`
lifecycle, offered as the band's fourth cell — clicking it swaps which fleet is
on screen rather than narrowing the live one.
`listPodsSummaryQuery` returns `livePods`/`endedPods` as absolute counts, with
the saturation buckets counted within the requested lifecycle so the band stays
a valid denominator for the list beside it.
The unfiltered empty state no longer says "install the Helm chart" when the
fleet has simply all ended — a finished job, an environment scaled to zero — and
points at the Ended scope instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🍁 Maple PR previewWarning Preview cleanup could not be confirmed. The Alchemy teardown outcome was Final commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Kubernetes pods view painted most of an autoscaled fleet red — hundreds of rows marked Down on a fleet where nothing was actually failing. Two causes, both fixed here.
Why "Down" fired on the normal case
deriveHostStatusreturned"down"— styled--severity-error— for any resource whose newest datapoint was older than 5 minutes. ButlastSeenismax(TimeUnix)over a whole query window (12h by default), so "down" really meant "this pod's metric series ended". On an autoscaled fleet that is the normal end of a pod's life: HPA scale-in, a rolling deploy, a replaced Fargate task, a reclaimed spot node.Continuity alone cannot prove a resource is down, so the state is now
endedand reads neutral (text-muted-foreground,bg-muted-foreground/40dot) instead of error red. A genuine down signal needs an expectation —k8s.pod.phase, or a workload's available-vs-desired replicas — and belongs beside these states, not instead of them.Why the list was full of dead pods
The list returned the union of everything that reported anywhere in the window, with no notion of who was still running. Dead pods inflated the "N pods in scope" denominator and outranked live ones in the default saturation sort.
PodLifecycle("live" | "ended" | "all") now defaults to live, cutting at the same 300s-before-endTimethreshold the badge uses — so the predicate that selected a row and the badge on it cannot disagree. It rides inlistPodsFiltersrather than with the scope, solistPodsCountnarrows identically and the page can't read "50 of 656".What changed
HostStatusis"active" | "idle" | "ended";downis gone. Shared policy, so nodes, hosts and workloads get the same treatment — an ASG or Karpenter scaling a node out is the same non-event.listPodsQueryapplies the lifecycle outside the grouping, alongside the scope predicate.listPodsSummaryQueryreturnslivePods/endedPodsas absolute counts, with the three saturation buckets counted within the requested lifecycle so the band stays a valid denominator for the list beside it.stalescope was the same idea under a worse name — it is now theendedlifecycle, and the band's fourth cell. Clicking it swaps which fleet is on screen rather than narrowing the live one.Reviewer notes
?scope=stale(pods) or?status=down(nodes) will now failvalidateSearch. I took the clean rename over carrying both literals — flag it if you'd rather keep the old values decoding.k8s.pod.phase,k8s.pod.status_reason,k8s.container.restarts, andk8s.deployment.desiredvs.availableare all collected by default indeploy/k8s-infrabut unqueried. That is what would let "Down" mean something again, at the workload level where replicas missing (rather than churning) is the actual incident.bun typecheck,bun run lint, and the query-engine + web infra test suites pass.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.