feat(compute): show host capacity above the Compute tabs, and measure kernels truthfully - #252
Merged
Merged
Conversation
…mes haven't advanced When the rolling baseline is used (second snapshot within 30s), if CPU times haven't advanced enough to calculate a meaningful delta, report 0 busy cores rather than omitting the field. This keeps the metric available without blocking for a full measurement cycle.
Per review ruling: do not unconditionally reset baseline on every call.
Instead, only update baseline when busy() returns a measurement. This
prevents rapid successive calls from starving the measurement window.
When the warm baseline path cannot measure (busy returns undefined),
return {} without updating baseline. The next call will measure across
the expanded time window.
Add deterministic test for baseline retention to verify the fix works
correctly. The test will fail if baseline is unconditionally reset.
Test 'serves the second snapshot from the rolling baseline without a blocking sample' now waits 50ms between snapshots, allowing os.cpus() times to advance. This ensures the warm path can produce a measurement while still proving the path never blocks (50ms << 200ms cold path cost). Both assertions remain load-bearing: < 150ms proves warm path, defined busy proves a real reading rather than empty object.
…seline retention
Extract the baseline advancement logic into a pure, exported advance()
function that takes Mark objects as input. This makes the retention rule
directly testable with hand-built data instead of relying on wall-clock
timing or real os.cpus() behavior.
Add Mark interface for { times: Times; at: number }.
Update load() to use advance() instead of duplicating logic.
Replace integration test with two deterministic unit tests:
- 'keeps the older baseline when the window produced no reading'
- 'advances the baseline only once the window produced a reading'
The identity assertions (toBe) directly validate that baseline is held
when measurement fails and advanced when it succeeds. These tests fail
immediately if the unconditional 'baseline = fresh' bug is restored.
Replace ps -o %cpu sampling with cumulative-CPU-second deltas across samples. %cpu is defined by the procps manual as the average over a process's entire lifetime, so a kernel that ran hot then idled kept reporting high. Deriving load from consecutive cumulative-time samples fixes that and gives Windows a path via Get-Process that needs no performance counters. KernelMetrics.sample(pid) is replaced by sampleAll(pids), batching the notebook route's per-kernel ps spawns into one call. Removes the now-superseded kernel-metrics.test.ts (tested the removed sample(pid) API); coverage moves to the new, more thorough test/science/kernel/metrics.test.ts.
…d about The prune loop at the end of sampleAll iterated the entire shared module-level baseline map, but readings only ever holds the pids the current call asked about. notebook.ts calls sampleAll scoped to one session's pids at a time, and the frontend polls /notebook/kernels per session, so two tabs on different sessions wiped each other's baseline entries every poll, permanently pinning cpu_percent at Unavailable. Prune only the pids passed to this call instead of every baseline key. A pid that died is still evicted (it was requested and is missing from readings); pids belonging to a different caller are no longer touched. Adds a regression test that spawns two real child processes and interleaves sampleAll([a]) / sampleAll([b]) / sampleAll([a]) to reproduce the cross-session wipe; verified it fails against the old loop and passes against the fix.
Two pollers now watch the same kernel pids on their own cadence: the Compute strip through /notebook/compute and the Kernels panel through /notebook/kernels. A baseline keyed by pid alone made each poll measure the window since the OTHER caller's poll, so an interleaved pair reported a fabricated 0 on a fully busy process or a percentage the machine could not physically produce. Key the baseline by scope and pid, pass a distinct scope from each route, and cover both the interleaved window and the death-eviction branch with real processes.
Stubbing (used / elapsed) * 100 to a literal 0 left the whole suite green: the only sampling assertion was cpu_percent >= 0, which 0 satisfies. Timing-based sampling cannot state an exact expectation, so lift the arithmetic into a pure KernelMetrics.derive the way KernelHost.advance was lifted, have sampleAll call it, and assert exact percentages from hand-built marks.
The fetcher threw on !response.ok and hostTiles(data()) reads the resource during render, so an errored resource re-threw where it was read. The nearest ErrorBoundary wraps the entire workspace, which made a server restart, a sleep/wake, or an in-place upgrade with the Compute pane open replace the whole app with the error page. Resolve a failed poll to no capacity instead. hostTiles already reads an absent capacity as "Unavailable" on all three tiles, which is the state the design asks for — never a 0, never a blank tile, never a thrown boundary. Refetch on visibilitychange too, so a tab that skipped its polls while hidden is current when it comes back rather than showing figures up to one interval stale. Cover the component itself, which had no test at all: the transport becomes a prop the way ComputeSurface already takes its children, so the degraded paths are mounted for real. A live Bun.serve endpoint cannot serve this suite — the happy-dom preload replaces globalThis.Response, so Bun.serve does not recognise a handler's return value and answers with its own placeholder — so the connection failure runs against a genuinely closed port and the error status is a real Response.
KernelHost keeps a rolling baseline in module state, so host.test.ts only held because it happened to be the first snapshot() caller in the process. Any earlier caller leaves a warm baseline, and a warm window that has not advanced returns no busy figure at all — so expect(undefined).toBeGreaterThanOrEqual(0) would fail for a reason that has nothing to do with the code under test. Give KernelHost the reset() KernelMetrics already has and call it in setup, so every test starts cold regardless of call order.
The route shipped without reaching either contract surface: the OpenAPI test asserts a hand-picked path list that never gained it, and tooling/sdk/openapi.json carries every other notebook route but not this one, so SDK consumers had no way to call it. Assert the route alongside its siblings and regenerate through tooling/repo/generate.ts, which also gives the JS client a typed notebook.compute().
Byproduct of running tooling/repo/generate.ts, which finishes with prettier across the repo. The doc landed unformatted, so format:check failed on this branch until now. Whitespace and emphasis markers only.
Two clients on the same scoped route polling the same pid milliseconds apart previously derived cpu_percent across a sub-second window. ps -o time= has whole-second resolution, so that window quantised to either a fabricated 0 or a wild multiple. Sub-second windows now omit cpu_percent entirely (surfacing as Unavailable in the UI) instead of computing across a window too short to mean anything; a 1 second window is the inclusive floor and still produces a value. memory_bytes is unaffected.
GET /notebook/compute conflated two cases behind one usage.length check: no live kernels (a real, exact zero) and live kernels that this poll could not sample (genuinely unknown). Both rendered "Unavailable" on an idle machine. Branch on live.length instead: zero live kernels sends kernels: 0 for both memory and cpu (a measurement), while live kernels with no sample for a field still omit it, unchanged. Amend the design doc's never-zero rule to state the same distinction explicitly.
.host-strip__caption clipped with nowrap + ellipsis, so a narrow pane
cut the caption before the numbers that are the point of the tile
("kernels · 9.2 GB fr…"). Let it wrap, and reserve a fixed two-line
min-height on every tile unconditionally so a caption that needs the
second line grows all three tiles together instead of leaving the
others short with their meter sitting at a different height.
For keys by referential identity, and every 2.5s poll parses a brand new response body, so hostTiles(data()) fed straight into <For> tore down and rebuilt all three tiles every tick. The tile list is always exactly memory/cpu/kernels in a fixed order, so key by position instead: memoize the tile computation and switch to <Index>, which patches the existing signal per slot rather than disposing the DOM.
Same root cause as the host strip, but the kernel list is genuinely dynamic (entries appear, disappear, and get re-sorted by last_activity_at), so position-keying (Index) would scramble cards on a re-sort. Reconcile the polled kernels into a store keyed by the kernel's stable id instead, so an unchanged kernel keeps its object identity and <For> leaves its card mounted; only a kernel whose fields actually changed gets patched in place. The resource (loading state, refetch call sites) is unchanged — only how the rendered list is derived from it. Pulled the store+reconcile into use-kernel-list.ts, matching this file's existing use-execution-authority.ts pattern of extracting reactive hooks, so the mechanism is unit-testable without KernelPanel's SDK/router context chain.
RightPane.tsx wraps ComputeSurface in a <Suspense>. Reading a resource with data() re-registers with the nearest Suspense boundary on every in-flight fetch, including background polls — so every 2.5s poll in HostStrip and KernelPanel suspended the entire right pane (tiles, tabs, kernel cards) back to the loading fallback and then restored it, twice per poll. data.latest only suspends on the first load and returns the previous value while a refetch is outstanding, so the pane now stays mounted across polls. The earlier <Index>/reconcile identity fixes are kept; they matter once suspension is no longer tearing the whole subtree down. Adds a Suspense-mounting regression test to HostStrip.test.ts, via a new host-strip-suspense-fixture.tsx test fixture (Suspense, render, and HostStrip loaded from one file's import graph so they share a solid-js module instance — Suspense's effect-resumption breaks across split instances in this harness, unlike ErrorBoundary).
KernelPanel gated its 2.5s poll on summary().running/queued, so a fresh session (0 live, 0 running, 0 queued) never created the interval that would have discovered a kernel starting. Follow HostStrip's shape instead: poll unconditionally while mounted, skip while the tab is hidden, and refresh immediately on visibilitychange. Also gives KernelPanel the same optional `request` transport prop HostStrip already has, so its own network path is test-seamable.
Kernels spawn under a sandbox wrapper (bwrap) with detached: true, so the pid metrics.ts sampled was always the wrapper's own pid -- the Python interpreter running as its child, plus anything it forks, was never counted. A kernel holding a deliberately allocated 1GiB reported MEMORY: 2MB. detached: true makes the kernel's pid its own process-group leader, so every descendant shares its pgid. Sample by group instead: one `ps -Ao` spawn per poll covering every host process, folded by pgid in a new pure `group()` step (GNU ps's own -g selects by session, not pgid, so it can't narrow the ps invocation itself -- verified against this host's procps-ng). Windows keeps its per-pid Get-Process reading; a parent-walk there is a separate, untestable-here job. Existing sampling tests spawned plain (non-detached) children or used process.pid directly, neither of which is its own group leader -- under the new pgid-based read, none of those pids would be found. Updated them to spawn detached, matching notebook.ts's real shape.
…ages Summing a process group's CUMULATIVE cpu seconds is only monotonic while the group never loses a member. When a forked worker is reaped its accumulated seconds leave the total, so the next delta reads near zero and derive()'s `used >= 0` guard lets it through as a MEASURED zero — a kernel pegging a core rendered "0.0 cores" on 9 of 13 polls under any fork-heavy workload (multiprocessing/joblib pools, subprocess loops, shell cells). Keep cpu per member inside the Reading and sum only the per-pid deltas for pids present in both samples: every term is one process's own non-negative progress, so the aggregate is monotonic by construction. A member that vanished contributes nothing further; one that appeared starts accumulating from the poll it appeared in. Summing RSS across the group double-counted every copy-on-write page: a leader holding 300 MB plus three forked children reported 1.29 GB, a 4.03x overcount that can exceed memory.total and silently peg the meter (ratio() clamps to 1). On Linux read /proc/<pid>/smaps_rollup and sum Pss instead, which divides each shared page by the number of processes mapping it. Where the rollup is unreadable — macOS, kernels before 4.14, permission denied — the summed RSS stays as the documented fallback: a wrong-but-real number, never a fabricated 0. Baseline entries were also immortal: the prune loop only visits pids the current call named, and the routes only pass ACTIVE pids, so a stopped kernel's mark lived as long as the process. Sweep every mark older than 30s on each poll, and refuse to derive across a window that old, so a recycled pid reads Unavailable for one poll rather than a fabricated 0.1388 (rendered "0.0 cores"). A true zero is untouched: a group that genuinely burned nothing across a measurable window still reports cpu_percent 0.
Ruling 1's minimum window went into KernelMetrics.derive only; busy() rejected a window solely when the counters had not advanced at all. Two concurrent /notebook/compute requests measure a sub-millisecond span in which at most one core has ticked one 10ms jiffy: landing on idle that reads `busy: 0` on a host running at three cores, landing on user/sys it reads every core pegged. Measured over 120 concurrent polls with a core deliberately held: 27 fabricated zeros and 5 fully-pegged readings. Refuse the window in advance() rather than inventing a second mechanism — its retention rule already keeps the older baseline when a window produces no reading, so the next call spans something real. The cold path keeps its own 200ms sample: it takes both marks itself, so no concurrent caller can truncate it, and blocking the first paint for a second buys precision nobody asked for. Two existing tests moved their `at` past the floor; every expect() in them is unchanged.
…hrowing KernelPanel's fetcher still rejected on !response.ok and on connection failure. `data.latest` re-throws a settled error on the render path, and the nearest ErrorBoundary is app.tsx's, which wraps the whole workspace — so a server restart, a sleep/wake, or one 503 replaced the entire UI with the error page. ccc2d24c fixed exactly this for HostStrip and did not carry it across; bece71a's unconditional poll made it reachable every 2.5s in every session. The fetcher now resolves to no inventory, the same shape HostStrip uses, which also makes the panel's existing "Kernel inventory unavailable" alert reachable for the first time. An empty list after a failed poll is not "No live kernels" — the panel does not know that — so the empty state says the inventory could not be read. A poll that fails degrades visibly rather than looking idle. hostTiles dereferenced capacity.kernels.live whenever capacity was truthy while memory and cpu were guarded, so a body missing that section threw inside HostStrip's createMemo — the same route to the app-wide boundary. Each of the three tiles is now guarded on its own section, and the parameter is a Partial because a partial body is what the route can really return. Delivers the second half of the spec's failure rule: "fails or returns a partial body -> each affected tile reads Unavailable".
The design still described `ps -o pid=,time=,rss= -p <pids>` and a Map<pid, …> parser, while the code has run a full-host `ps -Ao pid=,pgid=,time=,rss=` grouped by pgid since 03673cf. Record what ships: the process-group scope and why no portable pgid selector exists, per-member cpu deltas and the undercount they trade for monotonicity, PSS from /proc/<pid>/smaps_rollup with the documented RSS-sum fallback and its overcount, the 30s baseline expiry, and the one-second minimum window on both derive() and busy(). Failure handling gains the kernels-poll and smaps_rollup rows; the testing table gains the cases those rules are held to.
…unters Summing per-pid deltas over the members present in BOTH samples cannot see a kernel at all when the work is done by short-lived workers: a 2s worker never spans two 2.5s polls, and the leader that reaps it sits in waitpid burning nothing. A group pegging a full core reported cpu_percent 0 on 13 of 13 polls, which the strip renders as "0.0 cores" — a fabricated zero, not a measurement. /proc/<pid>/stat's cutime/cstime hold the time of a process's reaped children, recursively, so a worker's seconds do not vanish when it is waited for — they move into its parent's counters. Summing utime+stime+cutime+cstime across every live member is therefore monotonic by construction, which is the property every earlier attempt lacked. Every live member and not just the leader: the group leader in production is the sandbox wrapper, and it is the interpreter beneath it that reaps the workers, so a leader-only total under-reports the same group by 5x (measured: 0.21x of truth). Ticks also drop the smallest measurable reading from 0.4 cores — one whole `ps -o time=` second over a 2.5s poll — to roughly 0.004. The divisor is read from AT_CLKTCK in /proc/self/auxv rather than assumed, falling back to 100. macOS has no equivalent counter, so its readings stay `ps`-derived and derive() refuses any window a member left the group in, where the sum could have gone backwards. Unavailable is the honest answer there; a floored zero is not. Also: a persistent zombie no longer defeats PSS. A zombie has no address space, so its smaps_rollup stays unreadable for as long as it stays unreaped — which a routine using multiprocessing leaves indefinitely — and letting any unreadable member force the summed-RSS fallback silently restored the 4x overcount (866 MB reported against 304 MB held). A member whose RSS is measurably 0 holds nothing and is skipped; one that could hold pages still falls back. Measured on the reviewer's repro, 13 polls against ground truth read independently from /proc: reported/true 0.99-1.01, zero fabricated zeros.
The one-second floor was added to a single module-level baseline, which turns it into a race the first poller always wins. Two browser tabs with the Compute pane open poll /notebook/compute every 2.5s, staggered by ~150ms: whichever lands first each cycle advances the shared mark, so the other measures only that 150ms gap, is refused by the floor, and is refused again every cycle after. Measured: 7 of 8 polls absent for the second tab, its CPU caption stuck on "8 cores" with no busy figure at all, indefinitely. Baselines are now keyed per caller, the way KernelMetrics already keys its own, and the caller is threaded from the route: HostStrip names itself with a per-mount identity and GET /notebook/compute passes it to both samplers. The kernel-share sampler had the identical defect on the identical route — its scope was per-ROUTE, so two tabs shared one window there too — so it takes the same key. An abandoned caller's mark is swept after the same 30s staleness bound, so a closed tab cannot leak a baseline. A client that sends no identity shares the default key, which is exactly the behaviour before the parameter existed. What the floor fixed stays fixed: two requests from one caller racing inside a single scheduler tick still produce neither busy: 0 nor busy: cores (60 rounds of concurrent pairs against a held core, every reading absent or strictly inside the open interval).
The previous wave rewrote this section to describe the churn behaviour as an acceptable "undercount". That was a fabricated zero wearing a different word: the shipped sampler reported cpu_percent 0 on 13 of 13 polls while the group held a full core, and the spec said so approvingly. States the rule plainly — a figure that cannot be measured is omitted, never emitted as 0, and a measured 0 is shown as 0 — and describes the mechanism that actually ships: /proc/<pid>/stat's cutime/cstime summed across every live group member, monotonic by construction because a reaped child's time moves into its parent rather than vanishing. Records why the total covers every member and not just the leader, where the tick divisor comes from, what macOS does instead and why omitting beats approximating there, the zombie case in the PSS fallback, and why the one-second floor and the per-caller key have to ship together. Also closes the "ps cumulative time resolution" open risk: /proc's 10ms ticks drop the smallest measurable reading from 0.4 cores to roughly 0.004 on Linux.
/notebook/compute was scoped per caller so two tabs stopped starving each other, but /notebook/kernels kept a single shared scope. Per-kernel CPU is a delta across the window since that caller's previous poll, so two panels on one scope truncate each other's window to the stagger between them, fall under the one-second floor, and both read Unavailable forever. Scope the kernels sampler by caller too, and lift the per-mount identity out of HostStrip into poll-identity.ts so both surfaces name themselves the same way. The id is read off the raw query rather than the validated one: it identifies a poller, it is not part of the route's contract, and an absent or forged value can only cost its own sender a window.
The caption carries the numbers that make the tile worth reading — free of total, busy of cores — and beside a 15px value there is only room for both while the pane is wide. Narrower than that it wrapped into a few characters per line. Make each tile a query container so all three change together rather than each wrapping at whatever width its own caption needs, and below 180px drop the caption onto its own line under the value, where it has the full tile width. Widening the pane returns it to the side.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a machine-capacity strip above the Compute pane's Kernels/Jobs tabs, and fixes the per-kernel sampler underneath it so the numbers it shows are true.
Before this, the Compute tab reported only what the current session owned. Nothing on the surface answered the question asked before starting a run: does this machine have room? Free RAM, core count and utilisation were absent, so the operator left the app for
htopto decide whether to launch work.Design:
docs/specs/compute-host-strip-design.md.Backend
science/kernel/host.ts(new) — machine facts only; knows nothing about kernels, sessions or projects, so it tests standalone.totalisos.totalmem()everywhere.availableisos.freemem(), overridden by/proc/meminfoMemAvailablewhere readable — on Linuxos.freemem()reportsMemFree, which excludes reclaimable page cache, so a healthy 16 GB desktop otherwise reads as ~1.4 GB free. Busy cores come from a rolling baseline;advance()is pure, so the delta arithmetic is pinned to known inputs in tests.science/kernel/metrics.ts— rewritten. A kernel'sprocess_idis the sandbox wrapper, not the workload, so a sample of the leader alone read 1 GiB of real usage as 2 MB. Sampling now covers the whole process group.CPU comes from
/proc/<pid>/statutime + stime + cutime + cstime÷AT_CLKTCK. The child fields matter: they hold the time of reaped descendants, recursively, so the group total stays monotonic across a membership change by construction — a worker forked and reaped inside one poll window has not vanished from the sum, its time moved into its parent's counters. Where nothing accumulates reaped time (macOSps, WindowsGet-Process), a window that lost a member is refused rather than floored.ps -o %cpuis deliberately not used — procps defines it as the average over a process's entire lifetime, which is wrong for a live meter.Memory prefers PSS from
smaps_rollup, falling back to RSS, so two workers sharing pages are not double-counted.Truthfulness rule, enforced throughout: a field that cannot be measured is omitted and renders "Unavailable". It is never fabricated as 0. A field reported as 0 means the sampler measured exactly nothing — which an idle kernel, and a host with no kernels, genuinely does.
Baselines are scoped per caller. Sharing them between
/notebook/kernelsand/notebook/computelet two pollers consume each other's windows and producedcpu_percent: 2000— "20.0 cores" on an 8-core host. Windows are floored to one second, and marks older than 30s are evicted so a recycled pid cannot derive a percentage for work no kernel did.server/routes/notebook.ts— newGET /notebook/compute, published inopenapi.jsonand the generated SDK types.Frontend
HostStrip+host-tilesrender memory, busy cores and kernel count above the tabs, with partial tiles when only some fields are measurable.Three fixes worth calling out, all found by driving the real binary:
<Suspense>plus readingdata()during refetch tore the tab down and rebuilt it every 2.5s. Readingdata.latestkeeps the last good render while the next lands.<Index>keys by position instead of identity, so a fresh array of equal rows stops re-creating DOM.A failed host poll now degrades the tile instead of throwing, which kept a transient sampler error from replacing the whole workspace with the error page.
Testing
frontend/workspace: 175 pass / 0 fail (src/atlas), typecheck cleanbackend/cli: 80 pass / 0 fail acrosstest/science/kernel/,test/science/kernel-metrics.test.ts,test/server/notebook.test.tsThe full
backend/clisuite has pre-existing environmental failures that reproduce identically atmain, so it was not used as a signal here; the three touched suites were run instead.Review notes
tooling/sdk/js/src/v2/gen/*andopenapi.jsonare generated from the new route — worth confirming they match a fresh regeneration.files-pane-redesignorsettings-rail-and-local-models, so merge order does not matter.