You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
GPU task rows now lead with the real entry point instead of a bare
interpreter name: when argv0 is python, torchrun, bash or another
generic launcher, the module after -m or the first script argument
becomes the row title, with the interpreter and the detected Python
environment (conda env, project venv, uv tool, poetry virtualenv) shown
as chips. The fleet-wide program search, the per-card top-process label
and the name sort follow the same display name.
The workload identity pass now also reports each PID's host-side
footprint — cumulative CPU seconds and resident memory from the same /proc reads it already performs — and the task row renders it as
"CPU 均值 N 核 · 主机内存 N GiB", so a stalled data loader is visible
without shelling into the node. Snapshot workload objects gain cpu_seconds and rss_mib (both nullable).
The truncated command line in a task row expands in place on click
instead of requiring copy-and-paste to read the training config.
mocop deploy --help, mocop migrate --help, and mocop service --help
now describe every flag and action, and help output names the program mocop instead of __main__.py. The operations runbook gained a command,
flag, and exit-code reference (0, 1, 2, 75).
GET /api/meta is now a complete machine-readable contract: every GET
route lists its accepted query parameters with type, bounds, and default,
every write lists its body field schema and bodyLimitBytes, every route
names its responseType, and the document also publishes conventions,
writer requirements, the error-code catalog, and a documentation URL for
the running release. The manifest is generated from the same declarative
table (api_manifest.py) the handlers validate against, replacing five
hand-written query parsers.
mocop --version prints the package version. init, deploy, migrate,
and every service action accept --json and write one {ok, code?, ...}
document, matching config check --json and doctor --json. Refusals stay
on that envelope so an agent never has to parse English from stderr.
mocop api PATH asks the running service any public or authenticated GET
question from the monitor host without knowing the listen address, the
capability file, or the Bearer header: both come from the configuration
directory, the response body is written to stdout unchanged (/api/events
streams until interrupted), and the exit code follows the CLI table (0 for
a 2xx, 1 for any other status or an unreachable service, 2 for usage or
configuration problems) with a JSON {error, code} envelope on every
failure. Reader and writer routes are refused with DASHBOARD_ONLY because
their marker header and same-origin checks belong to the dashboard.
GET /api/meta capabilities now include updateSupported. Excess
connections answer 503 CONNECTION_LIMIT as JSON instead of an empty body.
Error envelopes carry field whenever exactly one query parameter or body
field is at fault — an unknown parameter's name, a malformed or out-of-bounds
parameter, a wrongly typed or unaccepted body field, and the single-field
cross checks on maintenance, host-group, and incident-action writes — so an
agent maps a rejection back to its input without parsing the message.
403 AUTHENTICATION_REQUIRED responses carry a hint naming the header to
send and where the capability file lives, plus the documentation URL, so
an agent can recover from a cold start without out-of-band knowledge.
mocop config check --json writes the validation report as one JSON
document on stdout (also for a rejected configuration), matching doctor --json.
GET /api/capacity?gpus=N&min_vram_gib=G&model=M answers the placement
question from one bounded authenticated call: ranked same-host, same-model
GPU groups with the available devices, deficit, minimum free VRAM, and the
maintenance and health exclusions, using exactly the dashboard's matching
rules. A shared fixture pins the browser leaf and the new capacity.py to
one ranking, and the browser leaf now orders hosts by code point instead of
the locale so both agree everywhere.
tests/gpu_tasks_test.mjs now runs in CI: the workflow globs every browser
leaf and contract test instead of maintaining a list that had drifted.
AGENTS.md at the repository root is the entry point AI coding agents read
by convention: the reading order, the drift tests that enforce each
contract, the boundaries that must hold, and how to operate a deployment,
each pointing at the document that owns the detail.
Changed
doctor --json now includes the local host in hosts[] (with local: true) and adds ok alongside the existing status field. --once / --strict with a subcommand are rejected as usage errors.
Inventory payloads must publish infrastructureHosts and sshDiscoveryMode; the dashboard no longer defaults those fields.
Workload record parsing moved from probe.py into a new workloads.py
module, and the GPU task projections moved from app.js into the gpu-tasks.js leaf (ADR-0021);
both budgets ratcheted down. The per-owner GPU occupancy rollup behind GET /api/usage moved from StateStore into a pure usage.py module that
takes a copied timeline and is unit-tested on its own; service.py
ratcheted from 2,725 to 2,425 lines. The test-only parse_linux_resource_payload wrapper left probe.py for the test module,
and the doctor and the route resolver share one ssh -G option resolver in ssh_topology.py instead of two copies that had already drifted on how a
bare option line is read. Configuration loading moved out of config.py
into config_loader.py (load_config, load_private_config, resolve_config_path, and friends), and the single 575-line parser became
one function per configuration section, run in the same order so every
error message and its precedence are unchanged; config.py keeps the
schema — limits, typed sections, validators — and ratchets from 1,550 to
375 lines. The same split applies to the HTTP contract: api_schema.py
owns the query and body field types and the two validators, api_manifest.py
the route tables built from them. The Host, marker, Origin, and Fetch
Metadata guards for dashboard reads and writes moved from the request
handler into hostnames.py beside the trust policy they enforce.
The dashboard's payload normalizers (snapshot and incidents envelopes,
inventory, collector settings, maintenance windows, host groups, topology)
moved from app.js into the api-contracts.js leaf, which now has its own
Node contract test covering every accepted shape and rejection. The host
trend and per-GPU history loaders, previously two copies of the same
bounded-backoff state machine spread over sixteen view fields, now share
the keyed-loader.js leaf, whose retry, single-flight, and stale-response
rules are tested with fake timers; selecting another host now also drops a
pending history retry. The custom-background code (IndexedDB storage,
container sniffing, size caps, WebP re-encoding) moved into the background-asset.js leaf over injected browser primitives, so its quality
bisection and shrink loop are tested in Node for the first time; format.js
gained the contract test the leaf pattern always required, and a repository
test now checks that every leaf is routed, loaded before app.js, and
tested. The attention panel's decisions — which active conditions a host
contributes, the shared-path and shared-storage grouping that consumes the
conditions it explains, per-host issues, and ranking — moved into the attention.js leaf with a contract test that pins every grouping rule
previously exercised only through the browser journey. The operator-facing
wording for collector failures, incident conditions, state labels, evidence
rows, and diagnosis guidance moved into the incident-text.js leaf, and a
repository test now requires one translation for every failure message the
probe or collector can emit and no dead translation. The app.js ceiling
ratcheted down to 5,850 lines.
Every HTTP server instance now requires the Bearer capability: the
unauthenticated server mode that only tests used is gone, GET /api/meta
no longer reports the constant authenticationRequired flag, and the
dashboard prompts for a missing capability without a /api/meta round trip.
GET /api/update now enforces the reader tier its manifest entry always
advertised (a marker-less request answers 403 UNTRUSTED_ORIGIN), and a
test proves every manifested tier against its handler.
Every GET route whose manifest query is empty now answers a query string
with 400 QUERY_NOT_ALLOWED; /api/snapshot, /api/events, /healthz,
and /readyz used to ignore one silently while five other routes rejected
it. The rule is enforced once in the dispatcher from the manifest, and a
test checks it for every route.
Every write route now validates its body through the same manifest the /api/metabody schemas are generated from, in one documented order:
shape and JSON-type problems are INVALID_SCHEMA, well-typed values
outside the published alias grammar, values, bounds, or text length are INVALID_SETTINGS, and only cross-field rules stay in the handlers. POST /api/settings/hosts and POST /api/probe therefore answer an unsafe
alias or unknown action with INVALID_SETTINGS (previously INVALID_SCHEMA), and a non-integer durationSeconds is INVALID_SCHEMA
(previously INVALID_SETTINGS). The manifest also publishes the text maximum of every reason, group, and condition-key field, nullable on incidentStartedAt, the three request-framing codes it had omitted, and
drops the one-route 500 INTERNAL_ERROR: a host-group persist failure is 503 SERVICE_UNAVAILABLE like every other configuration write. The
collector bounds and the dashboard duration set now have one owner
(api_manifest.py) instead of copies in web.py and inventory.py, and web.py shrank by 230 lines.
A managed unit that omits --access-token-file now exits 2 with a
message pointing at mocop service install instead of minting a token
nobody was shown; units generated since 0.9.0 always pass the flag.
The dashboard search box re-renders only the program-search panel and GPU
table, and GPU groups are keyed on the rows they show rather than on the
query text, so typing no longer rebuilds every visible group. Active
incidents are indexed by host on acceptance; the GPU history, capacity
candidate, owner, webhook endpoint, and All-servers resource views rebuild
their DOM only when the data they show changes; and the per-second tick
compares before it writes and skips cosmetic updates while the tab is hidden.
Documentation gives each fact one owner: CONTRIBUTING.md owns the quality
gate list, OPERATIONS.md the systemd unit and pinned install command,
PERFORMANCE.md the architecture thresholds, API.md the capability rules.
The architecture table lists every module and browser leaf, QUALITY.md
re-measures the runtime profile and coverage against the current tree, and
six ADRs record how later releases changed their decisions.
Fixed
The history database now actually shrinks after a bulk expiry. Python's sqlite3 steps PRAGMA incremental_vacuum once per execute(), so every
prune since persistence was introduced had returned exactly one page to the
filesystem. In steady state inserts reuse the freed pages and nothing shows,
but after a long downtime or a shortened retention the pages stay in the
file: on a copy of a 438 MB live history where two of its three days had
expired at once, 79,000 free pages (310 MB) remained, and because the
startup cap check counts free pages, lowering persistence.max_bytes below
that high-water mark refused to start even though the live data fitted.
Startup now rebuilds a file that has free pages with VACUUM before the cap
is compared (0.5 s for that copy, which came back at 115 MB), and each
60-second prune reclaims up to 8 MiB online in a bounded number of pragma
calls, because CPython 3.11's sqlite3 frees one page per call regardless
of how the cursor is consumed.
Restoring history at startup no longer scans the whole SQLite database: the
newest points of each host and GPU are read through the tables' primary
keys instead of a window function that SQLite executed as a full scan plus
temporary sort. Against a copy of a 476 MB live history (2 million GPU
points) the restore fell from 2.85 s to 0.4 s with identical results, and
from 9–18 s on the live host where the writer thread contended for the
lock; no schema change is involved.
mocop service install could not verify a service publishing the complete /api/meta manifest: the installer's liveness probe rejected any manifest
longer than 4 KiB, so installing a build with the complete manifest rolled
back with SERVICE_UNHEALTHY. The cap is now 64 KiB and a test pins the
real manifest under it with headroom. The verification window also grew from 8 to 60
seconds, because the service restores its retained history before binding
and a 476 MB history measured 9–18 s on a loaded host.
The capacity-watch notification and title marker now fire while the tab is
in the background. They were evaluated inside the requestAnimationFrame
render, which browsers pause for hidden documents, so the alert the feature
exists for arrived only when the operator returned to the tab.
The release pill now starts on the polling recovery path, not only when the
first snapshot arrives over the live stream.
The owners dialog shows when its data was last collected instead of a
static "实时更新" label.
The incident detail dialog no longer overwrites a reason the operator is
typing every time a snapshot arrives, and a condition that recovers while
the dialog is open is shown as resolved with its actions withdrawn instead
of closing the dialog under the cursor.
Fleet rail items are valid HTML: the rows inside each <button> are now
phrasing content, so assistive technology no longer has to flatten them.
Two collector failure messages reached operators in English because the
dashboard had no translation for them: Local SSH client could not be started and nvidia-smi output was malformed. A stale translation for a
message the collector no longer emits was removed at the same time.
Removed
GET /api/service and POST /api/settings/poll-interval, deprecated in
0.9.0. Use the capabilities block of GET /api/meta and POST /api/settings/collector (any non-empty subset) instead.
Dashboard fallbacks for server shapes that cannot occur (older snapshot,
collector, maintenance, and capability payloads, the native EventSource
transport, browser-side threshold defaults, and the createImageBitmap
and indexedDB feature probes), plus unused leaf exports, dead CSS, and
Python methods, protocol duck-typing, and Linux flag probes with no callers
or purpose.