Skip to content

Releases

Mocop 0.13.0

Choose a tag to compare

@github-actions github-actions released this 19 Sep 17:45
f773eb4

Added

  • Hysteresis for numeric conditions: incidents.recovery_margin (default 5).
    An open condition stays open until its reading falls the margin below the
    threshold that opened it, and a critical one stays critical until it falls
    the margin below the critical line; readings inside the band carry
    belowThreshold: true in GET /api/incidents and webhook events and read
    回落中 on the dashboard. They hold an open condition but never open or
    confirm one, so a dip also breaks a confirmation streak. On the live fleet
    one GPU's VRAM at 90% ± 5 crossed the line 216 times in six hours without
    ever falling under 85%, and gpu_memory alone produced 241 of the day's
    352 openings; each such reading is now one incident. The idle-VRAM
    condition applies the margin to both sides of its claim.
  • A situation brief: GET /api/brief?hours=N and mocop brief [--hours N] [--json]. The operator's morning scan as one document, in reading order:
    fleet status by the dashboard's badge rule; the actionable conditions worst
    first with their correlations, conditions on one network filesystem
    collapsed into a single entry across its hosts; openings, recoveries, and
    escalations over the window with the conditions that opened three or more
    times (on the live fleet this named one VRAM alert that had flapped 39
    times in a day); idle GPUs by host from the capacity matcher; per-owner
    GPU-hours with the share their reservation sat idle, flagging owners idle
    at least half the window across a working day of GPU-hours; and the hosts
    inside maintenance windows. Every number comes from a projection the
    dashboard already shows, so the brief cannot disagree with the screen, and
    the transition feed's coverage is reported (coveredFromAt) rather than a
    partial count being presented as a total. GET /api/incidents now carries
    eventCapacity, the size of its transition ring.
  • Optional authentication: "none" for direct dashboard and API access without
    a token. New tabs and reloads open immediately; CLI and service lifecycle
    commands support the same mode. Bearer authentication remains the default.
    Direct access grants the operator role to every reachable client while
    retaining Host, Origin, Fetch Metadata, and request validation.

Fixed

  • The header's last-sync label no longer collapses to a one-character
    column around 800 px. Flex shrink was squeezing .sync-time to ~11 px,
    so CJK wrapped glyph-by-glyph; the control now keeps its width, and the
    header wraps as a row before that width. Remaining 10px type uses
    --text-xs, so compact density actually reaches the GPU table and
    filters.
  • flush() on the history writer reports a dropped write even when the
    write and the flush barrier landed in different batches. The writer
    batches whatever is queued when it wakes, so a write could be processed
    (and dropped) alone and the barrier that followed it committed trivially;
    the barrier now answers from the drop counter. This was the intermittent
    CI failure of the prune-failure test on Python 3.10.
  • A single failed probe no longer closes a host's whole GPU process
    inventory. On a live deployment behind an SSH relay, every transient
    failure emitted a hidden stopped for each process, re-seeded them all a
    few seconds later, reset every first_seen_at (the dashboard's observed
    runtime), and flooded the retained timeline with these pairs. The inventory
    now follows the same staleness definition as the rest of a host's data: it
    is kept as a blind spot until the host has failed for
    collection_stale_cycles consecutive cycles, then closed at its last
    confirmed sample. A real process change across a shorter gap is attributed
    to the first sample after it. The usage rollup counts a failing host's live
    processes only up to that last confirmed sample, so a blind gap is never
    billed as occupancy.
  • /api/usage no longer loses the occupancy of runs whose started edge has
    left the retained event window. On a live deployment a 24-hour report
    dropped 644 records, 592 of them orphan stopped events from GPUs that
    churn hundreds of processes a day (the restore keeps the last 500
    transitions per device). Every transition now carries firstSeenAt, this
    monitor's first observation of the process on the device — a genuine
    GPU-occupancy observation, unlike the process start time — so an orphan
    stop is a complete run on its own; the value rides inside the existing
    workload_json column, keeping the process table's nine-column contract,
    and the dashboard's GPU timeline states how long each finished run held
    the device.
  • /api/usage reports partialGpus: the devices whose retained transition
    timeline is full yet begins inside the requested window, so their occupancy
    before that point is missing from the totals. earliestDataAt alone could
    not show this because it is the earliest record across all devices, and a
    quiet device made a report look complete while a busy one's timeline
    reached back only an hour; the dashboard's owner summary names the count.
  • Incidents that were open when the service stopped resume their generation
    at startup instead of opening again. A live deployment had re-emitted
    opened for every persisting condition after each restart — 17 per restart
    for 14 full filesystems and 3 unreachable hosts — which duplicated webhook
    deliveries, reset firstObservedAt, and filled the transition log with
    restart noise. With history persistence, each condition's latest persisted
    transition (read across the whole retained table, not just the display
    window) decides whether it was open, its last opened supplies
    firstObservedAt, the first live sample confirms or recovers it under the
    usual rules, bound acknowledgements keep applying, and webhook workers are
    primed so the eventual resolved still pairs with the opened an earlier
    process delivered.
  • The startup VACUUM that returns expired pages is best-effort: when it
    cannot run (typically a full disk, since it needs temporary space up to the
    file's size) the service starts anyway, reclaims what the bounded online
    path can, and leaves the condition to the persistence status it already
    reports, instead of refusing to start over it.
  • The README and the API reference no longer describe a no_nvidia_smi host
    status: servers[].status is pending, online, unreachable, or
    error, and a host without nvidia-smi is online with the message
    nvidia-smi is unavailable and an empty gpus array, as the probe test
    has always pinned. The Chinese README gains the same sentence.
  • The attention panel marks a resource condition whose host is not online
    with (离线前): the service keeps such conditions open across failed
    probes, so their figures date from the last successful sample. On a live
    deployment a node unreachable for 533 consecutive probes was listed with
    three VRAM figures that were two days old and read as current.
  • A GPU that vanishes from an online host (an XID fault or a bus drop takes
    the device out of nvidia-smi) now closes its confirmed process occupancy
    at the last process sample, the way a failed process query already did.
    The rollup previously had to drop that occupancy as an unanchorable start,
    undercounting the owner and raising droppedRecords.
  • Restored process starts on a host that is unreachable across a restart are
    closed once that host turns stale. Reconciliation of the transitions read
    from the history file waited for the device's first live process sample; a
    host that was already down when the service restarted and stayed down never
    delivered one, so a run confirmed up to the shutdown was dropped from
    /api/usage as an unanchorable start and its started lingered in the
    timeline until it fell out of the ring. Staleness — collection_stale_cycles
    consecutive failures — now closes such starts hidden at the device's last
    restored GPU sample, where observation ended, and an online host whose GPU
    list no longer contains the device closes them the same way; a device that
    is observed but skips a process sample still waits for its live table. The
    pure transition arithmetic (process_transitions.py) left service.py so
    this fits under the ratchet, and the ceiling dropped from 2175 to 2150.
  • A process seeded and closed in the same second no longer comes back from
    the history file as an open start. The process table has no sequence
    column, and the restore ordered every same-second stopped before the
    started, which is right for a reused PID (the collector emits
    stop-then-start) but inverted the zero-length run a host leaves when it
    answers one sample and then goes stale. On a live deployment the two
    unreachable hosts' "orphan" starts were all 28 such pairs, and on online
    hosts a phantom start paired with the next re-seed of the same PID into a
    run across the blind gap: replaying the live file through the store, a
    48-hour rollup went from 698 dropped records and 1502 GPU-hours to 249 and
    1394. The restore now uses the one distinction the collector guarantees —
    a PID reuse carries two different workload start times — and orders every
    other same-second pair start-then-stop.
  • The history writer prunes expired records once per minute while idle, as
    documented, instead of on every 100 ms wake-up. The short queue timeout
    exists so close() is answered promptly; treating it as the prune cadence
    ran the four retention deletes and the bounded page reclaim about ten times
    a second on an idle deployment (0.15 % of a core on a 386 MB file — cheap in
    steady state, but 600 times the intended reclaim rate while a backlog
    drained).
  • A history writer that has stopped (it could not open the database, or a
    corrupt internal record ended it) keeps that cause in the persistence
    status. Producers used to keep queueing behind the dead thread until t...
Read more

Mocop 0.12.0

Choose a tag to compare

@github-actions github-actions released this 05 Sep 16:29
5a7941e

Added

  • GPU task rows now lead with the real entry point instead of a bare
    interpreter name: when argv0 is python, torchrun, bash or another
    generic launcher, the module after -m or the first script argument
    becomes the row title, with the interpreter and the detected Python
    environment (conda env, project venv, uv tool, poetry virtualenv) shown
    as chips. The fleet-wide program search, the per-card top-process label
    and the name sort follow the same display name.
  • The workload identity pass now also reports each PID's host-side
    footprint — cumulative CPU seconds and resident memory from the same
    /proc reads it already performs — and the task row renders it as
    "CPU 均值 N 核 · 主机内存 N GiB", so a stalled data loader is visible
    without shelling into the node. Snapshot workload objects gain
    cpu_seconds and rss_mib (both nullable).
  • The truncated command line in a task row expands in place on click
    instead of requiring copy-and-paste to read the training config.
  • mocop deploy --help, mocop migrate --help, and mocop service --help
    now describe every flag and action, and help output names the program
    mocop instead of __main__.py. The operations runbook gained a command,
    flag, and exit-code reference (0, 1, 2, 75).
  • GET /api/meta is now a complete machine-readable contract: every GET
    route lists its accepted query parameters with type, bounds, and default,
    every write lists its body field schema and bodyLimitBytes, every route
    names its responseType, and the document also publishes conventions,
    writer requirements, the error-code catalog, and a documentation URL for
    the running release. The manifest is generated from the same declarative
    table (api_manifest.py) the handlers validate against, replacing five
    hand-written query parsers.
  • mocop --version prints the package version. init, deploy, migrate,
    and every service action accept --json and write one {ok, code?, ...}
    document, matching config check --json and doctor --json. Refusals stay
    on that envelope so an agent never has to parse English from stderr.
  • mocop api PATH asks the running service any public or authenticated GET
    question from the monitor host without knowing the listen address, the
    capability file, or the Bearer header: both come from the configuration
    directory, the response body is written to stdout unchanged (/api/events
    streams until interrupted), and the exit code follows the CLI table (0 for
    a 2xx, 1 for any other status or an unreachable service, 2 for usage or
    configuration problems) with a JSON {error, code} envelope on every
    failure. Reader and writer routes are refused with DASHBOARD_ONLY because
    their marker header and same-origin checks belong to the dashboard.
  • GET /api/meta capabilities now include updateSupported. Excess
    connections answer 503 CONNECTION_LIMIT as JSON instead of an empty body.
  • Error envelopes carry field whenever exactly one query parameter or body
    field is at fault — an unknown parameter's name, a malformed or out-of-bounds
    parameter, a wrongly typed or unaccepted body field, and the single-field
    cross checks on maintenance, host-group, and incident-action writes — so an
    agent maps a rejection back to its input without parsing the message.
  • 403 AUTHENTICATION_REQUIRED responses carry a hint naming the header to
    send and where the capability file lives, plus the documentation URL, so
    an agent can recover from a cold start without out-of-band knowledge.
  • mocop config check --json writes the validation report as one JSON
    document on stdout (also for a rejected configuration), matching
    doctor --json.
  • GET /api/capacity?gpus=N&min_vram_gib=G&model=M answers the placement
    question from one bounded authenticated call: ranked same-host, same-model
    GPU groups with the available devices, deficit, minimum free VRAM, and the
    maintenance and health exclusions, using exactly the dashboard's matching
    rules. A shared fixture pins the browser leaf and the new capacity.py to
    one ranking, and the browser leaf now orders hosts by code point instead of
    the locale so both agree everywhere.
  • tests/gpu_tasks_test.mjs now runs in CI: the workflow globs every browser
    leaf and contract test instead of maintaining a list that had drifted.
  • AGENTS.md at the repository root is the entry point AI coding agents read
    by convention: the reading order, the drift tests that enforce each
    contract, the boundaries that must hold, and how to operate a deployment,
    each pointing at the document that owns the detail.

Changed

  • doctor --json now includes the local host in hosts[] (with
    local: true) and adds ok alongside the existing status field.
    --once / --strict with a subcommand are rejected as usage errors.
    Inventory payloads must publish infrastructureHosts and
    sshDiscoveryMode; the dashboard no longer defaults those fields.
  • Workload record parsing moved from probe.py into a new workloads.py
    module, and the GPU task projections moved from app.js into the
    gpu-tasks.js leaf (ADR-0021);
    both budgets ratcheted down. The per-owner GPU occupancy rollup behind
    GET /api/usage moved from StateStore into a pure usage.py module that
    takes a copied timeline and is unit-tested on its own; service.py
    ratcheted from 2,725 to 2,425 lines. The test-only
    parse_linux_resource_payload wrapper left probe.py for the test module,
    and the doctor and the route resolver share one ssh -G option resolver in
    ssh_topology.py instead of two copies that had already drifted on how a
    bare option line is read. Configuration loading moved out of config.py
    into config_loader.py (load_config, load_private_config,
    resolve_config_path, and friends), and the single 575-line parser became
    one function per configuration section, run in the same order so every
    error message and its precedence are unchanged; config.py keeps the
    schema — limits, typed sections, validators — and ratchets from 1,550 to
    375 lines. The same split applies to the HTTP contract: api_schema.py
    owns the query and body field types and the two validators, api_manifest.py
    the route tables built from them. The Host, marker, Origin, and Fetch
    Metadata guards for dashboard reads and writes moved from the request
    handler into hostnames.py beside the trust policy they enforce.
  • The dashboard's payload normalizers (snapshot and incidents envelopes,
    inventory, collector settings, maintenance windows, host groups, topology)
    moved from app.js into the api-contracts.js leaf, which now has its own
    Node contract test covering every accepted shape and rejection. The host
    trend and per-GPU history loaders, previously two copies of the same
    bounded-backoff state machine spread over sixteen view fields, now share
    the keyed-loader.js leaf, whose retry, single-flight, and stale-response
    rules are tested with fake timers; selecting another host now also drops a
    pending history retry. The custom-background code (IndexedDB storage,
    container sniffing, size caps, WebP re-encoding) moved into the
    background-asset.js leaf over injected browser primitives, so its quality
    bisection and shrink loop are tested in Node for the first time; format.js
    gained the contract test the leaf pattern always required, and a repository
    test now checks that every leaf is routed, loaded before app.js, and
    tested. The attention panel's decisions — which active conditions a host
    contributes, the shared-path and shared-storage grouping that consumes the
    conditions it explains, per-host issues, and ranking — moved into the
    attention.js leaf with a contract test that pins every grouping rule
    previously exercised only through the browser journey. The operator-facing
    wording for collector failures, incident conditions, state labels, evidence
    rows, and diagnosis guidance moved into the incident-text.js leaf, and a
    repository test now requires one translation for every failure message the
    probe or collector can emit and no dead translation. The app.js ceiling
    ratcheted down to 5,850 lines.
  • Every HTTP server instance now requires the Bearer capability: the
    unauthenticated server mode that only tests used is gone, GET /api/meta
    no longer reports the constant authenticationRequired flag, and the
    dashboard prompts for a missing capability without a /api/meta round trip.
  • GET /api/update now enforces the reader tier its manifest entry always
    advertised (a marker-less request answers 403 UNTRUSTED_ORIGIN), and a
    test proves every manifested tier against its handler.
  • Every GET route whose manifest query is empty now answers a query string
    with 400 QUERY_NOT_ALLOWED; /api/snapshot, /api/events, /healthz,
    and /readyz used to ignore one silently while five other routes rejected
    it. The rule is enforced once in the dispatcher from the manifest, and a
    test checks it for every route.
  • Every write route now validates its body through the same manifest the
    /api/meta body schemas are generated from, in one documented order:
    shape and JSON-type problems are INVALID_SCHEMA, well-typed values
    outside the published alias grammar, values, bounds, or text length are
    INVALID_SETTINGS, and only cross-field rules stay in the handlers.
    POST /api/settings/hosts and POST /api/probe therefore answer an unsafe
    alias or unknown action with INVALID_SETTINGS (previously
    INVALID_SCHEMA), and a non-integer durationSeconds is INVALID_SCHEMA
    (previously INVALID_SETTINGS). The manifest also publishes the text
    maximum of every reason, group, and condition-key field, nullable on
    incidentStartedAt, the three request-framing codes it had omitted, and
    drops the one-route `500 I...
Read more

Mocop 0.11.0

Choose a tag to compare

@github-actions github-actions released this 25 Aug 12:22
60b9348

Added

  • Added an opt-in release-currency pill to the dashboard header
    (ADR-0026). updates.mode: "check"
    polls the hardcoded official repository on a bounded interval and shows
    whether the running release is current; "self-update" also enables a
    one-click apply that downloads the release wheel, verifies it against the
    release SHA-256 manifest, installs it with the environment's own pip or
    uv, proves the installed version through the target interpreter, and
    restarts through the supervised-restart path before the page reloads
    itself. The shipped default stays "off" with no outbound requests; the
    browser can never name a version, and a failed attempt keeps the running
    version serving with a visible manual-recovery hint
    (GET /api/update, POST /api/update/apply).

Mocop 0.10.2

Choose a tag to compare

@github-actions github-actions released this 25 Aug 05:22
43255b6

Changed

  • Service-install health verification no longer transmits the dashboard
    capability. It confirms liveness with an unauthenticated request the
    service must reject, closing a narrow window where the token could reach a
    process that had bound the loopback port while the real unit crash-looped,
    and it now also flags a service that started without a readable token.
  • Webhook status reports suppressedDeliveries separately from
    droppedDeliveries, so a deliberately withheld recovery (an unpaired
    resolved event, for example inside a maintenance window) no longer inflates
    the failure count or marks an endpoint unhealthy.

Fixed

  • Fixed the generated systemd unit's EnvironmentFile= line, which a prior
    change had quoted; systemd treats the whole line as one unquoted path, so
    the quoted form failed the absolute-path check and every install silently
    dropped its webhook credentials. The path is now emitted bare (verified
    with systemd-analyze), and paths with spaces load correctly.
  • A malformed nvidia-smi payload is now treated as unavailable GPU
    telemetry. It previously degraded to an authoritative empty inventory that,
    combined with identity-absence recovery, silently resolved every per-GPU
    hardware incident within two cycles without opening an availability alert.
  • Resolved-topology grouping now follows ADR-0022 exactly: only a trailing
    numeric segment (training-1/training-2) forms an inferred group. Shared
    word prefixes (paris-web/paris-db) and embedded model numbers
    (gpu-a100) no longer group hosts that are not one numbered fleet;
    explicit host_groups remain the override.
  • mocop migrate no longer rejects deploying the monitor onto a machine that
    is already a topology node when the source has no local host; the topology
    collision guard now fires only for an actual local-host rename.
  • The dashboard capacity watch reconciles with shared browser storage on
    every evaluation, so a watch stopped or replaced in one tab is no longer
    resurrected or overwritten by another tab. A wall-clock rollback no longer
    freezes its notification cooldown, and a watch on a long (up to 256-char)
    GPU model name now saves instead of failing silently.
  • GPU severity escalation now requires consecutive confirmable samples,
    matching incident opening: two critical observations split by a probe-
    failure gap no longer accumulate into an escalation.

Mocop 0.10.1

Choose a tag to compare

@github-actions github-actions released this 25 Aug 04:14
df08fc4

Fixed

  • One exotic literal Host entry in the OpenSSH configuration (an IPv6
    literal, a token carrying : or #) no longer stops discovery for every
    monitored host: unsafe aliases now leave the candidate set with a visible
    scan warning while explicit hosts stay monitored (ADR-0022).
  • Proxy-chain expansion for resolved topology is now iterative, so a
    pathologically deep ProxyJump chain can no longer overflow the
    interpreter stack and stop the collector.
  • mocop migrate now refuses a new local alias that already appears in the
    configured topology with a direct explanation instead of a downstream
    schema error about self-links.
  • After the webhook pairing table hits its capacity bound, unpaired
    recoveries are delivered instead of suppressed, so a long-lived incident
    whose pairing record was evicted can still resolve on the receiver.

Mocop 0.10.0

Choose a tag to compare

@github-actions github-actions released this 25 Aug 03:22
aa2d37e

Added

  • Added mocop deploy as the fresh-server fast path. After package installation, one
    non-overwriting command configures the current host, enables safe SSH alias/topology
    discovery, creates a fresh capability, and installs and verifies the user service.
  • Added non-destructive mocop migrate configuration generation for moving the
    monitor to another machine. It rebinds local identity, drops reported machine-bound
    policy, upgrades bounded topology discovery, preserves host admission by default,
    refuses existing targets/copied capabilities, and never copies secrets or history.
  • Added opt-in, cached SSH topology discovery from bounded ssh -G resolution.
    Effective ProxyJump chains and common SSH-backed ProxyCommand forms identify
    infrastructure aliases, keep automatically discovered jump hosts out of the probe
    inventory, build the display tree, and group targets by their closest known hop,
    with shared numbered alias prefixes as a fallback for direct targets.
    Explicit inventory, exclusions, groups, and configured topology remain authoritative;
    group metadata may predeclare auto-discovered aliases without authorizing probes;
    raw commands, users, and addresses are never published.
  • Added HTTPS-only trusted_web_hosts suffix entries (for example,
    *.preview.example) for ephemeral Host-rewriting dashboard proxies. Suffix entries
    authorize browser Origin only; exact backend Host validation and Bearer
    authentication remain mandatory.
  • Added an explicit dashboard token prompt for bare and forwarded URLs. Submitted
    credentials are retained only after successful API authentication; malformed or
    rejected tokens remain unstored and do not trigger automatic retries.
  • Added a capacity watch to the capacity matcher: one saved demand
    (count/model/per-GPU free VRAM) is re-evaluated on every accepted snapshot in
    the browser. The ready edge shows an in-page banner and title marker, sends an
    opt-in browser notification with a 60-second cooldown, re-arms only after the
    demand stops being satisfied, and persists per browser. No new API or SSH
    work is involved.
  • Added one-click ssh <alias> copy to capacity candidates and the GPU detail
    dialog. The copied command uses the configured OpenSSH alias; display names
    stay presentation-only.

Changed

  • Moved the runtime package to src/mocop (ADR-0025).
    Package names and wheel contents are unchanged. A source checkout still runs
    tests without installation: tests/__init__.py prepends src to sys.path,
    and discovery now names the repository root (python3 -m unittest discover -s tests -t .). Contributor commands reference src/mocop paths.

Fixed

  • Applied configured host display names consistently across node navigation, resource
    inventory, heatmap, topology, GPU details, search placement, incidents, and alerts.
  • Fixed managed-service shutdown when no host-specific probe timeout is configured.
  • Kept the nvidia-smi fallback query from emitting an empty success line when the
    combined query is unsupported and the base query returns no rows.
  • Fixed the collection script for hosts without block devices: an empty
    /sys/block glob aborted the system awk pass and dropped every core CPU,
    memory, load, network, and I/O row, permanently misreporting a healthy host
    as failed. Exotic mounts (for example FUSE backends reporting - capacity),
    a malformed nvidia-smi row, and malformed pressure rows now degrade only
    their own section instead of rejecting the whole host sample.
  • Restoring SQLite history now drops rows whose required percentage columns are
    NULL (foreign writers or partial corruption) instead of crashing the
    collector on every start until the row was deleted by hand.
  • Non-ASCII Bearer credentials now fail closed with the documented 403 instead
    of crashing the comparison and resetting the connection; trusted-host and
    Origin checks now canonicalize the absolute DNS form (trailing dot).
  • The generated unit now quotes EnvironmentFile= so configuration paths with
    spaces load webhook credentials instead of being silently ignored, and
    service health polling tolerates malformed HTTP responses during startup.
  • Resolved incident transitions now always reach the webhook queue, so an
    alert that recovers inside a maintenance window or silence no longer leaves
    the receiver hanging. Opening a condition now requires consecutively
    confirmable samples, so two isolated spikes separated by a probe-failure gap
    can no longer open an incident. Conditions bound to a GPU identity that left
    a fully observed inventory (a replaced card) now recover instead of pinning
    ghost alerts until a restart.
  • The dashboard now establishes its event stream and corrects the connection
    badge after recovering from a failed first snapshot, rejects structurally
    broken snapshots before they can freeze rendering, and backs off snapshot
    polling up to 30 seconds during an outage.

Mocop 0.9.0

Choose a tag to compare

@github-actions github-actions released this 16 Aug 08:59
5873754

Mocop 0.9.0 improves GPU process observability, access control, and operational reliability. Upgrading is recommended for all 0.8.x deployments.

Highlights

  • GPU process visibility: Adds fleet-wide and per-server search by program, command, PID, owner, workload, queue, and namespace. The GPU detail view now supports attribution filters, sorting, runtime and memory inspection, and copy actions.
  • Workload attribution and usage: Extends attribution to Slurm, Kubernetes, Docker, and Podman. The new GET /api/usage endpoint reports per-owner GPU occupancy and classified idle share.
  • Monitoring and diagnostics: Adds Linux PSI, GPU hardware health, transport-retry visibility, and persistence status. New operational commands include mocop config check, mocop doctor --probe, mocop doctor --profile, and mocop --once --strict.
  • Reliability and efficiency: Improves independent host scheduling, SSH timeout and retry handling, process-sampling cadence, bounded SQLite retention, and webhook delivery. Queues, outputs, retained history, and browser rendering now have explicit limits.

Security and compatibility

  • The HTTP API is now v2. All API, SSE, and /metrics requests require the per-installation Bearer capability, except for /api/meta, /healthz, and /readyz.
  • The internal collection protocol is now MONITOR_V8; V4–V7 payloads are no longer accepted. The collector script and parser are distributed together, so reinstall the package and restart the service during upgrade.
  • The default listener remains restricted to the loopback interface. Remote access requires an authenticated TLS reverse proxy or a private VPN.

Upgrade

If SQLite persistence is enabled, first back up the configuration, Bearer capability, and state directory as described in the operations runbook. Then run:

uv tool install --force "git+https://github.com/ChangWinde/mocop.git@v0.9.0"
mocop config check
mocop service install

Verification

  • Python 3.10–3.14
  • 467 unit and contract tests
  • Chrome end-to-end coverage
  • 88% branch coverage
  • Release wheel with SHA256SUMS

See the Changelog for the complete change record.

Mocop 0.8.0

Choose a tag to compare

@ChangWinde ChangWinde released this 16 Aug 08:59

Historical initial public release from 2026-08-09. The attached wheel was rebuilt from the immutable release tag and smoke-tested in an isolated Python 3.10 environment.

Historical CI note: the immutable 0.8.0 browser smoke test contains a cold-start timing assertion that reports connection delayed on current GitHub runners. Its Python 3.10–3.14 jobs and reproduced wheel pass; the browser startup and cleanup races are fixed in v0.9.0. Use v0.9.0 for current deployments.

Added

  • Initial public release of Mocop.
  • GPU/VRAM-first dashboard, scheduling heatmap and collapsed per-server inventory.
  • CPU, memory, Swap, disk, network, trends, incident history and safe CSV export.
  • Explicit config-based server allowlists with portable config discovery.
  • A safe bundled default and a publication-safe example configuration.
  • A configurable hard limit for combined SSH stdout and stderr.
  • Safe mocop init bootstrap and explicit user-level systemd lifecycle commands.
  • Python 3.10–3.14 CI, populated browser smoke coverage and security contracts.

Fixed

  • Existing failed-host retry deadlines are rebased when the runtime polling cadence changes.
  • Runtime cadence changes remain authoritative during initial snapshot and SSE races.

Security

  • Oversized remote process output now terminates the SSH process group instead of growing an unbounded in-memory buffer.
  • SSH targets, command arguments, output, timeouts and browser writes are validated and bounded.