Mocop 0.13.0

Added

• Hysteresis for numeric conditions: incidents.recovery_margin (default 5).
  An open condition stays open until its reading falls the margin below the
  threshold that opened it, and a critical one stays critical until it falls
  the margin below the critical line; readings inside the band carry
  belowThreshold: true in GET /api/incidents and webhook events and read
  回落中 on the dashboard. They hold an open condition but never open or
  confirm one, so a dip also breaks a confirmation streak. On the live fleet
  one GPU's VRAM at 90% ± 5 crossed the line 216 times in six hours without
  ever falling under 85%, and gpu_memory alone produced 241 of the day's
  352 openings; each such reading is now one incident. The idle-VRAM
  condition applies the margin to both sides of its claim.
• A situation brief: GET /api/brief?hours=N and mocop brief [--hours N] [--json]. The operator's morning scan as one document, in reading order:
  fleet status by the dashboard's badge rule; the actionable conditions worst
  first with their correlations, conditions on one network filesystem
  collapsed into a single entry across its hosts; openings, recoveries, and
  escalations over the window with the conditions that opened three or more
  times (on the live fleet this named one VRAM alert that had flapped 39
  times in a day); idle GPUs by host from the capacity matcher; per-owner
  GPU-hours with the share their reservation sat idle, flagging owners idle
  at least half the window across a working day of GPU-hours; and the hosts
  inside maintenance windows. Every number comes from a projection the
  dashboard already shows, so the brief cannot disagree with the screen, and
  the transition feed's coverage is reported (coveredFromAt) rather than a
  partial count being presented as a total. GET /api/incidents now carries
  eventCapacity, the size of its transition ring.
• Optional authentication: "none" for direct dashboard and API access without
  a token. New tabs and reloads open immediately; CLI and service lifecycle
  commands support the same mode. Bearer authentication remains the default.
  Direct access grants the operator role to every reachable client while
  retaining Host, Origin, Fetch Metadata, and request validation.

Fixed

• The header's last-sync label no longer collapses to a one-character
  column around 800 px. Flex shrink was squeezing .sync-time to ~11 px,
  so CJK wrapped glyph-by-glyph; the control now keeps its width, and the
  header wraps as a row before that width. Remaining 10px type uses
  --text-xs, so compact density actually reaches the GPU table and
  filters.
• flush() on the history writer reports a dropped write even when the
  write and the flush barrier landed in different batches. The writer
  batches whatever is queued when it wakes, so a write could be processed
  (and dropped) alone and the barrier that followed it committed trivially;
  the barrier now answers from the drop counter. This was the intermittent
  CI failure of the prune-failure test on Python 3.10.
• A single failed probe no longer closes a host's whole GPU process
  inventory. On a live deployment behind an SSH relay, every transient
  failure emitted a hidden stopped for each process, re-seeded them all a
  few seconds later, reset every first_seen_at (the dashboard's observed
  runtime), and flooded the retained timeline with these pairs. The inventory
  now follows the same staleness definition as the rest of a host's data: it
  is kept as a blind spot until the host has failed for
  collection_stale_cycles consecutive cycles, then closed at its last
  confirmed sample. A real process change across a shorter gap is attributed
  to the first sample after it. The usage rollup counts a failing host's live
  processes only up to that last confirmed sample, so a blind gap is never
  billed as occupancy.
• /api/usage no longer loses the occupancy of runs whose started edge has
  left the retained event window. On a live deployment a 24-hour report
  dropped 644 records, 592 of them orphan stopped events from GPUs that
  churn hundreds of processes a day (the restore keeps the last 500
  transitions per device). Every transition now carries firstSeenAt, this
  monitor's first observation of the process on the device — a genuine
  GPU-occupancy observation, unlike the process start time — so an orphan
  stop is a complete run on its own; the value rides inside the existing
  workload_json column, keeping the process table's nine-column contract,
  and the dashboard's GPU timeline states how long each finished run held
  the device.
• /api/usage reports partialGpus: the devices whose retained transition
  timeline is full yet begins inside the requested window, so their occupancy
  before that point is missing from the totals. earliestDataAt alone could
  not show this because it is the earliest record across all devices, and a
  quiet device made a report look complete while a busy one's timeline
  reached back only an hour; the dashboard's owner summary names the count.
• Incidents that were open when the service stopped resume their generation
  at startup instead of opening again. A live deployment had re-emitted
  opened for every persisting condition after each restart — 17 per restart
  for 14 full filesystems and 3 unreachable hosts — which duplicated webhook
  deliveries, reset firstObservedAt, and filled the transition log with
  restart noise. With history persistence, each condition's latest persisted
  transition (read across the whole retained table, not just the display
  window) decides whether it was open, its last opened supplies
  firstObservedAt, the first live sample confirms or recovers it under the
  usual rules, bound acknowledgements keep applying, and webhook workers are
  primed so the eventual resolved still pairs with the opened an earlier
  process delivered.
• The startup VACUUM that returns expired pages is best-effort: when it
  cannot run (typically a full disk, since it needs temporary space up to the
  file's size) the service starts anyway, reclaims what the bounded online
  path can, and leaves the condition to the persistence status it already
  reports, instead of refusing to start over it.
• The README and the API reference no longer describe a no_nvidia_smi host
  status: servers[].status is pending, online, unreachable, or
  error, and a host without nvidia-smi is online with the message
  nvidia-smi is unavailable and an empty gpus array, as the probe test
  has always pinned. The Chinese README gains the same sentence.
• The attention panel marks a resource condition whose host is not online
  with （离线前）: the service keeps such conditions open across failed
  probes, so their figures date from the last successful sample. On a live
  deployment a node unreachable for 533 consecutive probes was listed with
  three VRAM figures that were two days old and read as current.
• A GPU that vanishes from an online host (an XID fault or a bus drop takes
  the device out of nvidia-smi) now closes its confirmed process occupancy
  at the last process sample, the way a failed process query already did.
  The rollup previously had to drop that occupancy as an unanchorable start,
  undercounting the owner and raising droppedRecords.
• Restored process starts on a host that is unreachable across a restart are
  closed once that host turns stale. Reconciliation of the transitions read
  from the history file waited for the device's first live process sample; a
  host that was already down when the service restarted and stayed down never
  delivered one, so a run confirmed up to the shutdown was dropped from
  /api/usage as an unanchorable start and its started lingered in the
  timeline until it fell out of the ring. Staleness — collection_stale_cycles
  consecutive failures — now closes such starts hidden at the device's last
  restored GPU sample, where observation ended, and an online host whose GPU
  list no longer contains the device closes them the same way; a device that
  is observed but skips a process sample still waits for its live table. The
  pure transition arithmetic (process_transitions.py) left service.py so
  this fits under the ratchet, and the ceiling dropped from 2175 to 2150.
• A process seeded and closed in the same second no longer comes back from
  the history file as an open start. The process table has no sequence
  column, and the restore ordered every same-second stopped before the
  started, which is right for a reused PID (the collector emits
  stop-then-start) but inverted the zero-length run a host leaves when it
  answers one sample and then goes stale. On a live deployment the two
  unreachable hosts' "orphan" starts were all 28 such pairs, and on online
  hosts a phantom start paired with the next re-seed of the same PID into a
  run across the blind gap: replaying the live file through the store, a
  48-hour rollup went from 698 dropped records and 1502 GPU-hours to 249 and
  1394. The restore now uses the one distinction the collector guarantees —
  a PID reuse carries two different workload start times — and orders every
  other same-second pair start-then-stop.
• The history writer prunes expired records once per minute while idle, as
  documented, instead of on every 100 ms wake-up. The short queue timeout
  exists so close() is answered promptly; treating it as the prune cadence
  ran the four retention deletes and the bounded page reclaim about ten times
  a second on an idle deployment (0.15 % of a core on a 386 MB file — cheap in
  steady state, but 600 times the intended reclaim rate while a backlog
  drained).
• A history writer that has stopped (it could not open the database, or a
  corrupt internal record ended it) keeps that cause in the persistence
  status. Producers used to keep queueing behind the dead thread until the
  4096 slots ran out and then overwrite lastError with history write queue is full, so the status suggested back-pressure where the writer had in
  fact stopped; writes are now counted as dropped immediately and the writer's
  own message stands. The writer's failure paths — open failure, crash,
  prune failure, and a full disk whose recovery prune also fails — are under
  test for the first time.

Changed

• Dashboard design pass against published dark-UI practice (Linear's
  luminance-stacked system, WCAG 2.2 AA, the data-dense dashboard
  guidance): every corner radius comes from one eight-step scale
  (--radius-2xs … --radius-pill) instead of twenty distinct values, so
  controls of one height share a corner; the two secondary text tiers pass
  4.5:1 on every static surface of all six visual styles (the default
  --muted-2 sat at 2.6:1, studio at 2.8:1), which a repository test now
  computes from the tokens, resolving color-mix and translucent surfaces;
  keyboard focus wears a two-layer ring (2 px at 55 % plus 5 px at 22 %)
  that reads on dark surfaces where the former 3 px 18 % ring did not, and
  the browser smoke presses a real Tab to check it; the network card's two
  rates stack instead of colliding when a reading is wide. No dependency,
  build step, or new style.
• Dashboard legibility and small-screen layout. Secondary text no longer
  uses 7, 8, or 9 px sizes (146 declarations, below what CJK text needs at
  desk distance): it goes through a --text-2xs/--text-xs/--text-sm scale of
  9/10/10.5 px, and the compact density alone dials the scale back down. On
  narrow screens the header wraps its controls beneath a one-line title and
  a truncated one-line kicker instead of squeezing the title into three
  lines (135 px → 94 px of header at 390 px). Summary cards place their
  progress bar in normal flow beneath the footnote, so no visual style can
  push the footnote under the bar (the glass style did). The first card is
  titled 节点健康 rather than 节点可用性, because its badge reports fleet
  health, not availability. Attention items carry their severity as a left
  bar and dot, with the text badge kept for assistive technology, freeing
  the line for the message. A repository test keeps literal font sizes at
  or above 9 px, and the browser smoke pins the mobile header geometry.
• Resource conditions must now be sustained by the clock as well as by
  sample count: incidents.resource_open_seconds (default 60, range 0–3600)
  is the minimum span the confirming samples must cover before a CPU, memory,
  swap, filesystem, pressure, GPU-memory, temperature, idle-memory, or
  hardware-health condition opens or changes severity. On a live deployment
  polling every five seconds, VRAM spikes of ten to fifteen seconds satisfied
  resource_open_cycles: 2 and produced 43 gpu_memory incidents in five
  hours, 42 of them shorter than ten minutes and most resolved within twenty
  seconds. The idle-VRAM condition gets its own floor,
  incidents.gpu_idle_memory_seconds (default 300), because the same
  deployment logged sixteen two-minute gpu_idle_memory incidents from
  checkpoint and evaluation pauses. Connectivity and GPU-availability
  conditions still open immediately; set either floor to 0 to confirm by
  cycles alone as before.
• Seven modules crossed their line ceilings during this round and were split
  along existing seams instead: the struct-packed history records moved from
  service.py into telemetry_points.py, the bounded HTTPS delivery
  transport (pinned DNS, SSRF guards) from notifications.py into
  webhook_transport.py, the telemetry-domain rules from incidents.py into
  incident_domains.py, and the SQLite DDL and row contracts from
  persistence.py into persistence_schema.py; the duration floor then
  pushed two more over, so the incident vocabulary (incident_types.py) left
  incidents.py and the integration section parsers
  (config_integrations.py) left config_loader.py, the maintenance window
  type (maintenance.py) left config.py, the restore path
  (persistence_restore.py) left persistence.py, occupancy pairing
  (occupancy.py) left usage.py, and the snapshot's fleet totals
  (fleet_stats.py) left service.py. Every ceiling ratchets down.
• A connectivity incident's diagnosis.nextSteps (and the dashboard's
  incident dialog) now open with the step that follows from the failure
  classification — check the jump host's forwarding, the node's sshd
  load or a ban, known_hosts, authorized_keys, DNS, routing — ahead of the
  generic batch-mode check; unclassified failures keep the generic guidance.
  The dashboard's dialog guidance moved from incident-text.js into its own
  leaf, diagnosis-text.js.
• Every module above roughly 800 lines now sits under the line-budget
  ratchet (persistence.py, doctor.py, __main__.py, incidents.py, and
  notifications.py join it), so growth anywhere in the core has to come
  with an extraction.
• The owners dialog's aggregation and the usage bill's wording left app.js
  (seven lines under its ceiling) for the owner-usage.js leaf with a Node
  contract test; app.js's ceiling ratchets from 5850 to 5770 lines.

Added

• The dashboard's owners dialog bills usage from GET /api/reports/usage
  when history persistence is on — complete over the retained window instead
  of capped per device — and falls back to the in-memory rollup only when the
  service answers HISTORY_UNAVAILABLE. The summary names the source and its
  coverage, and a per-day strip shows the split the report carries.
• mocop api PATH --data JSON performs writer-tier POSTs from the monitor
  host — maintenance windows, incident acknowledgements and silences, host
  add/remove, collector settings, manual probes, restart, and update — so an
  agent or operator can act on what the read routes show without a browser.
  The client presents the listener's own loopback origin and the same-origin
  marker exactly as the dashboard does, the server validates the body against
  the manifest as always, a writer route without --data is refused
  (BODY_REQUIRED, so nothing restarts by accident), --data on a read route
  is refused (METHOD_NOT_ALLOWED), and reader routes stay DASHBOARD_ONLY
  because their marker on a read changes the collection cadence. Possession
  of the private capability file is the authority, as it already was.
• Long-window reports from the history database. GET /api/reports/usage
  pairs every retained process transition (no per-device cap, so
  partialGpus is always zero and owner GPU-hours are exact over the raw
  retention), classifies idle occupancy from hourly rollups, and adds a
  per-day split; GET /api/reports/utilization gives hourly busy share,
  utilization and memory per host or per device over up to 90 days. The
  rollups (gpu_hourly) are maintained by the writer in the same
  transaction as the raw samples and backfilled at startup from the raw
  table in well under a second per million rows; a database at its size cap
  keeps starting as before and the rollups wait for retention to free space.
  Both endpoints answer 503 HISTORY_UNAVAILABLE without persistence, and
  the usage-accounting playbook points long windows at them. The work split
  six modules along existing seams to stay under their ceilings:
  api_describe.py, persistence_api.py, persistence_rollups.py,
  persistence_transitions.py, usage_views.py, and reports.py; the
  GET dispatcher in web.py became a table.
• GET /api/incidents correlations gain a simultaneous_connectivity_loss
  kind: when at least three hosts and half of the monitored fleet lose
  connectivity within 120 seconds of each other, one fleet-wide correlation
  (anchor: null) says the monitor's own uplink or a shared relay is the
  likely cause, instead of leaving N unrelated-looking connectivity incidents.
  It needs no topology configuration, subsumes configured-path groups inside
  it, rides along in webhook payloads like the existing kind, and the
  dashboard's attention panel lists it first as "疑似监控端链路故障". The
  grouping logic moved from attention.js into an attention-groups.js
  leaf.
• Maintenance windows can recur daily: recurrence: {daily: true, start, duration_minutes} (duration strictly below one day) beside the existing
  weekly form. A fleet reached through a link that degrades at the same hour
  every evening — a home uplink whose relay reconnects nightly — can silence
  the hosts behind it for that span instead of alerting every night. Window
  objects in /api/snapshot and /api/inventory carry cadence
  (weekly or daily) beside recurring, the dashboard labels the plan
  badge accordingly, and the dashboard's inventory contract accepts the key.
• Two SSH failures that a fleet behind a bastion sees often now have their own
  sanitized classification instead of the generic SSH connection failed:
  SSH jump host could not reach the target (the ProxyJump/ProxyCommand
  host refused or could not open the forward) and SSH connection closed during key exchange (the peer closed or reset the connection before the
  banner or key exchange completed). A live deployment had shown the generic
  message for three hosts across 180 consecutive probes; the root cause was
  the jump host every time.
• servers[].message is a published vocabulary: GET /api/meta lists every
  string it can hold as serverMessages (exact strings plus the two prefixes
  that end in an exit status), the API reference tabulates each with its
  meaning, and the repository test that already kept the probe and the
  dashboard translations aligned now holds the manifest and the reference
  table to the same set. The SSH failure classification and the stale-multiplex
  retry rules moved from probe.py into ssh_failures.py, the module that
  owns that vocabulary's emit sites.
• The API reference specifies the webhook receiver contract — request headers,
  the HMAC-SHA256 signature over the raw body, the JSON body with its event
  and correlation shapes, the test delivery marker, and the retry, throttle,
  and suppression rules — and a repository test keeps the documented example
  body's keys equal to what the delivery code sends.
• An agent playbook for /api/usage in the API reference: how to bound
  hours, read earliestDataAt against sinceAt, partialGpus, and
  droppedRecords, quote idleShare with its sampledSeconds basis (on a
  live deployment about one hour of samples against days of occupancy), and
  when owner can be attributed at all; AGENTS.md points at it.
• Three decision records for this round's architectural choices:
  ADR-0027 (incident generations
  resume from each condition's latest persisted transition, webhook workers
  primed with the restored keys), ADR-0028
  (the wall-clock confirmation floor beside the cycle count), and
  ADR-0029 (the process
  inventory as a blind spot until collection_stale_cycles, self-anchored
  stops through firstSeenAt, partialGpus), each with the rejected
  alternatives and the live figures that motivated them.
• The self-update worker's uv install path — the one a uv tool install
  deployment takes, since those environments have no pip — and every
  refusal of the update state machine are under test (updates.py coverage
  76% → 90%), and a governance test proves every routed dashboard asset is
  shipped by the wheel's package-data globs.