Skip to content

v0.2.2

Choose a tag to compare

@github-actions github-actions released this 05 Sep 14:54
· 7 commits to master since this release

Added

  • A canonical activity ledger, and the fleet's own telemetry feeding it.
    Tool calls, turns, runs, model requests, compactions, skill activations, and test outcomes are reduced into one provenance-preserving ledger under <data_dir>/telemetry/ - monthly segments, content-free evidence (hashes, sizes, locators, never tool output), field-level source precedence with every contributing observation linked, exact closed-day rollups, and additive versioned migrations so a data directory written by an older build is upgraded on open rather than failing on the first write.
    Legacy telemetry and native transcripts are imported non-destructively and kept; nothing is deleted.
    canonical_telemetry_native_otel_enabled (Settings → Usage) hands each new Claude Code and Codex session an OTLP exporter pointed at the daemon over authenticated loopback; the contracts were measured against Claude Code 2.1.259 and Codex CLI 0.153.0, identity attributes are dropped and content attributes hashed before anything is stored, and every provider event name is counted per harness version so a renamed attribute shows as drift instead of silence.
    Resources → Fleet activity reads the ledger: exact totals over the selected window, cohort, backend, and layer; a tool-call audit down to its evidence; a collection-health readout with per-backend field coverage and parser signatures; deterministic inefficiency candidates with their denominators; and JSONL/CSV exports (GET /api/telemetry/v2/export/{kind}) that carry evidence identifiers and source locators.
    Schemas 3 and 4 (2026-09-03) finished the roadmap: every call carries its evidence quality and the approval wait it paired from request to resolution; a run's start says whether it was declared or estimated; closed hours are rolled up beside closed days so a 24-hour window is exact; skills, verifications, and compactions have daily rollups; Claude, Codex, OMP, Pi, and OpenCode native stores are reconciled straight into the ledger every five minutes and per run on demand (POST /api/telemetry/v2/reconcile); Codex's own metrics (codex.tool.call and its siblings) are exported to the daemon and compared per run against the ledger's count; a codex.sandbox_outcome verdict is a denied call whose cause stays named; and every displayed total names its range, cohort, denominator, and coverage.
    The view gained Project, model, family, outcome, and evidence-quality controls, an aggregate → calls → run → turn → evidence drill-down, skill and verification tabs, cohort comparison that refuses to compare cohorts differing on a dimension the split does not name, review buttons whose verdict is the only feedback a finding collects, and a legacy tab with a shadow comparison against the old tool_events table (canonical_telemetry_legacy_dashboard_enabled, on until an operator turns it off).
    Measured on 2026-09-03 at ten million calls, which is what forced schema 4: the quality readout is rolled up like the tool and workload figures, a page's exact count is summed from rollups, the repeated-call finding reads per-run counts the write path keeps, and a dimension filter is kept on the time index; every dashboard view answers from rollups under the 200 ms gate and every detail page under 500 ms, ingestion adds no measurable event-loop lag at two thousand observations a second, and a 24-hour live window audits call by call against the providers' own records (tools/telemetry_audit_window.py).

  • A stalled daemon now says where it was stuck, and the fleet no longer outranks it.
    When the event loop stops for three seconds or more, every thread's stack is dumped to <data_dir>/loop-stalls.log from a thread that needs no GIL, and the stall is explained once in daemon.log, kept in mux.db, and shown under stall_watchdog on /api/diagnostics/background - including whether a canary thread was starved too, which separates synchronous work on the loop from a native call holding the GIL.
    Session process trees run below normal priority and the daemon runs above it (session_process_priority, daemon_process_priority), so a wave of concurrent builds slows the builds rather than the person at the keyboard.

  • Reading a Project's config file no longer blocks the daemon.
    Every Projects poll read each Project's .swe-mux/config.toml on the event loop; the stall watchdog caught that read blocking for 6.6 s on a disk saturated by concurrent builds, and it now runs in a thread.

  • Daemon subprocesses are spawned off the event loop, and the health endpoint no longer stats the served frontend on every poll.
    asyncio starts a child synchronously on the loop that asks, and the stall watchdog caught that call holding the daemon for 23.5 s while a build saturated the disk; every helper the daemon runs - git queries and mutations, Tailscale, the firewall check, hook commands - now runs on a spawn loop on another thread, with callbacks, request context, and cancellation preserved, and commands a person is waiting on take a lane of their own so a poller's stuck spawn cannot hold them.
    A git query whose output exceeds the daemon's cap is refused rather than shown clipped.
    The served frontend's identity is answered from the last reading for a few seconds instead of a stat per request.

  • The UI says when the daemon is not answering.
    A slim banner appears after two missed health probes, counts the seconds, and clears on the first answer, so a stalled daemon reads as stalled rather than as a crashed app.

Fixed

  • An upstream rate limit no longer reads as a broken schema, and is retried.
    OpenRouter answers an upstream refusal with HTTP 200 and a body carrying only an error, so the call failed as "structured response must be an object" and - because 200 is in no retry set - was never retried, leaving a ledger row with zero tokens and no provider.
    The embedded status is now adopted whenever it names a real one, and the call takes the same bounded equal-jitter backoff any honest 429 would have; a body that answered and merely annotated the answer with an error is still an answer, and an unrecognisable code is handed to the caller's own ladder rather than guessed at.
    Measured over two days before the fix: 30 of 79 session-title calls lost this way, and one session that never got a title at all.
    The same frame on the streaming path used to leave the assistant answering a rate limit with silence; it is now raised, and retried only while nothing has been spoken.

Full changelog: https://github.com/jatoran/swe-mux/blob/v0.2.2/CHANGELOG.md