feat(studio): gzip run-log commits, behavior_v2 default, bridge log-level ack (PR 2 of 3) — Studio v0.72

CLAUDE.md

@@ -292,6 +292,17 @@ fix flows to every page automatically; two hand-written HTML pages never will.
   recurring mistake. The changelog lives ONLY in
   `docs/development/arena-studio-release-notes.md` — add an entry there for user-visible
   changes.
+- **Run logs are `.jsonl.gz` (v0.72+, `docs/development/runlog-behavior-v2-plan.md`).**
+  `commitRunLog` gzips the bridge export (`GH.gzipBytes`) and commits
+  `runlogs/<bench>/<name>.jsonl.gz` via `GH.commitFile`, which routes >30 MiB payloads
+  through the Git Database API (`GH.directCommitLarge`) because the Contents API
+  rejects ~35 MiB+ files. Every reader (dashboard, replay viewer, adapter) must inflate
+  on the gzip magic and accept both `behavior_v1` and `behavior_v2` line formats. The
+  log level is a runtime setting (File ▾ → Run logging, localStorage `studio_log_level`,
+  default `behavior_v2`); the runner asserts it via `log_control` and the bridge ACKS
+  the level it will actually write (`bridge.waitForLogLevelAck`) — a pre-3.0 bridge
+  never acks, so treat "no ack" as behavior_v1. Never write `log_format` into
+  `run_metadata` from anything but the acked/inferred level.
 - Bump the footer version/timestamp on every edit; never Prettier the HTML.
 
 ## Pattern Designer (`pattern_editor.html`)

docs/development/arena-studio-release-notes.md

@@ -4,6 +4,29 @@ The Studio's footer used to carry the full changelog inline; it now shows one li
 history lives here. Newest first. (Per-session engineering detail stays in
 `arena-studio-handover.md` and the design docs — this file is the user-facing what-changed list.)
 
+## v0.72 (2026-09-06) · Run logs commit gzipped; compact `behavior_v2` log format; the bridge confirms the log level
+
+- **Run logs now commit as `.jsonl.gz`.** The Studio gzips the exported log in the
+  browser before committing it to the course repo (lossless, 6–8× smaller), so 40 s
+  and hour-long runs no longer hit GitHub's ~35 MiB per-file ceiling (the rig03-sr
+  40 s run that failed to auto-commit was 51 MB raw → about 6 MB gzipped). Anything
+  still over 30 MiB after compression goes through GitHub's Git Database API
+  instead of the Contents API. The run summary and the upload dialog show the
+  committed size and, when used, the large-file path. **Readers must inflate:**
+  the analysis dashboard and the replay viewer learn to open `.jsonl.gz` in the
+  next release; until then open new logs with `gunzip` or the bridge's
+  `--convert`. Old `.jsonl` files are untouched.
+- **New default log format `behavior_v2`** (File ▾ → Run logging). Same frame
+  rows as before; the per-command arena echoes (three quarters of a log's bytes)
+  are written compactly by the bridge. `behavior_v1` stays selectable for one
+  release; `full` is unchanged. Needs bridge 3.0 — benches `git pull` and restart
+  `pixi run bridge`.
+- **The bridge now confirms the log level it writes.** The run-start banner names
+  the confirmed level and warns "bridge too old for behavior_v2 — logging
+  behavior_v1" when the running bridge cannot write the chosen format (an old
+  bridge used to fall back silently). The Console's read-only level mirror shows
+  the confirmed level. `run_metadata` gains `log_format`.
+
 ## v0.71 (2026-09-04) · Closed-loop apply is reset at run start and abort
 
 - **No more error floods at the start of a run.** If FicTrac closed-loop "apply"

docs/development/cshl-pipeline-test-plan.md

@@ -355,7 +355,16 @@ sensitive hops; I measured both against the real services:
 | Hop | Verified up to | Binding limit |
 |---|---|---|
 | Bridge → browser (WebSocket `log_export`) | **50 MB round-trips fine** | not the bottleneck |
-| Browser → GitHub (`directCommit` PUT) | **35 MiB OK; 40 MiB rejected** | **~35 MiB per file** |
+| Browser → GitHub (`directCommit` PUT, Contents API) | **35 MiB OK; 40 MiB rejected** | **~35 MiB per file — RAW** |
+| Browser → GitHub (`directCommitLarge`, Git Database API) | used automatically above 30 MiB gzipped | GitHub's 100 MiB hard limit |
+
+**Since Studio v0.72 (2026-09-06) run logs commit as `.jsonl.gz`** — gzipped in the
+browser (`GH.gzipBytes`, lossless, 6–8× smaller on v1 logs, ~3× on the denser
+`behavior_v2` format) — and `GH.commitFile` routes anything still over 30 MiB
+through the Git Database API (blob → tree → commit → ref). Measured on the
+course corpus: 1282 MB of v1 logs → 656 MB as v2 → 211 MB as v2.gz; the 51 MB
+40 s rig03-sr run → 6 MB. Readers must inflate (gzip magic `1f 8b`). The table
+below describes the RAW Contents-API limit the gzip path was added to clear.
 
 **The binding constraint is GitHub's Contents API: ~35 MiB per committed
 file.** Measured: 5 / 10 / 25 / 30 / 35 MiB all commit (201) and pull back
@@ -377,6 +386,9 @@ FicTrac frame, ~65 bytes/line):
 - `bridge.py --log-frames` (full 25-field record per frame) is **~3–4× larger**
   and could blow the ceiling on a long run — **leave it off for the course**
   (it's off by default). It's a debugging switch, not a course setting.
+- With v0.72's gzip + `behavior_v2` an hour-long run lands around 4 MB, so the
+  ceiling is no longer the constraint in practice; the Git Database fallback is
+  the safety net for `full`-level or multi-hour logs.
 
 **Failure is graceful, never data loss.** If a run log ever exceeds the ceiling,
 `directCommit` gets the 422 and the Studio shows *"Run log commit failed … saved

docs/development/runlog-behavior-v2-plan.md

@@ -1,10 +1,12 @@
-# Run-log format `behavior_v2` + large-file commit path — PLAN (PR 1 bridge shipped; PRs 2–3 pending)
+# Run-log format `behavior_v2` + large-file commit path — PLAN (PRs 1–2 shipped; PR 3 readers pending)
 
 Owner: Michael. Drafted 2026-09-06 from the analysis of rig03-sr run `rydc2tql`
-(40 s trials, 51.4 MB, failed to auto-commit). Status: **PR 1 (bridge) implemented
-2026-09-06** — see "PR 1 implementation notes" at the end; PR 2 (Studio) and PR 3
+(40 s trials, 51.4 MB, failed to auto-commit). Status: **PR 1 (bridge) and PR 2
+(Studio) implemented 2026-09-06** — see the implementation notes at the end; PR 3
 (dashboard/readers) not started. Implementation lands as three PRs (bridge, Studio,
-dashboard) in that order.
+dashboard) in that order. **Merge PR 2 together with (or after) PR 3:** once PR 2 is
+live, new course runs commit as `.jsonl.gz`, which the dashboard and replay viewer
+cannot open until PR 3 lands.
 
 ## Problem
 
@@ -226,3 +228,31 @@ Studio (PR 2) and reader (PR 3) work must build on:
 - **Corpus result (164 logs, 1.28 GB, origin/main of cshl-2026-course):** all pass;
   totals in PR 1's description (1281.6 MB v1 → 655.6 MB v2 → 211.3 MB v2.gz at gzip
   level 6). The 51 MB `rydc2tql` run → 20.4 MB v2 → 6.2 MB v2.gz.
+
+## PR 2 implementation notes (Studio, 2026-09-06)
+
+- **`js/fictrac-bridge-client.js`:** `LOG_LEVELS = ['behavior_v2','behavior_v1','full']`
+  (default v2); handles `hello_ack` (→ `bridgeInfo`, `bridgeSupportsLevel(level)`) and
+  `log_control_ack` (→ `ackedLogLevel`, cleared by every `setLogging()` and on close);
+  new `'loglevel'` event `{source, requested, level, ok, levels, enabled, file}` plus an
+  `'err'` log line on mismatch; `waitForLogLevelAck(ms)` resolves the acked level or
+  null (old bridge / not connected / timeout).
+- **`js/studio-github.js`:** `gzipBytes(input)` (CompressionStream; rejects where
+  unavailable), `isGzip(bytes)`, Git Database builders (`reqCreateBlob`, `reqGetCommit`,
+  `reqCreateTree` — allowlisted path, `reqCreateCommit`, `reqUpdateRef` fast-forward
+  only), `directCommitLarge` (7-call sequence, per-step error reporting),
+  `commitFile` (routes by size; `LARGE_FILE_BYTES` = 30 MiB; `thresholdBytes` test hook;
+  result carries `via` + `bytes`).
+- **`arena_studio.html` v0.72:** `#fmLogLevel` v2/v1/full (v2 default; a stored v1 is
+  honored); Console `#cFtLogLevel` follows the `'loglevel'` event (⚠ + tooltip on
+  mismatch); run start awaits the ack ≤ 800 ms and names the level in the banner +
+  transcript (`WARN` level when it disagrees or is unconfirmed); `run_metadata` gets
+  `log_format` = acked level, or the inferred one for a non-acking bridge (`behavior_v1`
+  when v2 was requested, else the request itself); `commitRunLog` gzips → `<name>.jsonl.gz`
+  via `GH.commitFile`, falls back to raw `.jsonl` without CompressionStream, and reports
+  raw → gz size + the path used in the log line, the modal and the run-summary line.
+- **Not done here (PR 3):** readers. Until PR 3, `.jsonl.gz` files from a v0.72 Studio are
+  opened with `gunzip` / `bridge.py --convert`.
+- **Bench check when hardware is back:** recorded 20 s run → `.jsonl.gz` committed with the
+  size shown; banner names `behavior_v2`; then run against a deliberately old bridge
+  checkout and confirm the "bridge too old" warning + `log_format: behavior_v1`.

js/fictrac-bridge-client.js

@@ -14,9 +14,16 @@
  *                  "x":<rad>,"y":<rad>,"hd":<rad>}   (behavior_v1 fields — the live
  *                    oscilloscope's raw state; index/seq/t kept for back-compat)
  *                 {"type":"log_export_result","name":..,"content":..}  (or {"error":..})
+ *                 {"type":"hello_ack","bridge":..,"levels":[..],"level":..,"logging":..}
+ *                    (bridge ≥ 3.0 only — an OLD bridge never answers hello)
+ *                 {"type":"log_control_ack","enabled":..,"level":..,"requested":..,"file":..}
+ *                    (`level` = the level the bridge will ACTUALLY write; an unknown
+ *                     requested level is ignored by the bridge, so this is how a
+ *                     stale bridge becomes visible instead of silently logging v1)
  *   us → bridge:  {"type":"hello","client":...,"v":1}
  *                 {"type":"config","fictrac_port":..,"gain":..,"offset":..,"frames":..}
- *                 {"type":"log_control","enabled":<bool>,"level":"behavior_v1"|"full"}
+ *                 {"type":"log_control","enabled":<bool>,
+ *                                       "level":"behavior_v2"|"behavior_v1"|"full"}
  *                     (opens/closes the log file; level picks the frame-row format)
  *                 {"type":"log", ...}                        (an event to append)
  *                 {"type":"log_export"}                      (close + stream back the log)
@@ -43,11 +50,31 @@
  *   'blocked' (reason:string)              — a frame could not be applied (canApply false)
  *   'apply'   (on:bool)                     — closed-loop apply was enabled/disabled
  *   'log'     (msg:string, kind:string)    — human-readable trace line
+ *   'loglevel' (info:object)               — the bridge acknowledged a log level:
+ *                                            {requested, level, ok, levels, enabled,
+ *                                             file, source:'log_control'|'hello'}.
+ *                                            ok=false ⇒ the bridge cannot write the
+ *                                            requested level (too old) and `level`
+ *                                            is what it logs instead.
  */
 (function (global) {
     'use strict';
 
-    const EVENTS = ['status', 'stats', 'frame', 'sample', 'applied', 'blocked', 'apply', 'log'];
+    const EVENTS = [
+        'status',
+        'stats',
+        'frame',
+        'sample',
+        'applied',
+        'blocked',
+        'apply',
+        'log',
+        'loglevel'
+    ];
+    // Log levels this client knows how to request, most-preferred first. The
+    // bridge advertises ITS list in hello_ack; a level missing there is one the
+    // running bridge is too old for. Mirrors bridge.py LOG_LEVELS.
+    const LOG_LEVELS = ['behavior_v2', 'behavior_v1', 'full'];
 
     class FicTracBridgeClient {
         /**
@@ -86,16 +113,21 @@
             this._lastBlockedMs = 0;
 
             // Bridge config (mirrors the console inputs). Sent on connect + on change.
-            // logLevel is the frame-logging level requested when logging starts
-            // ('behavior_v1' default | 'full'); the browser ASSERTS it so the runner
-            // logs behavior_v1 regardless of how the bridge process was launched.
+            // logLevel is the log format requested when logging starts
+            // ('behavior_v2' default | 'behavior_v1' | 'full'); the browser ASSERTS
+            // it so the runner logs deterministically regardless of how the bridge
+            // process was launched. The bridge answers with the level it will
+            // actually write (log_control_ack) — see ackedLogLevel.
             this._config = {
                 fictrac_port: 60000,
                 gain: 1.8,
                 offset: 0,
                 frames: null,
-                logLevel: 'behavior_v1'
+                logLevel: LOG_LEVELS[0]
             };
+            this._bridgeInfo = null; // from hello_ack: {version, levels, level} (null = old bridge / not yet)
+            this._ackedLevel = null; // from log_control_ack while logging is enabled
+            this._ackWaiters = []; // waitForLogLevelAck() resolvers
         }
 
         // ---- events ----------------------------------------------------------
@@ -193,11 +225,16 @@
                 }
                 if (msg && msg.type === 'frame') this.handleFrame(msg.index, msg);
                 else if (msg && msg.type === 'log_export_result') this._handleExportResult(msg);
+                else if (msg && msg.type === 'hello_ack') this._handleHelloAck(msg);
+                else if (msg && msg.type === 'log_control_ack') this._handleLogControlAck(msg);
             };
             ws.onerror = () => this._emit('status', 'error', 'err');
             ws.onclose = () => {
                 this._stopRateTimer();
                 this._ws = null;
+                this._bridgeInfo = null;
+                this._ackedLevel = null;
+                this._settleAckWaiters(null);
                 this._settleExport('reject', new Error('bridge disconnected during log export'));
                 this._emit('status', 'disconnected', 'dim');
                 this._emit('stats', this.stats);
@@ -265,18 +302,129 @@
         }
 
         /**
-         * Select the frame-logging level for the NEXT log the bridge opens:
-         * 'behavior_v1' (compact, the runner default) or 'full' (25-column). Takes
-         * effect at the next setLogging(true) / reconnect (the bridge applies it
-         * when it opens a fresh file). Unknown values are ignored.
+         * Select the log level for the NEXT log the bridge opens: 'behavior_v2'
+         * (compact arena echoes, the default), 'behavior_v1' (the pre-2026-09
+         * format) or 'full' (25-column FicTrac record). Takes effect at the next
+         * setLogging(true) / reconnect (the bridge applies it when it opens a fresh
+         * file). Unknown values are ignored.
          */
         setLogLevel(level) {
-            if (level === 'behavior_v1' || level === 'full') this._config.logLevel = level;
+            if (LOG_LEVELS.includes(level)) this._config.logLevel = level;
+        }
+        /** The level this client will request (not necessarily what the bridge writes). */
+        get logLevel() {
+            return this._config.logLevel;
+        }
+        /**
+         * The level the bridge acknowledged for the CURRENT log file (log_control_ack
+         * with enabled=true), or null: not yet acked, logging off, or an old bridge
+         * that never acks (then it writes behavior_v1 / whatever --log-frames said).
+         */
+        get ackedLogLevel() {
+            return this._ackedLevel;
+        }
+        /** hello_ack facts {version, levels, level} — null until a ≥3.0 bridge answers. */
+        get bridgeInfo() {
+            return this._bridgeInfo;
+        }
+        /**
+         * Can the connected bridge write `level`? true/false from hello_ack; null when
+         * unknown (no hello_ack yet — including every pre-3.0 bridge).
+         */
+        bridgeSupportsLevel(level) {
+            const lv = level || this._config.logLevel;
+            if (!this._bridgeInfo || !Array.isArray(this._bridgeInfo.levels)) return null;
+            return this._bridgeInfo.levels.includes(lv);
+        }
+        /**
+         * Resolve with the acked level once the bridge answers the pending
+         * log_control (or immediately if it already has); null after `timeoutMs`
+         * (default 1000) — i.e. an old bridge, or not connected.
+         */
+        waitForLogLevelAck(timeoutMs) {
+            if (this._ackedLevel) return Promise.resolve(this._ackedLevel);
+            if (!this.connected || !this._logging) return Promise.resolve(null);
+            return new Promise((resolve) => {
+                const w = { resolve, timer: null };
+                w.timer = setTimeout(() => {
+                    this._ackWaiters = this._ackWaiters.filter((x) => x !== w);
+                    resolve(null);
+                }, timeoutMs || 1000);
+                this._ackWaiters.push(w);
+            });
+        }
+        _settleAckWaiters(level) {
+            const ws = this._ackWaiters;
+            this._ackWaiters = [];
+            for (const w of ws) {
+                if (w.timer && typeof clearTimeout !== 'undefined') clearTimeout(w.timer);
+                w.resolve(level);
+            }
+        }
+        _handleHelloAck(msg) {
+            this._bridgeInfo = {
+                version: msg.bridge || null,
+                levels: Array.isArray(msg.levels) ? msg.levels.slice() : [],
+                level: msg.level || null
+            };
+            const requested = this._config.logLevel;
+            const ok = this._bridgeInfo.levels.includes(requested);
+            if (!ok) {
+                this._emit(
+                    'log',
+                    'bridge ' +
+                        (msg.bridge || '?') +
+                        ' cannot write ' +
+                        requested +
+                        ' (it offers ' +
+                        this._bridgeInfo.levels.join(', ') +
+                        ') — restart `pixi run bridge` from the current checkout',
+                    'err'
+                );
+            }
+            this._emit('loglevel', {
+                source: 'hello',
+                requested: requested,
+                level: ok ? requested : this._bridgeInfo.level,
+                ok: ok,
+                levels: this._bridgeInfo.levels,
+                enabled: !!msg.logging,
+                file: null
+            });
+        }
+        _handleLogControlAck(msg) {
+            const requested = msg.requested != null ? msg.requested : this._config.logLevel;
+            const level = msg.level || null;
+            const enabled = !!msg.enabled;
+            this._ackedLevel = enabled ? level : null;
+            const ok = !enabled || level === requested;
+            if (!ok) {
+                this._emit(
+                    'log',
+                    'bridge too old for ' +
+                        requested +
+                        ' — logging ' +
+                        level +
+                        ' instead (restart `pixi run bridge` from the current checkout)',
+                    'err'
+                );
+            }
+            this._emit('loglevel', {
+                source: 'log_control',
+                requested: requested,
+                level: level,
+                ok: ok,
+                levels: this._bridgeInfo ? this._bridgeInfo.levels : null,
+                enabled: enabled,
+                file: msg.file || null
+            });
+            if (enabled) this._settleAckWaiters(level);
         }
 
         /** Turn the bridge's session log file on/off (sends log_control + the level). */
         setLogging(on) {
             this._logging = !!on;
+            this._ackedLevel = null; // pending until the bridge acks this request
             const msg = { type: 'log_control', enabled: this._logging };
             if (this._logging) msg.level = this._config.logLevel;
             this._send(msg);
@@ -438,6 +586,8 @@
     }
 
     // Dual-export: CommonJS (Node tests) + window global (classic <script src>).
+    FicTracBridgeClient.LOG_LEVELS = LOG_LEVELS.slice();
+
     if (typeof module !== 'undefined' && module.exports) {
         module.exports = FicTracBridgeClient;
     }