feat(readers): open .jsonl.gz + behavior_v2 run logs everywhere via shared js/runlog-format.js (PR 3 of 3) — Studio v0.73 - #188
Open
mbreiser wants to merge 1 commit into
Conversation
…red js/runlog-format.js (PR 3 of runlog-behavior-v2-plan) — Studio v0.73 Reader side of docs/development/runlog-behavior-v2-plan.md (Part 3). Stacks on the Studio PR (v0.72). With this, everything that opens a run-log FILE inflates gzip on the magic bytes and expands behavior_v2's compact arena echoes back to the v1 `arena_command` object at parse time, so every existing consumer (p3LedEpochs, stall forensics, replay frame decoding) is unchanged. - js/runlog-format.js (new; classic dual-export, vendored byte-identical at dashboard/data-browser/vendor/): isGzip, isRunlogName, stripGz, inflateIfGzip, readRunlogText (string|bytes|ArrayBuffer|Blob → text), readRunlogPrefixText (truncation-tolerant gunzip for the catalog's 64 KB metadata reads), isArenaArray, expandV2Line (exact v1 key order; timeout ⇒ status/echo/ok all null), compactV1Line, detectFormat, createNormalizer() (per-file state machine), convertV1ToV2Text / convertV2ToV1Text (JS mirror of the bridge converter, for tests + parity). - Dashboard: analysis-core.js parseJsonl normalizes every line, run.logFormat + run.rawBytes, parseFilename strips .gz; github-repo.js fetchRaw reads bytes and inflates (prefix mode inflates a truncated gz); app.js loaders (repo / URL / local server / dropped file) go through readRunlogText, catalog filters accept .jsonl.gz, new size column (compressed size; inflated size + format in the hover), ?v= cache busters bumped, file input accepts .gz. - Replay: js/runlog-replay.js parseRunLog normalizes records (loads js/runlog-format.js first in arena_studio.html; degrades to "v2 echoes skipped" + console warning under a stale cache); js/arena-studio-alt.js replay picker lists .jsonl.gz and inflates the picked file. arena_replay_viewer.html (protocol-fed) and js/studio-runlog-adapter.js (no file reads) unchanged. - Tests: tests/test-runlog-format.js (new, 86 checks, in pixi run test — also enforces the vendored copy); tests/test-runlog-replay.js v2 case identical to the equivalent v1 (76 checks); dashboard test-analysis.js re-reads the P3 fixture as v2 + gz asserting identical frames, arena_command objects, preference indices, LED epochs and page CSV rows, plus metadata from a gz prefix; dashboard/data-browser/tests/corpus-v2-parity.js runs the same over a whole clone (164/164 on cshl-2026-course origin/main). - Browser-verified: the dashboard opens a converted .jsonl.gz via ?data= (52,369 frames / 29 steps, size column "1.6 MB gz", no console errors). - Docs: dashboard README, release notes v0.73, plan doc status + PR 3 notes, CLAUDE.md reader rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reader side (PR 3 of 3) of docs/development/runlog-behavior-v2-plan.md. Stacks on the Studio PR #186 (base = its branch; retarget to
mainas the stack merges). Once this is in, everything that opens a run-log file reads both.jsonland.jsonl.gz, and bothbehavior_v1andbehavior_v2. Merge order: this PR must land before or together with #186 — a v0.72 Studio commits.jsonl.gzfiles that only these readers open.The shared module —
js/runlog-format.jsOne classic dual-export module knows how a run-log FILE is encoded; every reader goes through it. The dashboard uses an exact vendored copy (
dashboard/data-browser/vendor/runlog-format.js, same convention askinematics.js) andtests/test-runlog-format.jsfails if the copies diverge.isGzip(magic1f 8b, never by name alone),inflateIfGzip,readRunlogText(string | bytes | ArrayBuffer | Blob/File)→ text,readRunlogPrefixText(a truncated.gzprefix still inflates to therun_metadataline near the top, which is what the catalog's 64 KB metadata reads need).isRunlogNameaccepts.jsonl|.ndjson|.jsonwith optional.gz;stripGzfor filename parsers.isArenaArray,expandV2Line(arr, schema)→ the exact v1arena_commandobject in the original key order, including the corpus-verified timeout rule (status,echo,okall null);compactV1Line;detectFormat;createNormalizer()— the per-file state machine each reader feeds every parsed line through, so v2's["a", …]echoes reach every existing consumer (p3LedEpochs, stall forensics, replay frame decoding) as v1 objects. A v2 file may still carry a verbatim v1 echo (non-fitting bulk command) and a converted legacy file may carry"cols": null— both handled.convertV1ToV2Text/convertV2ToV1Text: the JS mirror of the bridge's--convert, used by the tests and the parity script to exercise the readers on v2 versions of every existing v1 log.Readers changed
dashboard/data-browser/):analysis-core.js parseJsonlnormalizes every line and reportsrun.logFormat/run.rawBytes;parseFilenamestrips.gz.github-repo.js fetchRawnow reads bytes and inflates (prefix mode inflates a truncated gz).app.js: the repo, URL, local-server and dropped-file loaders all go throughreadRunlogText; catalog filters accept.jsonl.gz; new size column (committed/compressed size, with the inflated size and line format in the hover); file input accepts.gz;?v=cache busters bumped.js/runlog-replay.js parseRunLog): normalizes records;arena_studio.html(v0.73) loadsjs/runlog-format.jsbefore it. Under a stale cache it degrades to "v2 echoes skipped" with a console warning while frames and runner events still replay.js/arena-studio-alt.js: the replay picker lists.jsonl.gzfrom the repo and inflates the picked file.arena_replay_viewer.html(fed over its protocol module, never reads files),js/studio-runlog-adapter.js(no file reads),js/kinematics.js.Tests
tests/test-runlog-format.js(new, inpixi run test): 86 checks — gzip detection and inflation for every input type incl. Blob, truncated-prefix inflation of a >64 KB gz,expandV2Lineexact objects (ok / reject / timeout / other command byte) and malformed-input throws,compactV1LinePython-identical arrays + round trips + must-not-compact cases, the normalizer state machine (orphan arrays before a schema, full-level detection),detectFormat, whole-text conversion both ways incl. the legacy no-schema insertion, gz → text → normalizer end to end, and the vendored-copy identity.tests/test-runlog-replay.js65 → 76 checks: a behavior_v2 log yields identical samples, arena frames, events and bounds to the equivalent v1 log; rejected and timed-out compact echoes are not frames.dashboard/data-browser/tests/test-analysis.js: re-reads the P3 fixture as v2 + gzip and asserts identical frames, identical expandedarena_commandobjects (51,091 of them), identical preference indices, LED epochs and every page's CSV rows, plusrun_metadatafrom a 64 KB gz prefix and run id from a.jsonl.gzname.pixi run testpasses; touched JS is Prettier-clean.?data=<converted .jsonl.gz>loads (all scripts with fresh stamps, the gz fetched once), catalog row shows1.6 MB gzwith hover "compressed (gzip) size 1.6 MB · 4.9 MB of JSONL text · format behavior_v2", 52,369 frames / 29 steps (identical to the v1 fixture), no console errors. Not browser-tested: the Alt Studio replay picker path (one-line change to the file read; the parser is covered in Node) and the private-repo prefix read against real GitHub (covered by the mocked client test + the Node gz-prefix test).164 / 164 identical (frames, events, step summaries, metadata, run id; plus every P3 preference index and every P3 page's CSV rows for the 30 P3 runs). Formats: 14 legacy, 149 behavior_v1, 1 full.
All files
🤖 Generated with Claude Code