Arena Studio v0.5: CSHL course data pipeline (Session 2)#139
Merged
Conversation
The course write path end-to-end: 7 bench rigs direct-commit protocols,
patterns, and run logs to one shared course-data repo, namespaced by an
instructor-set bench id.
js/studio-github.js
- b64Bytes (binary-safe base64 for .pat commits), bytesEqual
- runlogs/ added to WRITABLE_PREFIXES; root roster.yaml readable (read-only)
- reqGetContentsRaw (Accept: application/vnd.github.raw — Contents API omits
content >1MB) + runBytes executor
- reqPutContents accepts contentBytes; directCommit() orchestration
(GET repo -> GET sha -> PUT default branch; no branch, no PR)
js/studio-url-state.js
- ?repo=owner/name (shape-validated); with repo present, p becomes a
repo-relative protocols/ path (new validator — SAFE_PATH_RE is the rig:
field's, doesn't match repo paths); encodeApp repo/repoPath provenance;
human-readable slashes in emitted URLs
js/studio-meta.js
- canRunExperiment gains bridgeConnected (universal run logging — a dead
bridge blocks recorded runs loudly) and missingPatterns (preflight names
the Console SD-upload fix)
fictrac-bridge (bridge.py + js/fictrac-bridge-client.js + README)
- log_export/log_export_result: close the active log, stream it back whole
to the asking client (retry-safe re-export of the same file)
- WS max_size 16MiB; --log-dir for on-demand logs; client exportLog()
(single-in-flight Promise, timeout, disconnect rejection)
arena_studio.html (v0.5)
- File-menu gh-block: repo owner/name, bench id, direct-commit checkbox
(localStorage; PR and direct modes mutually exclusive); Studio.rigId
- Universal bridge logging at run start (fictrac plugin no longer required)
+ run_metadata JSONL line (meta + rig_id) right after log rotation
- Run completion (COMPLETED only — never aborted/test) exports the bridge
log and direct-commits runlogs/<bench-id>/<proto>__<exp>__<STAMP>__
<run_id>.jsonl (colon-free stamp); clear notices on success, bench-id
unset, bridge down, and GitHub down ("saved locally on the bridge machine")
- Roster at connect: fetch roster.yaml, prefill experimenter for this bench,
course datalist from roster names, MAC (0xC2) cross-check chip
- Save routing: direct mode -> protocols/<bench-id>/<name>.yaml in the
course repo (repoRef provenance -> ?repo= URL); PR flow generalized to the
configured repo; local unchanged
- Pickers: "Open from library..." (site registry) and "Open from course
repo..." (array-aware directory listing of bench + shared)
- Promote-to-shared with byte-compare guard (yaml + each .pat; identical =
idempotent skip, different = blocked)
- Repo-hosted .pat preview byte-sources (raw media type) feeding the
existing PatPreview pipeline
- Missing-pattern preflight wired into the run gate (sequence-reachable
conditions, same resolution as the runner incl. pattern_ID fallback)
pattern_editor.html (v0.9.41)
- "Push to course repo": encodes the current pattern and direct-commits it
to protocols/<bench-id>/<protocol>_patterns/ using the same-origin
localStorage PAT/repo/bench-id (no new auth UI)
Tests: 761/761 (new coverage: b64Bytes/allowlists/raw/directCommit, repo
URL codec, gate additions, exportLog; bridge log_export verified live incl.
a >1MiB round-trip). Browser-verified against a mocked GitHub API: save,
picker, promote guard, roster prefill + MAC chip, runlog commit matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Test plan covering both 2026-07-03 sessions: what's already verified (unit / mocked-browser / live-repo / bench-signed-off) vs. what remains (real hardware end-to-end, multi-bench, guest-account setup). Parts: prereqs (guest account + PAT + per-bench config), Session-1 hardware (references 135-bench-checklist.md), a browser-only pipeline dry-run, the full hardware+bridge end-to-end, multi-bench/edge cases, five "day in the life" use-case narratives, and a time budget. Companion to the live repo reiserlab/cshl-2026-course-data (created + push/pull-verified this session; roster seeded with the 4 pipeline-test students). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- P1 expanded to a full walkthrough: create the guest account (space-free handle, "CSHL 2026" as display name), add it as a Write collaborator + accept the invite, generate the fine-grained PAT (Contents RW, this repo only, dated expiry), and the revoke path. - New "File size limits" section with measured ceilings: GitHub Contents API commits ~35 MiB and rejects 40 MiB+ (HTTP 422) — the binding limit; the bridge log_export WebSocket hop handles 50 MB (not the bottleneck). Per-run log rotation keeps a typical 10-min run ~4 MB; oversize commits fail gracefully (log stays on the bridge machine). Mitigations listed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… pending Correction from bench (user, 2026-07-03): the int16 negative frame_rate (Mode-2 reverse) is bench-verified via the Studio Console — mark §C2's Console checks done. The editor→runner path (a protocol authored with frame_rate: -30, run via Run) is NOT yet bench-verified; split it out as the remaining item. Fixes the stale "bench-unverified"/"still open" wording in both the bench checklist and the pipeline test plan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From bench-test feedback (Michael, 2026-07-04): - File ▾ menu now shows in Run/Edit/Console (was Edit-only, which hid the course-repo/library pickers from the student's Run view). Open items (import / from library / from course repo) appear in every view; New / Save / Promote / GitHub-settings are .edit-only. The bare "📂 Open" button (ambiguous — didn't say what it opened) is retired in favor of the menu. - Course-specific genotype list: studio-github READABLE_EXACT gains genotypes.yaml (read-only, like roster.yaml); Studio fetches the course repo's root genotypes.yaml at connect and populates the Fly-genotype datalist, independent of roster/bench-id. - Run-details "↗ source" links now follow the ACTIVE source: experimenter → course roster.yaml, genotype → course genotypes.yaml (when a course repo is configured); otherwise the default lab configs/metadata files. Links got ids (#mExpSrc/#mGenoSrc). - Console pattern-info label "stretch" → "duty cycle". - Removed the stale "Need v2? Open the v2 Experiment Designer" line from the Edit banner. Verified in-browser against the live course repo: File ▾ visibility per view, experimenter list incl. Guest, genotypes incl. wild-type/none, both source links repointed. Tests 763/763. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bench-test feedback round 2 (Michael, 2026-07-04): - GitHub settings block: now VISIBLE in all three views (was Edit-only / hidden) but LOCKED by default — inputs disabled + grayed, with a 🔒/🔓 toggle. Students can see the bench config (signed-in user, repo, bench id) but can't change the token/repo/bench id; the instructor unlocks to set up, and it re-locks on the next load. Mirrors the session-rig lock pattern. - Console Debug ▾ menu (SPI clock, refresh rate, Reset controller): locked by default with an Unlock toggle, so students can't change or reboot the controller by accident. The connection-gate toggles the #consoleFields fieldset (not per-button), so the per-control lock survives connect; the Unlock button carries no data-cmd and is never disabled. - Repo rename: user renamed the course repo to reiserlab/cshl-2026-course — updated all references (test plan, proposal, repo tooltip, and the course README on GitHub). NOTE for anyone hitting a commit failure: point the Studio Repo field at the NEW name; the old name 301-redirects and the PUT fails through the redirect. Verified in-browser: both locks locked-by-default on load, unlock/re-lock toggle correctly (Debug lock confirmed with the console fieldset enabled — its .click() is a no-op only while the fieldset is disabled/disconnected). Tests 763/763. Course repo roster: Anna Marie → Hannah Marie (done on GitHub). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e locks Bench-test feedback round 3 (Michael, 2026-07-04): - FIX: course roster/genotypes/source-links loaded ONLY on arena connect (Studio.onConnected), so a signed-in bench with no rig showed the lab defaults. Refactored into Studio.refreshCourseMeta(), now called on sign-in, on repo-field change, and at page load if a token+repo are already set — independent of any arena connection. onConnected still calls it (the roster MAC cross-check needs a connected controller's 0xC2, and correctly no-ops when disconnected). - Sign-out button (was missing — only ✓ @user showed). Clears the token from both storages, drops authed state, and reverts the metadata to lab defaults. Gated by the gh-block lock like sign-in. - Controller ▾ advanced lock: panel display mode + rig I/O roles locked by default with an Unlock toggle (mirrors the Debug lock). Top query items (Get info/IP/frames) stay usable. Prevents students changing display/I/O. - Pattern name-mismatch WARNING (non-blocking): when a referenced pattern NAME isn't on the SD but a numeric pattern_ID fallback resolves, the run would silently play that index — now a yellow warning surfaces in the Run view (Studio.unresolvedPatternNames). The hard BLOCK still fires when nothing resolves at all. Verified in-browser: metadata loads with NO arena connected; sign-out clears + reverts; both new locks default-locked and toggle (with the console fieldset enabled); pattern warning shows for a fallback-only reference and clears when resolved. Tests 763/763. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bench-test blocker (Michael, 2026-07-04): a protocol's pattern name (all_on) would not reconcile against the card, which stores 001_all_on.pat. onSdListing keyed the pattern set by the RAW filename with an exact-match lookup, so "all_on" never matched "001_all_on.pat" once connected (it worked offline only because the built-in MANIFEST keys by logical name). The NNN_ prefix is NOT removable — js/pattern-set.js:242 documents it as deliberate + bench-verified: the zero-padded index pins the firmware's SD-scan order (== pattern_ID) and the >8.3 name forces a FAT long-filename entry (pure pat0001.pat names are mis-read by the bench G6 build). So instead of renaming files, the WEB side now matches flexibly: - sdLogicalName(fn): strip a leading index prefix + .pat → logical name. - onSdListing keys the pattern set by that logical name (raw filename kept as sd_name for preview bytes + upload). - patternIndexByName / patternFramesByName fall back to the normalized key. This also makes one SD card hold SEVERAL protocols' sets: each set's 1-based prefixes collide globally, but the logical names stay unique and resolve by name to whatever scan index they land at. Verified in-browser (single + multi set listings). Tests 763/763. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bench feedback (Michael): the Run-view RUN LOG was a fixed 190px box with dead space beneath it — make it fill down to the footer. - run-view: height:100% + grid-template-rows:minmax(0,1fr) so the single content row takes the full viewport height (capped, so it can't overflow the footer); run-main is a flex column (align-self:stretch). - run-log-strip: flex:1 1 auto; min-height:190px — grows into the free space; collapsed state overrides to flex:0 0 auto. - seqlist: flex:0 1 auto + #seqBody scrolls, so on a SHORT window the sequence list shrinks/scrolls and the log keeps its 190 minimum instead of pushing content past the footer. Verified across window heights: tall (1100) → log grows to ~300px, bottom 20px above the footer (the run-view padding); short (680) → seqlist scrolls, log at 190, nothing overflows the footer. No console errors; tests 763/763. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the two SD upload buttons (Upload .pat… / Upload folder…) with one "Upload ▾" dropdown — a single pattern OR a whole pattern set, from a local file/folder, the site library, or the course repo. Library/repo sources browse by PROTOCOL and resolve its colocated _patterns/ via the GitHub Contents API; the load-bearing NNN_ filenames are preserved so the firmware's SD scan order (= pattern_ID) is kept end-to-end. All sources funnel through the one uploader (0x8D write + 0x83 rename) with per-file progress + error logging. Uploaded bytes are now registered as picker preview sources (for EVERY source) via Studio.registerUploadedPreviews, so thumbnails/hover-animations come straight from the upload. That lets the old "Load set…" folder step (offline local-folder previews) be removed with no capability lost. - no js/ modules changed (all in arena_studio.html); 763/763 tests pass - verified in preview: menu UX + kiosk lock, signed-out banners, library protocol picker, graceful empty-set, and end-to-end thumbnail render from registered upload bytes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fined savePatternFile saveFrameAnimationAsPat() called savePatternFile() — a function that exists nowhere, so it threw ReferenceError. Point it at handleSave() (the shared #saveBtn handler: buildPatternDataForSave → PatEncoder.downloadPattern). The function was also orphaned — no UI called it. Add a dedicated "💾 Save .pat" button to the Frame Animation pane (#animationMode) so a sequence can be saved in one click, instead of the two-step global GENERATE then SAVE. Consolidates the fix explored in a separate worktree session onto this branch. Verified in preview end-to-end: generate → capture frames → Add All → Save .pat → "Saved pattern: G6_2x10_animation_2f.pat" (200×40, 2 frames, GS16), no error. v0.9.42. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t retirement, v0.9.42 - New 2.12: fill the SD from the course repo / library / local via Console Upload ▾ (single pattern or whole set, picked by protocol); NNN_ prefixes preserved, previews render from uploaded bytes, "Load set…" retired. - UC1 (instructor setup) + UC3 (student pattern) now pre-load / sync the SD via Upload ▾ — the repo→SD sync that was a post-course follow-on now ships. - 2.7: name matched by logical name; document the non-blocking name-mismatch warning alongside the hard preflight block. - P3 kiosk note: Controller ▾ advanced (panel mode + rig I/O) locked too; how to switch accounts via Sign out. - Risk table row for the new browser→SD fill path; 761→763 checks; Pattern Editor v0.9.41→v0.9.42 (Frame Animation 💾 Save .pat). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…edback+retry; manual "⇪ Push log" Three bench-reported fixes. io_ext connect race (Controller ▾ rig I/O): - The 'state' event fires applyRigIo (via initFromController) before the Connect button's awaited GET_CONTROLLER_INFO populates Studio.capabilities, so on first connect io_ext read as absent → rig io_ext defaults (framescan, AO frame_number) were silently skipped with a spurious "lacks io_ext" warning. Fix: ensureCapabilities() fetches 0xC2 before gating if caps aren't known yet (sends are FIFO-queued, so a duplicate read is safe); reset Studio.capabilities on disconnect so each connect re-detects. - The fw-gated role options (in_trigger / out_debug_framescan / frame_number) stayed greyed after caps arrived because gating only ran inside applyRigIo / rig-change. Now updateRigIoGating() re-runs on every capability update (connect 0xC2 + manual "Get info"), so they un-grey as soon as io_ext is seen. - Relabel "Apply rig defaults" → "Apply I/O roles"; tooltip clarifies it sends the roles shown (rig defaults OR your session-local overrides, which never edit the rig YAML). Override was always the design; the race just masked it. SD Upload ▾ reliability + feedback: - Log each file as it's written+confirmed one at a time (SD ✓ i/n name (KB)). - 750 ms settle pause between writes + one retry (after 1.5 s) on a failed file — a large .pat could be dropped while the controller was still committing the previous one (bench: 813 KB failed in a batch, uploaded fine alone). Run-log manual push: - Refactor maybeCommitRunLog into a shared commitRunLog(log, opts) core; add Studio.commitRunLogNow (bypasses the aborted/disconnected outcome guard) and a "⇪ Push log" button in the run-log strip, enabled after a recorded run. For when auto-commit was skipped (a DISCONNECTED/aborted outcome — e.g. a controller reset mid-run — logs a brown "NOT auto-committed" line) or failed; the bridge re-serves the last run's file until the next one. No js/ changed; 763/763 tests; verified in-browser (gating toggles with mocked caps, button wiring). v0.5 footer 12:48 ET. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…p indicator) + "▶ Test" tidy Run view gains a bridge control line in the launch card, sharing it with the Run/Test experiment buttons and staying visible during a run: - Connect/disconnect + connection dot + status + live rx/applied counts, all driven by the SAME shared ArenaSession bridge as the Console — so state and counts stay in sync when switching between Run and Console. - A "closed-loop" indicator: very light by default, bright-green + pulsing while closed-loop is live. Reacts to a new 'apply' event on the bridge client (emitted from setApply on change; the runner toggles it for Mode-3/4 trials). - Gain / closed-loop config are deliberately NOT exposed here — Console only. Sequence rows: "▶ Test on arena" → "▶ Test"; the per-condition test buttons are hidden (and click-guarded) while an experiment runs — they were a distracting mid-run foot-gun. bridge client: add 'apply' event (fires on setApply transitions) + test (765/765). Verified in-browser: strip renders + mirrors state, closed-loop label dim→accent, "▶ Test" relabel, buttons hidden under body.running. v0.5 footer 13:04 ET. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…otal run timer - Drop the "⏱ host-timed · plugins skipped (log + fictrac run)" launch note (+ its dead CSS) — legacy messaging that no longer earns the space. - Running panel now shows a run clock under the step counter: elapsed / estimated total (M:SS / M:SS), ticking every second. Total = sum of step durations (RunnerLib.conditionDuration, host-timed estimate); elapsed = wall-time since run start. Started by beginRun (full sequence) and runCondition (single test); frozen by refreshState when running clears (covers complete/abort/disconnect). Verified in-browser: legacy note gone; timer 0:00/1:30 → ticks → freezes on stop; sits below "step N / total" beside the progress bar. 765/765 tests. v0.5 13:21 ET. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…arer export-timeout Run-log push (auto at run end + manual "⇪ Push log") now shows a modal that greys the screen: a spinner + "exporting… / committing to <repo>…" while busy, then either "✓ Run log committed" (auto-dismisses) or the FULL error text with a Close button. Previously a failure was a transient banner — the user couldn't tell what went wrong. The likeliest cause of a repeatable export failure is a STALE bridge build (the log_export handler is on this branch): the client's 15 s timeout message now says so — "no log_export reply from the bridge … restart `pixi run bridge` from the current version" — instead of a bare "timed out". Both sides of the export path are otherwise verified correct (web enables logging at run start + re-sends log_control on WS open; bridge closes+streams the file to the asking client). No behavior change to the commit itself — this is diagnosis + visibility. Verified in-browser: export-fail modal shows the exact message (red, Close); success path busy→✓→auto-dismiss. 765/765 tests; touched JS Prettier-clean. v0.5 footer 13:31 ET. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Run log is now bottom-anchored and capped at max-height:40vh (was flex:1 — it grew to fill and squeezed the launch card + sequence during a run). The SEQUENCE takes the flexible middle (flex:1); the logbox scrolls inside the cap. Verified: 200 log lines → strip stays exactly 40vh, sequence keeps its height. - Notes textarea defaults to ~38vh (plenty of room on the Run screen; still resizable); the meta-panel scrolls (overflow-y:auto) so the Auto-captured card is never pushed off by a tall Notes box. (Run-log CONTENT scoping — the 900k-vs-60MB variability — is the stale bridge not rotating; the current bridge.py opens a fresh file per run start. No code change.) CSS only; 765/765 tests. v0.5 footer 13:49 ET. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…"run the current bridge" - Risk table: full spine (connect → run → run-scoped log auto-commit) now ✅ bench-verified 2026-07-04 (2 runs, bench01); Upload ▾ SD-fill bench-used; 763→765 checks. - Prominent ⚠ callout: run the CURRENT bridge.py (log_export + per-run rotation are in this merge, not on old main) — git pull + restart pixi run bridge on each bench, else exports time out / logs never rotate. - Part 3.2/3.3: bridge strip live counts + closed-loop pulse + run timer + 40vh log cap; upload modal; per-run file rotation; ⇪ Push log fallback; expected ~196 KB/50 s and why two same-protocol runs are near-identical size. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Session 2 of the CSHL course plan (
~/.claude/plans/adaptive-hopping-island.md) — the course write path end-to-end. Builds on the merged Session 1 (#135 / #137). 7 bench rigs direct-commit protocols, patterns, and run logs to one shared course-data repo, namespaced by an instructor-set bench id.What's here
js/studio-github.js—b64Bytes(binary-safe base64 for.pat),bytesEqual,runlogs/write prefix + rootroster.yamlread-only,reqGetContentsRaw/runBytes(raw media type — Contents API omitscontent>1 MB),reqPutContentscontentBytes, anddirectCommit()(GET repo → GET sha → PUT default branch; no branch, no PR).js/studio-url-state.js—?repo=owner/name; withrepopresentpbecomes a repo-relativeprotocols/path (new validator);encodeApprepo/repoPath provenance.js/studio-meta.js—canRunExperimentgainsbridgeConnected(universal run logging: a dead bridge blocks recorded runs loudly) andmissingPatterns(preflight names the Console SD-upload fix).FicTrac bridge (
bridge.py+js/fictrac-bridge-client.js+ README) —log_export/log_export_result(close the active log, stream it back whole to the asking client; retry-safe),WS_MAX_SIZE16 MiB,--log-dir, and a single-in-flightexportLog()with timeout + disconnect rejection.arena_studio.html(v0.5) — File-menu repo/bench-id/direct-commit settings (localStorage; PR and direct modes mutually exclusive); universal bridge logging at run start +run_metadataJSONL line; on COMPLETED runs, export + direct-commit the log torunlogs/<bench-id>/<proto>__<exp>__<STAMP>__<run_id>.jsonl(colon-free) with graceful failure notices; roster prefill + MAC (0xC2) cross-check chip; save →protocols/<bench-id>/; "Open from library…" + "Open from course repo…" pickers; promote-to-shared with a byte-compare guard; repo-hosted.patpreview byte-sources.pattern_editor.html(v0.9.41) — "⇪ Push to course repo" →protocols/<bench-id>/<protocol>_patterns/using the same stored PAT/repo/bench-id.Docs —
docs/development/cshl-pipeline-test-plan.md(prereqs incl. step-by-step guest account + PAT, Session-1 hardware refs, browser dry-run, hardware end-to-end, multi-bench/edge, measured file-size limits, use-case narratives, time budget); proposal doc marked course-scope shipped.Verification
pixi run test→ 761/761 (new coverage:b64Bytes/allowlists/raw/directCommit,?repo=codec, gate additions,exportLog).log_exportverified live incl. a >1 MiB round-trip.reiserlab/cshl-2026-course-data(created this session; test artifacts cleaned): protocol YAML byte-exact,.patpush, promote + collision guard, runlog commit (colon-free), roster prefill + MAC chip.js/*.jsformatted.Not in this PR (tracked, not regressions)
frame_rate(§C2): Console path bench-verified (2026-07-03); the editor→Run path is still to bench-verify. Trigger-input / Mode-4 AI calibration explicitly deferred.🤖 Generated with Claude Code