feat(jspsych): add record_session option for high-fidelity replay capture#3661
feat(jspsych): add record_session option for high-fidelity replay capture#3661
Conversation
…ture
Adds an `initJsPsych({ record_session: true })` option that, when enabled,
captures everything needed to reconstruct a participant's session as a
visual replay: per-trial parameters (with stimulus function source for
canvas-style plugins), an initial DOM snapshot taken at on_load plus all
subsequent mutations within the display element, mouse/touch/keyboard/
clipboard input, video and audio playback events, viewport changes
including pinch zoom, and every Math.random output (Math.random is
patched while recording is active so calls are logged into the active
trial). The recording is retrieved via `jsPsych.getSessionRecording()`
and is JSON-serializable; the schema is documented as schema_version: 1.
Adds a global `onTrialLoad` hook to TimelineNodeDependencies, fired from
Trial.onLoad, so the recorder can snapshot the post-render DOM at the
right lifecycle moment.
https://claude.ai/code/session_01WCetXEmRj6Y2cBVsvYz7vM
|
There was a problem hiding this comment.
Pull request overview
Adds an opt-in record_session capability to jsPsych that captures trial parameters, DOM state/mutations, user inputs, media events, viewport changes, and RNG outputs to enable high-fidelity session replay via a new jsPsych.getSessionRecording() API.
Changes:
- Introduces
SessionRecorderand schema types for capturing replay data (schema_version: 1). - Adds a new
onTrialLoadlifecycle hook to snapshot post-render DOM before trialon_loadcallbacks. - Wires recording lifecycle into
JsPsych.run()/abortExperiment()and documents the new option + API; adds unit tests.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/jspsych/src/modules/recording.ts | New recorder implementation + JSON-serializable schema for replay capture |
| packages/jspsych/src/JsPsych.ts | Adds record_session option, recorder lifecycle, and getSessionRecording() |
| packages/jspsych/src/timeline/index.ts | Extends dependencies with onTrialLoad hook |
| packages/jspsych/src/timeline/Trial.ts | Fires onTrialLoad before per-trial/extension on_load callbacks |
| packages/jspsych/tests/core/record-session.test.ts | New unit tests covering opt-in recording, RNG capture, event capture, metadata |
| packages/jspsych/tests/test-utils.ts | Updates test dependency mock to include onTrialLoad |
| docs/reference/jspsych.md | Documents record_session option and jsPsych.getSessionRecording() |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| test("restores Math.random after the experiment finishes", async () => { | ||
| let patchedRandom: () => number; | ||
| const jsPsych = initJsPsych({ record_session: true }); | ||
| await startTimeline( | ||
| [ |
There was a problem hiding this comment.
This test only checks that Math.random after the experiment is not the patched wrapper captured during on_load. That can still pass even if record_session permanently replaces Math.random (e.g., via seeding).
Consider capturing const original = Math.random before initJsPsych({ record_session: true }) and asserting it is fully restored after completion.
| this.sessionRecorder?.start(this.getDisplayElement()); | ||
|
|
||
| await this.timeline.run(); | ||
| await Promise.resolve(this.options.on_finish(this.data.get())); | ||
|
|
There was a problem hiding this comment.
run() starts/stops the session recorder, but the stop/unpatch cleanup won't run if timeline.run() or the user-provided on_finish callback throws/rejects. Since recording wraps Math.random and attaches global listeners, this can leak patched state beyond the experiment.
Consider wrapping the body of run() in a try { ... } finally { ... } that always calls this.sessionRecorder?.stop("finished") (or an appropriate reason) and removes interaction listeners, even on error.
| private patchMathRandom() { | ||
| if (this.mathRandomPatched) return; | ||
| if (this.recording.rng.seed === null) { | ||
| this.recording.rng.seed = randomization.setSeed(); | ||
| } | ||
| this.originalMathRandom = Math.random.bind(Math); | ||
| const orig = this.originalMathRandom; |
There was a problem hiding this comment.
patchMathRandom() always calls randomization.setSeed() when recording.rng.seed is null, but recording.rng.seed is an internal field that starts as null every time. This means enabling record_session will re-seed (and replace) Math.random even if the user already seeded it earlier, changing experiment behavior.
If the intent is “seed only when not already seeded”, consider tracking seed state in the randomization module (e.g. export getSeed() / isSeeded()), or at minimum avoid re-seeding when Math.random is already a seedrandom PRNG, and store the pre-recording Math.random so it can be restored.
| private patchMathRandom() { | |
| if (this.mathRandomPatched) return; | |
| if (this.recording.rng.seed === null) { | |
| this.recording.rng.seed = randomization.setSeed(); | |
| } | |
| this.originalMathRandom = Math.random.bind(Math); | |
| const orig = this.originalMathRandom; | |
| private isNativeMathRandom(fn: typeof Math.random) { | |
| return /\{\s*\[native code\]\s*\}/.test(Function.prototype.toString.call(fn)); | |
| } | |
| private patchMathRandom() { | |
| if (this.mathRandomPatched) return; | |
| const preRecordingMathRandom = Math.random; | |
| this.originalMathRandom = preRecordingMathRandom.bind(Math); | |
| if ( | |
| this.recording.rng.seed === null && | |
| this.isNativeMathRandom(preRecordingMathRandom) | |
| ) { | |
| this.recording.rng.seed = randomization.setSeed(); | |
| } | |
| const orig = | |
| this.recording.rng.seed === null | |
| ? this.originalMathRandom | |
| : Math.random.bind(Math); |
| private unpatchMathRandom() { | ||
| if (!this.mathRandomPatched) return; | ||
| Math.random = this.originalMathRandom; | ||
| this.mathRandomPatched = false; |
There was a problem hiding this comment.
unpatchMathRandom() restores Math.random to this.originalMathRandom, but originalMathRandom is captured after the optional call to randomization.setSeed(). If recording auto-seeds, stopping the recorder leaves Math.random as the seeded PRNG rather than restoring the true pre-recording Math.random, which contradicts the docs/tests intent of reverting the patch.
Suggestion: capture the pre-recording Math.random in start() (or before any seeding), and restore exactly that function in stop().
| this.bind(document, "keydown", this.handleKey("key.down"), true); | ||
| this.bind(document, "keyup", this.handleKey("key.up"), true); | ||
| this.bind(document, "copy", this.handleClipboard("clipboard.copy"), true); | ||
| this.bind(document, "cut", this.handleClipboard("clipboard.cut"), true); | ||
| this.bind(document, "paste", this.handleClipboard("clipboard.paste"), true); |
There was a problem hiding this comment.
Keyboard and clipboard listeners are attached at document scope in capture phase, and the recorder pushes events even when targetId(...) returns null. This can record keystrokes/clipboard data originating outside the jsPsych display element (e.g., other page controls), which is a privacy risk and can bloat recordings.
Consider filtering these events to only record when the event target is within this.displayElement (e.g., this.displayElement.contains(e.target as Node)), or when a tracked target id exists.
There was a problem hiding this comment.
The privacy concern is fair, but the proposed filter as written would break default usage in real browsers, not just tests.
Structural detail: when display_element is unspecified, jsPsych's prepareDom uses <body> as the containerElement and creates #jspsych-content inside it as the displayElement. So displayElement is the inner div, and body is its parent. With this layout:
- The container is what gets
tabIndexand.focus()at trial start. - Real browsers route key events to the focused element (or
<body>when nothing is focused). The event target is therefore the container or<body>, not thedisplayElementdiv. this.displayElement.contains(<body>)isfalse(a child can't contain its parent). The proposed filter would drop legitimate keypresses every time focus is on the container or body — which is the default for most plugins.
The same applies to pressKey in tests, which dispatches at body. Tests would also fail.
A version that filters by the container (displayContainerElement.contains(target) || target === document) would be safer, but it requires plumbing the container reference into the recorder — currently only displayElement is exposed. Happy to make that change in a follow-up if we want to enforce a tighter scope.
For v1 my read is that capturing all keydown / clipboard events while the experiment runs is the intended scope (the option's docstring already calls out that text input is captured verbatim). Skipping this fix for now to avoid a regression.
Generated by Claude Code
| media.addEventListener("play", handler); | ||
| media.addEventListener("pause", handler); | ||
| media.addEventListener("ended", handler); | ||
| media.addEventListener("seeked", handler); | ||
| media.addEventListener("timeupdate", handler); |
There was a problem hiding this comment.
attachMediaListeners() adds event listeners directly to each media element, but they are never removed in detachTrialListeners()/stop(). Over a long session (or with media created/destroyed across trials) this can accumulate handlers and keep the SessionRecorder reachable longer than necessary.
Consider registering these via the same bind(...) mechanism (so they get cleaned up), or explicitly iterating this.mediaListeners on trial finish/stop to remove the listeners from each tracked element.
Adds a ScrollRecord event type with two variants: - scroll.window: fires when the document or window is scrolled - scroll.element: fires when any tracked element is scrolled, keyed by its DOM node id A single capture-phase scroll listener at document scope catches both (scroll events do not bubble). High-frequency scroll bursts are rAF-throttled the same way as mousemove, with a per-target Map of pending positions flushed at each animation frame and on trial end. Also extends the test suite with two new cases (window and element scroll) and updates the option documentation.
Addresses review feedback on the record_session feature.
- Capture the pre-recording `Math.random` *before* any auto-seeding so
`stop()` restores exactly what was there. Auto-seeding now runs only
when `Math.random` is the native function (detected via a [native code]
toString check), so a user-set seeded PRNG is no longer overwritten.
- Wrap the body of `JsPsych.run()` in `try { ... } finally { ... }` so
the recorder's `Math.random` patch and global listeners are always
removed even if `timeline.run()` or the user-provided `on_finish`
callback throws or rejects.
- Track media elements with attached recorder listeners in a Set and
iterate it on trial end / stop to remove every listener; previously
the `mediaListeners` WeakMap was never cleaned up explicitly.
- Strengthen the Math.random restoration test to capture the original
function reference *before* `initJsPsych` and assert exact equality
after `stop()`, so a future regression that leaves a seeded PRNG in
place would be caught.
Reworks the recorder lifecycle so that `stop()` and `start()` are both idempotent and a recorder can be cleanly restarted after stopping. This removes a class of latent bugs from the previous implementation where calling `JsPsych.run()` a second time on the same instance would silently re-patch `Math.random` (wrapper-on-wrapper) and double-attach session listeners while `stop()` short-circuited via the `end_reason` guard. Changes: - Track lifecycle state in a dedicated `running` flag instead of using `recording.end_reason`. - `start()`: no-op if already running. If the previous recording has ended, swap in a fresh `SessionRecording` (the previous object is preserved for any caller that captured it via `getRecording()` before the restart). Per-session ephemeral state (currentTrial, node ids, throttle flags, pendingScroll) is reset. - `stop()`: no-op if not running. Flushes pending mouse/scroll events from any in-flight trial *before* tearing down listeners, so events near an abort boundary are no longer silently dropped. - Adds two tests that lock in the contract: a full stop/start/stop cycle produces an independent second recording with no double-patch on Math.random, and stop() while a trial is in flight flushes the last pending throttled event.
Two cleanup tasks from the review: 1. Move `rng_calls` from `TrialRecording` to `SessionRecording`. The previous per-trial bucket silently dropped any `Math.random` call that happened with no `currentTrial` set: pre-trial parameter evaluation (Trial.run runs `processParameters()` *before* `onStart()`, which means a function-valued `stimulus` that uses `Math.random` was invisible to the recorder), the post-trial gap, and the experimenter's `on_finish` callback. A session-level chronological log captures all of them and is also simpler for replayers, which only need to consume calls in order. The schema is still `schema_version: 1` (we are pre-release). New test asserts that calls made inside a function-valued `stimulus` appear in `rng_calls` with `t < trials[0].t_start`. 2. Remove the `idToNode: Map<number, Node>` field on `SessionRecorder`. It was populated and reset but never read — a remnant of an earlier design that anticipated reverse lookups by id. As a strong reference to DOM nodes it also briefly pinned them in memory between trials; the matching `nodeIds` is a `WeakMap`, so dropping `idToNode` lets detached nodes be GC'd promptly.
Drops trial-parameter capture from the recording. The DOM snapshot at each trial's `on_load` plus the mutation log already capture everything needed for visual replay; carrying parameter values in parallel was duplicate state and forced fragile decisions about which parameters might be functions. The previous approach also implied — but never really delivered — a parametric replay path where a replayer would re-execute plugin code with captured parameters. That path is fragile (closures, plugin-version drift, RNG state) and not actually needed when the DOM mutations carry the visual outcome directly. Schema changes (still `schema_version: 1`, pre-release): - TrialRecording: drop `trial_params`, `stimulus_source`, and the perpetually-null `rng_state_at_start`. Keep `trial_index`, `plugin`, timing markers, `initial_dom`, `events`, and `trial_data`. - Removes the `extractStimulusSource` helper. - `serializeJson` no longer round-trips functions via a sentinel; it drops them. The function is now only used for `trial_data`, which doesn't contain functions in practice. Per-trial `initial_dom` snapshots are kept: jsPsych wipes `#jspsych-content` between trials, so each trial is a natural DOM resync point. Replayers can seek to a trial in O(1) by dropping in its snapshot and applying its events, instead of replaying the entire session prefix. Net effect: the recording is purely observational — what was on screen at trial start, what changed, what data resulted. No code, no closures, no parameter-vs-function-type distinctions to handle.
…aths; tighten types Adds the previously-untested capture paths from the review: - Multi-trial recording: three trials, each with its own initial_dom baseline, distinct stimulus rendered into the DOM, and chronological ordering across trials. - Window viewport changes: redefines `innerWidth`/`innerHeight` and fires a `resize` event, then waits past the 100ms debounce and asserts the new dimensions appear in `viewport_changes`. - Focus / blur capture: dispatches `blur` and `focus` on `window` and asserts both end up in the trial events. - Media play/pause capture: renders a `<video>` element via the stimulus, dispatches `play` and `pause` on it, and asserts the corresponding `media.*` events appear. - Abort flow: an `on_finish` callback that calls `abortExperiment` — the recording's `end_reason` is `"aborted"` and only the first trial is captured. Schema/code tightening: - Replaces the `(b as any).__trial` escape hatch with a properly typed `trial?: boolean` field on `boundHandlers` entries. `bind()` sets it based on whether the mutation observer is active (i.e. inside `attachTrialListeners`). - Narrows `SessionRecording.rng.seed` from `string | number | null` to `string | null`. `randomization.setSeed()` only ever returns a string. 16 record-session tests now; 379 total in the package; tsc clean.
📦 Preview build readyBuilt from PR head Changed packages: Quick-start HTML: <script src="https://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/jspsych/dist/index.browser.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/jspsych/css/jspsych.css">
<script src="https://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-html-keyboard-response/dist/index.browser.min.js"></script>All package URLs
Last updated 2026-05-03 21:46 UTC for PR head |
The exported TypeScript types in `recording.ts` were the only reference for anyone writing a replayer against `getSessionRecording()`. This adds a markdown reference under docs/reference/session-recording-schema.md covering: - The replayer model (DOM-canonical, no code re-execution) - Top-level shape with field-by-field documentation - Per-trial structure and the procedure for replaying a single trial - The DOM representation (node ids, element extras, what is and isn't captured) - Each event type in the discriminated union (DOM mutations, input, clipboard, media, focus, scroll), with field-level meaning - Viewport state and how to resolve "what was the viewport at time t?" - The session-level rng_calls log and how to consume it on replay - End reasons - Privacy notes (verbatim text capture, clipboard payloads, Math.random patching) - Backward compatibility expectations (additive minor changes ignore policy for replayers) Wires the new page into mkdocs.yml under Reference and cross-links it from the `getSessionRecording` reference.
Previously the recorder only serialized the display element subtree, so replays reconstructed DOM structure but rendered unstyled — class hooks like .jspsych-display-element had no rules to attach to. Snapshot document.styleSheets at session start. For each <style> tag we record the resolved cssRules text (falling back to textContent); for each <link rel=stylesheet> we record the href and, when accessible, the resolved CSS so cross-origin sheets degrade to href-only rather than breaking. Also covers @import-only sheets that have no DOM owner.
The session-start snapshot only sees stylesheets that exist when start() is called. Plugins that inject <style> blocks into <head> mid-experiment (e.g. a survey widget mounting its theme) would still render unstyled in replay because their rules were never captured. Add a MutationObserver scoped to document.head that emits stylesheet.add, stylesheet.remove, and stylesheet.update events with timestamps into a new top-level stylesheet_events log. Snapshot ids assigned at start are reused so add/remove/update can reference earlier entries. Pending records are drained on stop() so changes near the boundary aren't lost.
The MutationObserver only sees the DOM `value` *attribute*; the IDL `value` property the browser writes when a participant types into a field, toggles a checkbox or radio, or changes a select is invisible to it. As a result, replays of survey trials reconstructed the form controls but rendered them empty. Add three new InputRecord variants — input.value, input.checked, input.select — emitted from capture-phase `input`/`change` listeners on document. input.value is RAF-coalesced (rapid typing produces one event per frame with the latest value) and flushed at trial finish so the final value is never lost. input.checked and input.select are unthrottled and fire on commit. <input type=file> is intentionally ignored.
The MutationObserver cannot see drawing operations inside <canvas> — the pixels are not in the DOM tree. Without explicit snapshots, replay reconstructs the canvas element but renders it blank, so anything the participant drew (sketchpad strokes, plugin-rendered visualizations) is invisible in the playback even though cursor motion is recorded. Introduce a generic canvas.snapshot RecordedEvent that carries a PNG data URL keyed by node id. Every <canvas> in the trial display element is tracked; snapshots fire on gesture release (mouseup/touchend, RAF- deferred so the post-gesture paint has landed), at the moment a canvas is removed from the DOM (jsPsych core clears the display element via innerHTML = "" between trials, which would otherwise lose final state), and on stop() of an in-flight trial. Per-canvas snapshots are throttled to 250ms and deduped by data URL so unchanged canvases don't produce noise. Tainted canvases (cross-origin images without CORS) throw on toDataURL — these are caught and skipped so the recording survives. The implementation is plugin-agnostic: it walks the display element for any <canvas>, has no awareness of plugin-sketchpad, and works equally for any plugin or experiment that draws to a canvas.
The recorder serialized only the display element subtree (<div id="jspsych-content">), so replays were missing the jspsych-content-wrapper and jspsych-display-element ancestors that the centering CSS targets. Without those wrappers, .jspsych-content's margin: auto has no flex parent to center against — content rendered top-aligned with no vertical centering. Walk from the display element up to (and including) the configured display container, wrapping the serialized content in each ancestor's tag + attributes. Sibling content (e.g. unrelated body elements when display_element defaults to <body>) is intentionally omitted; the result is a "spine" that carries just the layout chain. Pass the display container explicitly via SessionRecorder.start() so the stopping point is unambiguous and we don't class-sniff our way up the DOM tree. Backwards-compatible: when no container is given, the recorder still serializes only the display element.
Bumps the pinned commit to 7998445b, the latest head of jspsych/jsPsych#3661 (record_session feature branch).
…play
Replaces per-gesture full-canvas PNGs with an initial baseline plus
incremental partials. Adds an optional `region: {x,y,w,h}` to
`canvas.snapshot`: when omitted the data URL is a full baseline; when
present it is a cropped patch the replayer composites at (region.x,
region.y) without touching the rest of the canvas.
Snapshot triggers:
- initial baseline scheduled at the next animation frame after a
canvas is registered (`onTrialLoad` or mid-trial `dom.add`), so
plugins that draw their stimulus synchronously (canvas-button-
response, canvas-keyboard-response) get baseline coverage instead
of relying on the trial-end `releaseSubtree` snapshot;
- draw-driven ticks: `start()` patches 9 pixel-mutating methods on
`CanvasRenderingContext2D.prototype` (`clearRect`, `fillRect`,
`strokeRect`, `fill`, `stroke`, `fillText`, `strokeText`,
`drawImage`, `putImageData`). Each wrapper sets a per-canvas dirty
flag and schedules a coalesced RAF tick. The tick diffs every
dirty canvas past a 66 ms throttle (~15 Hz cap for animations);
- existing gesture and removal paths retained as safety nets.
Diff path: per-canvas shadow `ImageData` is held while the canvas is
tracked. `getReadable2dContext` + `computeDiffBbox` (edge-shrink walk
that early-exits on the first dirty row from each side) finds the
bounding box of changed pixels; if it covers >=80% of the canvas a
full snapshot is emitted instead. WebGL canvases (no readable 2d
context) always emit full snapshots — same gap as before.
Lifecycle: prototype methods are restored on `stop()` alongside the
existing `Math.random` unpatch. Shadow buffers are released when
canvases leave the tracked set; `WeakMap`s collect the rest.
Tests: 5 unit tests for `computeDiffBbox`; one integration test that
the baseline snapshot lands close to `t_dom_ready`; one that the
prototype methods are wrapped during the trial and restored after
`stop()`; one (with a stub `getImageData`) that a draw after baseline
emits a partial whose `region` matches the changed pixel. All 718
workspace tests pass; turbo tsc is clean across 58 packages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Adds an
initJsPsych({ record_session: true })option that captures everything needed to reconstruct a participant's session as a visual replay. When enabled,jsPsych.getSessionRecording()returns a JSON-serializableSessionRecording(schema_version: 1) that can be persisted alongside trial data.The default is
false, so this is fully opt-in. One flag turns it on with no further configuration.What gets captured
trial.trialObjectat trial start, withFunction.prototype.toString()source preserved for canvas-style stimulus functions.#jspsych-contenttaken aton_load(post-render), then all subsequent mutations (childList, attributes, characterData) tracked viaMutationObserverwith monotonically assigned node IDs.mousemove(rAF-throttled),mousedown/mouseup/click,touchstart/touchmove/touchend,keydown/keyup, andcopy/cut/paste(with text + html payloads). Keyboard and clipboard listeners attach todocumentin capture phase so they run before any userstopPropagation.play/pause/ended/seekedand throttledtimeupdatefor any<video>or<audio>that appears in the display element.visualViewportscale/offsets, plus debounced changes (catches both layout zoom and pinch zoom).randomization.setSeed()if not already seeded;Math.randomis patched while recording is active to log every call's result into the active trial'srng_calls. The patch is reverted at experiment end.Architecture
packages/jspsych/src/modules/recording.ts(schema types +SessionRecorderclass).onTrialLoadadded toTimelineNodeDependenciesand fired fromTrial.onLoad, so the recorder can snapshot the post-render DOM at the right moment without going through the extension system.JsPsychinstantiates the recorder in its constructor whenrecord_sessionis true, starts it inrun()afterprepareDom, and stops it on normal completion orabortExperiment(recording reason:"finished"/"aborted").jsPsych.getSessionRecording().Caveats documented in the option description
Math.randomis wrapped while recording is active.Test plan
packages/jspsych/tests/core/record-session.test.ts(6 tests): opt-out returns undefined, recording structure when enabled, RNG capture,Math.randomrestoration, key event capture, and metadata.turbo tscacross all 58 workspace packages typechecks cleanly.https://claude.ai/code/session_01WCetXEmRj6Y2cBVsvYz7vM
Generated by Claude Code