Skip to content

feat(jspsych): add record_session option for high-fidelity replay capture#3661

Open
jodeleeuw wants to merge 16 commits intomainfrom
claude/add-hifi-data-recording-jfxSP
Open

feat(jspsych): add record_session option for high-fidelity replay capture#3661
jodeleeuw wants to merge 16 commits intomainfrom
claude/add-hifi-data-recording-jfxSP

Conversation

@jodeleeuw
Copy link
Copy Markdown
Member

Summary

Adds an initJsPsych({ record_session: true }) option that captures everything needed to reconstruct a participant's session as a visual replay. When enabled, jsPsych.getSessionRecording() returns a JSON-serializable SessionRecording (schema_version: 1) that can be persisted alongside trial data.

The default is false, so this is fully opt-in. One flag turns it on with no further configuration.

What gets captured

  • Per-trial parameters — snapshot of trial.trialObject at trial start, with Function.prototype.toString() source preserved for canvas-style stimulus functions.
  • DOM — initial subtree snapshot of #jspsych-content taken at on_load (post-render), then all subsequent mutations (childList, attributes, characterData) tracked via MutationObserver with monotonically assigned node IDs.
  • Inputmousemove (rAF-throttled), mousedown/mouseup/click, touchstart/touchmove/touchend, keydown/keyup, and copy/cut/paste (with text + html payloads). Keyboard and clipboard listeners attach to document in capture phase so they run before any user stopPropagation.
  • Mediaplay/pause/ended/seeked and throttled timeupdate for any <video> or <audio> that appears in the display element.
  • Viewport — initial size, dpr, and visualViewport scale/offsets, plus debounced changes (catches both layout zoom and pinch zoom).
  • RNG — auto-seeds via randomization.setSeed() if not already seeded; Math.random is patched while recording is active to log every call's result into the active trial's rng_calls. The patch is reverted at experiment end.

Architecture

  • New module: packages/jspsych/src/modules/recording.ts (schema types + SessionRecorder class).
  • New global timeline hook onTrialLoad added to TimelineNodeDependencies and fired from Trial.onLoad, so the recorder can snapshot the post-render DOM at the right moment without going through the extension system.
  • JsPsych instantiates the recorder in its constructor when record_session is true, starts it in run() after prepareDom, and stops it on normal completion or abortExperiment (recording reason: "finished" / "aborted").
  • New public method jsPsych.getSessionRecording().

Caveats documented in the option description

  • Text typed into form inputs is captured verbatim — experimenters must inform participants.
  • Canvas/WebGL pixel content and audio waveforms are not captured by DOM mutations; only element parameters and media events are recorded.
  • Math.random is wrapped while recording is active.

Test plan

  • New unit suite packages/jspsych/tests/core/record-session.test.ts (6 tests): opt-out returns undefined, recording structure when enabled, RNG capture, Math.random restoration, key event capture, and metadata.
  • Existing jest suite (369 tests) passes.
  • turbo tsc across all 58 workspace packages typechecks cleanly.
  • Manual smoke test in a real browser with a multi-trial timeline (canvas + video + survey) — not done in this PR; would benefit from a reviewer's eyes.
  • Performance: verify GC pressure from mutation/event records doesn't perturb RT measurement on a long timeline. Recommended before broad adoption.

https://claude.ai/code/session_01WCetXEmRj6Y2cBVsvYz7vM


Generated by Claude Code

…ture

Adds an `initJsPsych({ record_session: true })` option that, when enabled,
captures everything needed to reconstruct a participant's session as a
visual replay: per-trial parameters (with stimulus function source for
canvas-style plugins), an initial DOM snapshot taken at on_load plus all
subsequent mutations within the display element, mouse/touch/keyboard/
clipboard input, video and audio playback events, viewport changes
including pinch zoom, and every Math.random output (Math.random is
patched while recording is active so calls are logged into the active
trial). The recording is retrieved via `jsPsych.getSessionRecording()`
and is JSON-serializable; the schema is documented as schema_version: 1.

Adds a global `onTrialLoad` hook to TimelineNodeDependencies, fired from
Trial.onLoad, so the recorder can snapshot the post-render DOM at the
right lifecycle moment.

https://claude.ai/code/session_01WCetXEmRj6Y2cBVsvYz7vM
@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Apr 29, 2026

⚠️ No Changeset found

Latest commit: c7feb4a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in record_session capability to jsPsych that captures trial parameters, DOM state/mutations, user inputs, media events, viewport changes, and RNG outputs to enable high-fidelity session replay via a new jsPsych.getSessionRecording() API.

Changes:

  • Introduces SessionRecorder and schema types for capturing replay data (schema_version: 1).
  • Adds a new onTrialLoad lifecycle hook to snapshot post-render DOM before trial on_load callbacks.
  • Wires recording lifecycle into JsPsych.run() / abortExperiment() and documents the new option + API; adds unit tests.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
packages/jspsych/src/modules/recording.ts New recorder implementation + JSON-serializable schema for replay capture
packages/jspsych/src/JsPsych.ts Adds record_session option, recorder lifecycle, and getSessionRecording()
packages/jspsych/src/timeline/index.ts Extends dependencies with onTrialLoad hook
packages/jspsych/src/timeline/Trial.ts Fires onTrialLoad before per-trial/extension on_load callbacks
packages/jspsych/tests/core/record-session.test.ts New unit tests covering opt-in recording, RNG capture, event capture, metadata
packages/jspsych/tests/test-utils.ts Updates test dependency mock to include onTrialLoad
docs/reference/jspsych.md Documents record_session option and jsPsych.getSessionRecording()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +65 to +69
test("restores Math.random after the experiment finishes", async () => {
let patchedRandom: () => number;
const jsPsych = initJsPsych({ record_session: true });
await startTimeline(
[
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test only checks that Math.random after the experiment is not the patched wrapper captured during on_load. That can still pass even if record_session permanently replaces Math.random (e.g., via seeding).

Consider capturing const original = Math.random before initJsPsych({ record_session: true }) and asserting it is fully restored after completion.

Copilot uses AI. Check for mistakes.
Comment thread packages/jspsych/src/JsPsych.ts Outdated
Comment on lines 158 to 162
this.sessionRecorder?.start(this.getDisplayElement());

await this.timeline.run();
await Promise.resolve(this.options.on_finish(this.data.get()));

Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run() starts/stops the session recorder, but the stop/unpatch cleanup won't run if timeline.run() or the user-provided on_finish callback throws/rejects. Since recording wraps Math.random and attaches global listeners, this can leak patched state beyond the experiment.

Consider wrapping the body of run() in a try { ... } finally { ... } that always calls this.sessionRecorder?.stop("finished") (or an appropriate reason) and removes interaction listeners, even on error.

Copilot uses AI. Check for mistakes.
Comment on lines +641 to +647
private patchMathRandom() {
if (this.mathRandomPatched) return;
if (this.recording.rng.seed === null) {
this.recording.rng.seed = randomization.setSeed();
}
this.originalMathRandom = Math.random.bind(Math);
const orig = this.originalMathRandom;
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

patchMathRandom() always calls randomization.setSeed() when recording.rng.seed is null, but recording.rng.seed is an internal field that starts as null every time. This means enabling record_session will re-seed (and replace) Math.random even if the user already seeded it earlier, changing experiment behavior.

If the intent is “seed only when not already seeded”, consider tracking seed state in the randomization module (e.g. export getSeed() / isSeeded()), or at minimum avoid re-seeding when Math.random is already a seedrandom PRNG, and store the pre-recording Math.random so it can be restored.

Suggested change
private patchMathRandom() {
if (this.mathRandomPatched) return;
if (this.recording.rng.seed === null) {
this.recording.rng.seed = randomization.setSeed();
}
this.originalMathRandom = Math.random.bind(Math);
const orig = this.originalMathRandom;
private isNativeMathRandom(fn: typeof Math.random) {
return /\{\s*\[native code\]\s*\}/.test(Function.prototype.toString.call(fn));
}
private patchMathRandom() {
if (this.mathRandomPatched) return;
const preRecordingMathRandom = Math.random;
this.originalMathRandom = preRecordingMathRandom.bind(Math);
if (
this.recording.rng.seed === null &&
this.isNativeMathRandom(preRecordingMathRandom)
) {
this.recording.rng.seed = randomization.setSeed();
}
const orig =
this.recording.rng.seed === null
? this.originalMathRandom
: Math.random.bind(Math);

Copilot uses AI. Check for mistakes.
Comment on lines +662 to +665
private unpatchMathRandom() {
if (!this.mathRandomPatched) return;
Math.random = this.originalMathRandom;
this.mathRandomPatched = false;
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unpatchMathRandom() restores Math.random to this.originalMathRandom, but originalMathRandom is captured after the optional call to randomization.setSeed(). If recording auto-seeds, stopping the recorder leaves Math.random as the seeded PRNG rather than restoring the true pre-recording Math.random, which contradicts the docs/tests intent of reverting the patch.

Suggestion: capture the pre-recording Math.random in start() (or before any seeding), and restore exactly that function in stop().

Copilot uses AI. Check for mistakes.
Comment on lines +354 to +358
this.bind(document, "keydown", this.handleKey("key.down"), true);
this.bind(document, "keyup", this.handleKey("key.up"), true);
this.bind(document, "copy", this.handleClipboard("clipboard.copy"), true);
this.bind(document, "cut", this.handleClipboard("clipboard.cut"), true);
this.bind(document, "paste", this.handleClipboard("clipboard.paste"), true);
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keyboard and clipboard listeners are attached at document scope in capture phase, and the recorder pushes events even when targetId(...) returns null. This can record keystrokes/clipboard data originating outside the jsPsych display element (e.g., other page controls), which is a privacy risk and can bloat recordings.

Consider filtering these events to only record when the event target is within this.displayElement (e.g., this.displayElement.contains(e.target as Node)), or when a tracked target id exists.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The privacy concern is fair, but the proposed filter as written would break default usage in real browsers, not just tests.

Structural detail: when display_element is unspecified, jsPsych's prepareDom uses <body> as the containerElement and creates #jspsych-content inside it as the displayElement. So displayElement is the inner div, and body is its parent. With this layout:

  • The container is what gets tabIndex and .focus() at trial start.
  • Real browsers route key events to the focused element (or <body> when nothing is focused). The event target is therefore the container or <body>, not the displayElement div.
  • this.displayElement.contains(<body>) is false (a child can't contain its parent). The proposed filter would drop legitimate keypresses every time focus is on the container or body — which is the default for most plugins.

The same applies to pressKey in tests, which dispatches at body. Tests would also fail.

A version that filters by the container (displayContainerElement.contains(target) || target === document) would be safer, but it requires plumbing the container reference into the recorder — currently only displayElement is exposed. Happy to make that change in a follow-up if we want to enforce a tighter scope.

For v1 my read is that capturing all keydown / clipboard events while the experiment runs is the intended scope (the option's docstring already calls out that text input is captured verbatim). Skipping this fix for now to avoid a regression.


Generated by Claude Code

Comment on lines +631 to +635
media.addEventListener("play", handler);
media.addEventListener("pause", handler);
media.addEventListener("ended", handler);
media.addEventListener("seeked", handler);
media.addEventListener("timeupdate", handler);
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attachMediaListeners() adds event listeners directly to each media element, but they are never removed in detachTrialListeners()/stop(). Over a long session (or with media created/destroyed across trials) this can accumulate handlers and keep the SessionRecorder reachable longer than necessary.

Consider registering these via the same bind(...) mechanism (so they get cleaned up), or explicitly iterating this.mediaListeners on trial finish/stop to remove the listeners from each tracked element.

Copilot uses AI. Check for mistakes.
claude added 7 commits April 29, 2026 19:34
Adds a ScrollRecord event type with two variants:

- scroll.window: fires when the document or window is scrolled
- scroll.element: fires when any tracked element is scrolled, keyed
  by its DOM node id

A single capture-phase scroll listener at document scope catches both
(scroll events do not bubble). High-frequency scroll bursts are
rAF-throttled the same way as mousemove, with a per-target Map of
pending positions flushed at each animation frame and on trial end.

Also extends the test suite with two new cases (window and element
scroll) and updates the option documentation.
Addresses review feedback on the record_session feature.

- Capture the pre-recording `Math.random` *before* any auto-seeding so
  `stop()` restores exactly what was there. Auto-seeding now runs only
  when `Math.random` is the native function (detected via a [native code]
  toString check), so a user-set seeded PRNG is no longer overwritten.

- Wrap the body of `JsPsych.run()` in `try { ... } finally { ... }` so
  the recorder's `Math.random` patch and global listeners are always
  removed even if `timeline.run()` or the user-provided `on_finish`
  callback throws or rejects.

- Track media elements with attached recorder listeners in a Set and
  iterate it on trial end / stop to remove every listener; previously
  the `mediaListeners` WeakMap was never cleaned up explicitly.

- Strengthen the Math.random restoration test to capture the original
  function reference *before* `initJsPsych` and assert exact equality
  after `stop()`, so a future regression that leaves a seeded PRNG in
  place would be caught.
Reworks the recorder lifecycle so that `stop()` and `start()` are both
idempotent and a recorder can be cleanly restarted after stopping. This
removes a class of latent bugs from the previous implementation where
calling `JsPsych.run()` a second time on the same instance would
silently re-patch `Math.random` (wrapper-on-wrapper) and double-attach
session listeners while `stop()` short-circuited via the `end_reason`
guard.

Changes:

- Track lifecycle state in a dedicated `running` flag instead of using
  `recording.end_reason`.
- `start()`: no-op if already running. If the previous recording has
  ended, swap in a fresh `SessionRecording` (the previous object is
  preserved for any caller that captured it via `getRecording()`
  before the restart). Per-session ephemeral state (currentTrial,
  node ids, throttle flags, pendingScroll) is reset.
- `stop()`: no-op if not running. Flushes pending mouse/scroll events
  from any in-flight trial *before* tearing down listeners, so events
  near an abort boundary are no longer silently dropped.
- Adds two tests that lock in the contract: a full stop/start/stop
  cycle produces an independent second recording with no double-patch
  on Math.random, and stop() while a trial is in flight flushes the
  last pending throttled event.
Two cleanup tasks from the review:

1. Move `rng_calls` from `TrialRecording` to `SessionRecording`.

   The previous per-trial bucket silently dropped any `Math.random` call
   that happened with no `currentTrial` set: pre-trial parameter
   evaluation (Trial.run runs `processParameters()` *before* `onStart()`,
   which means a function-valued `stimulus` that uses `Math.random` was
   invisible to the recorder), the post-trial gap, and the experimenter's
   `on_finish` callback. A session-level chronological log captures all
   of them and is also simpler for replayers, which only need to consume
   calls in order.

   The schema is still `schema_version: 1` (we are pre-release).

   New test asserts that calls made inside a function-valued `stimulus`
   appear in `rng_calls` with `t < trials[0].t_start`.

2. Remove the `idToNode: Map<number, Node>` field on `SessionRecorder`.

   It was populated and reset but never read — a remnant of an earlier
   design that anticipated reverse lookups by id. As a strong reference
   to DOM nodes it also briefly pinned them in memory between trials;
   the matching `nodeIds` is a `WeakMap`, so dropping `idToNode` lets
   detached nodes be GC'd promptly.
Drops trial-parameter capture from the recording. The DOM snapshot at
each trial's `on_load` plus the mutation log already capture everything
needed for visual replay; carrying parameter values in parallel was
duplicate state and forced fragile decisions about which parameters
might be functions.

The previous approach also implied — but never really delivered — a
parametric replay path where a replayer would re-execute plugin code
with captured parameters. That path is fragile (closures, plugin-version
drift, RNG state) and not actually needed when the DOM mutations carry
the visual outcome directly.

Schema changes (still `schema_version: 1`, pre-release):

- TrialRecording: drop `trial_params`, `stimulus_source`, and the
  perpetually-null `rng_state_at_start`. Keep `trial_index`, `plugin`,
  timing markers, `initial_dom`, `events`, and `trial_data`.
- Removes the `extractStimulusSource` helper.
- `serializeJson` no longer round-trips functions via a sentinel; it
  drops them. The function is now only used for `trial_data`, which
  doesn't contain functions in practice.

Per-trial `initial_dom` snapshots are kept: jsPsych wipes
`#jspsych-content` between trials, so each trial is a natural DOM
resync point. Replayers can seek to a trial in O(1) by dropping in
its snapshot and applying its events, instead of replaying the entire
session prefix.

Net effect: the recording is purely observational — what was on
screen at trial start, what changed, what data resulted. No code,
no closures, no parameter-vs-function-type distinctions to handle.
…aths; tighten types

Adds the previously-untested capture paths from the review:

- Multi-trial recording: three trials, each with its own initial_dom
  baseline, distinct stimulus rendered into the DOM, and chronological
  ordering across trials.
- Window viewport changes: redefines `innerWidth`/`innerHeight` and
  fires a `resize` event, then waits past the 100ms debounce and
  asserts the new dimensions appear in `viewport_changes`.
- Focus / blur capture: dispatches `blur` and `focus` on `window` and
  asserts both end up in the trial events.
- Media play/pause capture: renders a `<video>` element via the
  stimulus, dispatches `play` and `pause` on it, and asserts the
  corresponding `media.*` events appear.
- Abort flow: an `on_finish` callback that calls `abortExperiment` —
  the recording's `end_reason` is `"aborted"` and only the first trial
  is captured.

Schema/code tightening:

- Replaces the `(b as any).__trial` escape hatch with a properly typed
  `trial?: boolean` field on `boundHandlers` entries. `bind()` sets it
  based on whether the mutation observer is active (i.e. inside
  `attachTrialListeners`).
- Narrows `SessionRecording.rng.seed` from `string | number | null` to
  `string | null`. `randomization.setSeed()` only ever returns a string.

16 record-session tests now; 379 total in the package; tsc clean.
github-actions Bot pushed a commit that referenced this pull request Apr 30, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 30, 2026

📦 Preview build ready

Built from PR head c7feb4a and published at 51aadba on branch preview/pr-3661.
URLs below are pinned to an immutable commit SHA, so they are safe to share and are cached permanently by jsDelivr.

Changed packages: jspsych

Quick-start HTML:

<script src="https://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/jspsych/dist/index.browser.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/jspsych/css/jspsych.css">
<script src="https://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-html-keyboard-response/dist/index.browser.min.js"></script>
All package URLs
  • @jspsych/extension-mouse-trackinghttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/extension-mouse-tracking/dist/index.browser.min.js
  • @jspsych/extension-record-videohttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/extension-record-video/dist/index.browser.min.js
  • @jspsych/extension-webgazerhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/extension-webgazer/dist/index.browser.min.js
  • jspsychhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/jspsych/dist/index.browser.min.js
  • @jspsych/plugin-animationhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-animation/dist/index.browser.min.js
  • @jspsych/plugin-audio-button-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-audio-button-response/dist/index.browser.min.js
  • @jspsych/plugin-audio-keyboard-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-audio-keyboard-response/dist/index.browser.min.js
  • @jspsych/plugin-audio-slider-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-audio-slider-response/dist/index.browser.min.js
  • @jspsych/plugin-browser-checkhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-browser-check/dist/index.browser.min.js
  • @jspsych/plugin-call-functionhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-call-function/dist/index.browser.min.js
  • @jspsych/plugin-canvas-button-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-canvas-button-response/dist/index.browser.min.js
  • @jspsych/plugin-canvas-keyboard-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-canvas-keyboard-response/dist/index.browser.min.js
  • @jspsych/plugin-canvas-slider-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-canvas-slider-response/dist/index.browser.min.js
  • @jspsych/plugin-categorize-animationhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-categorize-animation/dist/index.browser.min.js
  • @jspsych/plugin-categorize-htmlhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-categorize-html/dist/index.browser.min.js
  • @jspsych/plugin-categorize-imagehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-categorize-image/dist/index.browser.min.js
  • @jspsych/plugin-clozehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-cloze/dist/index.browser.min.js
  • @jspsych/plugin-external-htmlhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-external-html/dist/index.browser.min.js
  • @jspsych/plugin-free-sorthttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-free-sort/dist/index.browser.min.js
  • @jspsych/plugin-fullscreenhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-fullscreen/dist/index.browser.min.js
  • @jspsych/plugin-html-audio-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-html-audio-response/dist/index.browser.min.js
  • @jspsych/plugin-html-button-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-html-button-response/dist/index.browser.min.js
  • @jspsych/plugin-html-keyboard-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-html-keyboard-response/dist/index.browser.min.js
  • @jspsych/plugin-html-slider-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-html-slider-response/dist/index.browser.min.js
  • @jspsych/plugin-html-video-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-html-video-response/dist/index.browser.min.js
  • @jspsych/plugin-iat-htmlhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-iat-html/dist/index.browser.min.js
  • @jspsych/plugin-iat-imagehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-iat-image/dist/index.browser.min.js
  • @jspsych/plugin-image-button-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-image-button-response/dist/index.browser.min.js
  • @jspsych/plugin-image-keyboard-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-image-keyboard-response/dist/index.browser.min.js
  • @jspsych/plugin-image-slider-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-image-slider-response/dist/index.browser.min.js
  • @jspsych/plugin-initialize-camerahttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-initialize-camera/dist/index.browser.min.js
  • @jspsych/plugin-initialize-microphonehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-initialize-microphone/dist/index.browser.min.js
  • @jspsych/plugin-instructionshttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-instructions/dist/index.browser.min.js
  • @jspsych/plugin-maxdiffhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-maxdiff/dist/index.browser.min.js
  • @jspsych/plugin-mirror-camerahttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-mirror-camera/dist/index.browser.min.js
  • @jspsych/plugin-preloadhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-preload/dist/index.browser.min.js
  • @jspsych/plugin-reconstructionhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-reconstruction/dist/index.browser.min.js
  • @jspsych/plugin-resizehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-resize/dist/index.browser.min.js
  • @jspsych/plugin-same-different-htmlhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-same-different-html/dist/index.browser.min.js
  • @jspsych/plugin-same-different-imagehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-same-different-image/dist/index.browser.min.js
  • @jspsych/plugin-serial-reaction-time-mousehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-serial-reaction-time-mouse/dist/index.browser.min.js
  • @jspsych/plugin-serial-reaction-timehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-serial-reaction-time/dist/index.browser.min.js
  • @jspsych/plugin-sketchpadhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-sketchpad/dist/index.browser.min.js
  • @jspsych/plugin-survey-html-formhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-survey-html-form/dist/index.browser.min.js
  • @jspsych/plugin-survey-likerthttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-survey-likert/dist/index.browser.min.js
  • @jspsych/plugin-survey-multi-choicehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-survey-multi-choice/dist/index.browser.min.js
  • @jspsych/plugin-survey-multi-selecthttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-survey-multi-select/dist/index.browser.min.js
  • @jspsych/plugin-survey-texthttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-survey-text/dist/index.browser.min.js
  • @jspsych/plugin-surveyhttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-survey/dist/index.browser.min.js
  • @jspsych/plugin-video-button-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-video-button-response/dist/index.browser.min.js
  • @jspsych/plugin-video-keyboard-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-video-keyboard-response/dist/index.browser.min.js
  • @jspsych/plugin-video-slider-responsehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-video-slider-response/dist/index.browser.min.js
  • @jspsych/plugin-virtual-chinresthttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-virtual-chinrest/dist/index.browser.min.js
  • @jspsych/plugin-visual-search-circlehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-visual-search-circle/dist/index.browser.min.js
  • @jspsych/plugin-webgazer-calibratehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-webgazer-calibrate/dist/index.browser.min.js
  • @jspsych/plugin-webgazer-init-camerahttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-webgazer-init-camera/dist/index.browser.min.js
  • @jspsych/plugin-webgazer-validatehttps://cdn.jsdelivr.net/gh/jspsych/jsPsych@51aadba8de0ecad375c3095594c5979cfafdac20/packages/plugin-webgazer-validate/dist/index.browser.min.js

Last updated 2026-05-03 21:46 UTC for PR head c7feb4a.

The exported TypeScript types in `recording.ts` were the only reference
for anyone writing a replayer against `getSessionRecording()`. This adds
a markdown reference under docs/reference/session-recording-schema.md
covering:

- The replayer model (DOM-canonical, no code re-execution)
- Top-level shape with field-by-field documentation
- Per-trial structure and the procedure for replaying a single trial
- The DOM representation (node ids, element extras, what is and isn't
  captured)
- Each event type in the discriminated union (DOM mutations, input,
  clipboard, media, focus, scroll), with field-level meaning
- Viewport state and how to resolve "what was the viewport at time t?"
- The session-level rng_calls log and how to consume it on replay
- End reasons
- Privacy notes (verbatim text capture, clipboard payloads, Math.random
  patching)
- Backward compatibility expectations (additive minor changes ignore
  policy for replayers)

Wires the new page into mkdocs.yml under Reference and cross-links it
from the `getSessionRecording` reference.
claude added 4 commits May 1, 2026 12:51
Previously the recorder only serialized the display element subtree, so
replays reconstructed DOM structure but rendered unstyled — class hooks
like .jspsych-display-element had no rules to attach to.

Snapshot document.styleSheets at session start. For each <style> tag we
record the resolved cssRules text (falling back to textContent); for each
<link rel=stylesheet> we record the href and, when accessible, the
resolved CSS so cross-origin sheets degrade to href-only rather than
breaking. Also covers @import-only sheets that have no DOM owner.
The session-start snapshot only sees stylesheets that exist when start()
is called. Plugins that inject <style> blocks into <head> mid-experiment
(e.g. a survey widget mounting its theme) would still render unstyled in
replay because their rules were never captured.

Add a MutationObserver scoped to document.head that emits stylesheet.add,
stylesheet.remove, and stylesheet.update events with timestamps into a
new top-level stylesheet_events log. Snapshot ids assigned at start are
reused so add/remove/update can reference earlier entries. Pending
records are drained on stop() so changes near the boundary aren't lost.
The MutationObserver only sees the DOM `value` *attribute*; the IDL
`value` property the browser writes when a participant types into a
field, toggles a checkbox or radio, or changes a select is invisible to
it. As a result, replays of survey trials reconstructed the form
controls but rendered them empty.

Add three new InputRecord variants — input.value, input.checked,
input.select — emitted from capture-phase `input`/`change` listeners on
document. input.value is RAF-coalesced (rapid typing produces one event
per frame with the latest value) and flushed at trial finish so the
final value is never lost. input.checked and input.select are
unthrottled and fire on commit. <input type=file> is intentionally
ignored.
The MutationObserver cannot see drawing operations inside <canvas> — the
pixels are not in the DOM tree. Without explicit snapshots, replay
reconstructs the canvas element but renders it blank, so anything the
participant drew (sketchpad strokes, plugin-rendered visualizations) is
invisible in the playback even though cursor motion is recorded.

Introduce a generic canvas.snapshot RecordedEvent that carries a PNG
data URL keyed by node id. Every <canvas> in the trial display element
is tracked; snapshots fire on gesture release (mouseup/touchend, RAF-
deferred so the post-gesture paint has landed), at the moment a canvas
is removed from the DOM (jsPsych core clears the display element via
innerHTML = "" between trials, which would otherwise lose final state),
and on stop() of an in-flight trial. Per-canvas snapshots are throttled
to 250ms and deduped by data URL so unchanged canvases don't produce
noise. Tainted canvases (cross-origin images without CORS) throw on
toDataURL — these are caught and skipped so the recording survives.

The implementation is plugin-agnostic: it walks the display element for
any <canvas>, has no awareness of plugin-sketchpad, and works equally
for any plugin or experiment that draws to a canvas.
github-actions Bot pushed a commit that referenced this pull request May 1, 2026
The recorder serialized only the display element subtree
(<div id="jspsych-content">), so replays were missing the
jspsych-content-wrapper and jspsych-display-element ancestors that the
centering CSS targets. Without those wrappers, .jspsych-content's
margin: auto has no flex parent to center against — content rendered
top-aligned with no vertical centering.

Walk from the display element up to (and including) the configured
display container, wrapping the serialized content in each ancestor's
tag + attributes. Sibling content (e.g. unrelated body elements when
display_element defaults to <body>) is intentionally omitted; the
result is a "spine" that carries just the layout chain.

Pass the display container explicitly via SessionRecorder.start() so
the stopping point is unambiguous and we don't class-sniff our way up
the DOM tree. Backwards-compatible: when no container is given, the
recorder still serializes only the display element.
github-actions Bot pushed a commit that referenced this pull request May 1, 2026
jodeleeuw pushed a commit to jodeleeuw/jspsych-replay-test that referenced this pull request May 1, 2026
Bumps the pinned commit to 7998445b, the latest head of
jspsych/jsPsych#3661 (record_session feature branch).
…play

Replaces per-gesture full-canvas PNGs with an initial baseline plus
incremental partials. Adds an optional `region: {x,y,w,h}` to
`canvas.snapshot`: when omitted the data URL is a full baseline; when
present it is a cropped patch the replayer composites at (region.x,
region.y) without touching the rest of the canvas.

Snapshot triggers:
- initial baseline scheduled at the next animation frame after a
  canvas is registered (`onTrialLoad` or mid-trial `dom.add`), so
  plugins that draw their stimulus synchronously (canvas-button-
  response, canvas-keyboard-response) get baseline coverage instead
  of relying on the trial-end `releaseSubtree` snapshot;
- draw-driven ticks: `start()` patches 9 pixel-mutating methods on
  `CanvasRenderingContext2D.prototype` (`clearRect`, `fillRect`,
  `strokeRect`, `fill`, `stroke`, `fillText`, `strokeText`,
  `drawImage`, `putImageData`). Each wrapper sets a per-canvas dirty
  flag and schedules a coalesced RAF tick. The tick diffs every
  dirty canvas past a 66 ms throttle (~15 Hz cap for animations);
- existing gesture and removal paths retained as safety nets.

Diff path: per-canvas shadow `ImageData` is held while the canvas is
tracked. `getReadable2dContext` + `computeDiffBbox` (edge-shrink walk
that early-exits on the first dirty row from each side) finds the
bounding box of changed pixels; if it covers >=80% of the canvas a
full snapshot is emitted instead. WebGL canvases (no readable 2d
context) always emit full snapshots — same gap as before.

Lifecycle: prototype methods are restored on `stop()` alongside the
existing `Math.random` unpatch. Shadow buffers are released when
canvases leave the tracked set; `WeakMap`s collect the rest.

Tests: 5 unit tests for `computeDiffBbox`; one integration test that
the baseline snapshot lands close to `t_dom_ready`; one that the
prototype methods are wrapped during the trial and restored after
`stop()`; one (with a stub `getImageData`) that a draw after baseline
emits a partial whose `region` matches the changed pixel. All 718
workspace tests pass; turbo tsc is clean across 58 packages.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions Bot pushed a commit that referenced this pull request May 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants