Adding a simulation mode (#2287)

Implements simulation mode in the core library, supporting API features, and plugin support in most plugins.

.changeset/eighty-chairs-happen.md

@@ -0,0 +1,5 @@
+---
+"@jspsych/test-utils": minor
+---
+
+Add `simulateTimeline()` testing utility that mimics startTimeline but calls `jsPsych.simulate()` instead.

.changeset/few-teachers-beg.md

@@ -0,0 +1,5 @@
+---
+"jspsych": patch
+---
+
+The weights argument for `randomization.sampleWithReplacement()` is now explicitly marked as optional in TypeScript. This has no impact on usage, as the implementation was already treating this argument as optional.

.changeset/fifty-cameras-remember.md

@@ -0,0 +1,5 @@
+---
+"@jspsych/plugin-animation": patch
+---
+
+Fixed a bug that caused a crash when `frame_isi` was > 0. This bug was introduced in 1.0.0.

.changeset/hungry-bats-hide.md

@@ -0,0 +1,5 @@
+---
+"jspsych": minor
+---
+
+Added `randomInt(lower, upper)`, `sampleBernoulli(p)`, `sampleNormal(mean, std)`, `sampleExponential(rate)`, and `sampleExGaussian(mean, std, rate, positive=false)` to `jsPsych.randomization`.

.changeset/khaki-fishes-pull.md

@@ -0,0 +1,5 @@
+---
+"jspsych": minor
+---
+
+Added the ability to run the experiment in simulation mode using `jsPsych.simulate()`. See the [simulation mode](https://www.jspsych.org/latest/overview/simulation) documentation for information about how to get started.

.changeset/large-lions-leap.md

@@ -0,0 +1,5 @@
+---
+"jspsych": minor
+---
+
+Added methods to assist with simulation (e.g., `pressKey` for dispatching a keyboard event and `clickTarget` for dispatching a click event) to the PluginAPI module.

.changeset/lazy-parents-sort.md

@@ -0,0 +1,5 @@
+---
+"@jspsych/plugin-canvas-button-response": patch
+---
+
+Fixed a bug that resulted in `data.response` being `NaN` instead of the index of the button.

.changeset/proud-rings-warn.md

@@ -0,0 +1,44 @@
+---
+"@jspsych/plugin-animation": minor
+"@jspsych/plugin-audio-button-response": minor
+"@jspsych/plugin-audio-keyboard-response": minor
+"@jspsych/plugin-audio-slider-response": minor
+"@jspsych/plugin-browser-check": minor
+"@jspsych/plugin-call-function": minor
+"@jspsych/plugin-canvas-button-response": minor
+"@jspsych/plugin-canvas-keyboard-response": minor
+"@jspsych/plugin-canvas-slider-response": minor
+"@jspsych/plugin-categorize-animation": minor
+"@jspsych/plugin-categorize-html": minor
+"@jspsych/plugin-categorize-image": minor
+"@jspsych/plugin-cloze": minor
+"@jspsych/plugin-external-html": minor
+"@jspsych/plugin-fullscreen": minor
+"@jspsych/plugin-html-button-response": minor
+"@jspsych/plugin-html-keyboard-response": minor
+"@jspsych/plugin-html-slider-response": minor
+"@jspsych/plugin-iat-html": minor
+"@jspsych/plugin-iat-image": minor
+"@jspsych/plugin-image-button-response": minor
+"@jspsych/plugin-image-keyboard-response": minor
+"@jspsych/plugin-image-slider-response": minor
+"@jspsych/plugin-instructions": minor
+"@jspsych/plugin-maxdiff": minor
+"@jspsych/plugin-preload": minor
+"@jspsych/plugin-reconstruction": minor
+"@jspsych/plugin-same-different-html": minor
+"@jspsych/plugin-same-different-image": minor
+"@jspsych/plugin-serial-reaction-time": minor
+"@jspsych/plugin-serial-reaction-time-mouse": minor
+"@jspsych/plugin-survey-likert": minor
+"@jspsych/plugin-survey-multi-choice": minor
+"@jspsych/plugin-survey-multi-select": minor
+"@jspsych/plugin-survey-text": minor
+"@jspsych/plugin-video-button-response": minor
+"@jspsych/plugin-video-keyboard-response": minor
+"@jspsych/plugin-video-slider-response": minor
+"@jspsych/plugin-visual-search-circle": minor
+"@jspsych/test-utils": minor
+---
+
+Added support for `data-only` and `visual` simulation modes.

.changeset/strong-crabs-nail.md

@@ -0,0 +1,5 @@
+---
+"@jspsych/plugin-same-different-image": patch
+---
+
+Fixed a bug where the blank screen would not show for the correct duration. Instead it would show very briefly, if at all.

.changeset/tidy-tomatoes-cross.md

@@ -0,0 +1,5 @@
+---
+"jspsych": minor
+---
+
+Added several functions to the `pluginAPI` module in order to support the new simulation feature.

.changeset/tricky-vans-sell.md

@@ -0,0 +1,5 @@
+---
+"@jspsych/plugin-categorize-image": patch
+---
+
+Fixed a bug where the default value of `incorrect_text` was not defined.

docs/developers/plugin-development.md

@@ -216,6 +216,29 @@ trial(display_element, trial){
 
 The data recorded will be that `correct` is `true` and that `rt` is `350`. [Additional data for the trial](../overview/plugins.md#data-collected-by-all-plugins) will also be collected automatically.
 
+## Simulation mode
+
+Plugins can optionally support [simulation modes](../overview/simulation.md).
+
+To add simulation support, a plugin needs a `simulate()` function that accepts four arguments
+
+`simulate(trial, simulation_mode, simulation_options, load_callback)`
+
+* `trial`: This is the same as the `trial` parameter passed to the plugin's `trial()` method. It contains an object of the parameters for the trial.
+* `simulation_mode`: A string, either `"data-only"` or `"visual"`. This specifies which simulation mode is being requested. Plugins can optionally support `"visual"` mode. If `"visual"` mode is not supported, the plugin should default to `"data-only"` mode when `"visual"` mode is requested.
+* `simulation_options`: An object of simulation-specific options.
+* `load_callback`: A function handle to invoke when the simulation is ready to trigger the `on_load` event for the trial. It is important to invoke this at the correct time during the simulation so that any `on_load` events in the experiment execute as expected.
+
+Typically the flow for supporting simulation mode involves:
+
+1. Generating artificial data that is consistent with the `trial` parameters.
+2. Merging that data with any data specified by the user in `simulation_options`.
+3. Verifying that the final data object is still consistent with the `trial` parameters. For example, checking that RTs are not longer than the duration of the trial.
+4. In `data-only` mode, call `jsPsych.finishTrial()` with the artificial data.
+5. In `visual` mode, invoke the `trial()` method of the plugin and then use the artificial data to trigger the appropriate events. There are a variety of methods in the [Plugin API module](../reference/jspsych-pluginAPI.md) to assist with things like simulating key presses and mouse clicks.
+
+We plan to add a longer guide about simulation development in the future. For now, we recommend browsing the source code of plugins that support simulation mode to see how the flow described above is implemented.
+
 ## Advice for writing plugins
 
 If you are developing a plugin with the aim of including it in the main jsPsych repository we encourage you to follow the [contribution guidelines](contributing.md#contributing-to-the-codebase). 

docs/overview/simulation.md

@@ -0,0 +1,177 @@
+# Simulation Modes
+*Added in 7.1*
+
+Simulation mode allows you run your experiment automatically and generate artificial data. 
+
+## Getting Started
+
+To use simulation mode, replace `jsPsych.run()` with `jsPsych.simulate()`.
+
+```js
+jsPsych.simulate(timeline);
+```
+
+This will run jsPsych in the default `data-only` simulation mode. 
+To use the `visual` simulation mode you can specify the second parameter.
+
+```js
+jsPsych.simulate(timeline, "data-only");
+jsPsych.simulate(timeline, "visual");
+```
+
+## What happens in simulation mode
+
+In simulation mode, plugins call their `simulate()` method instead of calling their `trial()` method.
+If a plugin doesn't implement a `simulate()` method, then the trial will run as usual (using the `trial()` method) and any interaction that is needed to advance to the next trial will be required. 
+If a plugin doesn't support `visual` mode, then it will run in `data-only` mode.
+
+### `data-only` mode
+
+In `data-only` mode plugins typically generate resonable artificial data given the parameters specified for the trial. 
+For example, if the `trial_duration` parameter is set to 2,000 ms, then any response times generated will be capped at this value. 
+Generally the default data generated by the plugin randomly selects any available options (e.g., buttons to click) with equal probability. 
+Response times are usually generated by sampling from an exponentially-modified Gaussian distribution truncated to positive values using `jsPsych.randomization.sampleExGaussian()`.
+
+In `data-only` mode, the plugin's `trial()` method usually does not run. 
+The data are simply calculated based on trial parameters and the `finishTrial()` method is called immediately with the simulated data.
+
+### `visual` mode
+
+In `visual` mode a plugin will typically generate simulated data for the trial and then use that data to mimic the kinds of actions that a participant would do. 
+The plugin's `trial()` method is called by the simulation, and you'll see the experiment progress in real time. 
+Mouse, keyboard, and touch events are simulated to control the experiment.
+
+In `visual` mode each plugin will generate simulated data in the same manner as `data-only` mode, but this data will instead be used to generate actions in the experiment and the plugin's `trial()` method will ultimately be responsible for generating the data. 
+This can create some subtle differences in data between the two modes. 
+For example, if the simulated data generates a response time of 500 ms, the `data.rt` value will be exactly `500` in `data-only` mode, but may be `501` or another slightly larger value in `visual` mode.
+This is because the simulated response is triggered at `500` ms and small delays due to JavaScript's event loop might add a few ms to the measure.
+
+## Controlling simulation mode with `simulation_options`
+
+The parameters for simulation mode can be set using the `simulation_options` parameter in both `jsPsych.simulate()` and at the level of individual trials.
+
+### Trial-level options
+
+You can specify simulation options for an individual trial by setting the `simulation_options` parameter.
+
+```js
+const trial = {
+  type: jsPsychHtmlKeyboardResponse,
+  stimulus: '<p>Hello!</p>',
+  simulation_options: {
+    data: {
+      rt: 500
+    }
+  }
+}
+```
+
+Currently the three options that are available are `data`, `mode`, and `simulate`.
+
+#### `data`
+
+Setting the `data` option will replace the default data generated by the plugin with whatever data you specify.
+You can specify some or all of the `data` parameters. 
+Any parameters you do not specify will be generated by the plugin.
+
+In most cases plugins will try to ensure that the data generated is consistent with the trial parameters.
+For example, if a trial has a `trial_duration` parameter of `2000` but the `simulation_options` specify a `rt` of `2500`, this creates an impossible situation because the trial would have ended before the response at 2,500ms.
+In most cases, the plugin will act as if a response was attempted at `2500`, which will mean that no response is generated for the trial. 
+As you might imagine, there are a lot of parameter combinations that can generate peculiar cases where data may be inconsistent with trial parameters. 
+We recommend double checking the simulation output, and please [alert us](https://github.com/jspsych/jspsych/issues) if you discover a situation where the simulation produces inconsistent data.
+
+#### `mode`
+
+You can override the simulation mode specified in `jsPsych.simulate()` for any trial. Setting `mode: 'data-only'` will run the trial in data-only mode and setting `mode: 'visual'` will run the trial in visual mode.
+
+#### `simulate`
+
+If you want to turn off simulation mode for a trial, set `simulate: false`.
+
+#### Functions and timeline variables
+
+The `simulation_options` parameter is compatible with both [dynamic parameters](dynamic-parameters.md) and [timeline variables](timeline.md#timeline-variables). 
+Dynamic parameters can be especially useful if you want to randomize the data for each run of the simulation. 
+For example, you can specify the `rt` as a sample from an ExGaussian distribution.
+
+```js
+const trial = {
+  type: jsPsychHtmlKeyboardResponse,
+  stimulus: '<p>Hello!</p>',
+  simulation_options: {
+    data: {
+      rt: ()=>{
+        return jsPsych.randomization.sampleExGaussian(500, 50, 1/100, true)
+      }
+    }
+  }
+}
+```
+
+### Experiment-level options
+
+You can also control the parameters for simulation by passing in an object to the `simulation_options` argument of `jsPsych.simulate()`.
+
+```js
+const simulation_options = {
+  default: {
+    data: {
+      rt: 200
+    }
+  }
+}
+
+jsPsych.simulate(timeline, "visual", simulation_options)
+```
+
+The above example will set the `rt` for any trial that doesn't have its own `simulation_options` specified to `200`. 
+This could be useful, for example, to create a very fast preview of the experiment to verify that everything is displaying correctly without having to wait through longer trials.
+
+You can also specify sets of parameters by name using the experiment-level simulation options.
+
+```js
+const simulation_options = {
+  default: {
+    data: {
+      rt: 200
+    }
+  },
+  long_response: {
+    data: {
+      rt: () => {
+        return jsPsych.randomization.sampleExGaussian(5000, 500, 1/500, true)
+      }
+    }
+  }
+}
+
+const trial = {
+  type: jsPsychHtmlKeyboardResponse,
+  stimulus: '<p>This is gonna take a bit.</p>',
+  simulation_options: 'long_response'
+}
+timeline.push(trial);
+
+jsPsych.simulate(timeline, "visual", simulation_options)
+```
+
+In the example above, we specified the `simulation_options` for `trial` using a string (`'long_response'`).
+This will look up the corresponding set of options in the experiment-level `simulation_options`.
+
+We had a few use cases in mind with this approach:
+
+1. You could group together trials with similar behavior without needing to specify unique options for each trial.
+2. You could easily swap out different simulation options to test different kinds of behavior. For example, if you want to test that a timeline with a `conditional_function` is working as expected, you could have one set of simulation options where the data will cause the `conditional_function` to evaluate to `true` and another to `false`. By using string-based identifiers, you don't need to change the timeline code at all. You can just change the object being passed to `jsPsych.simulate()`.
+3. In an extreme case of the previous example, every trial on the timeline could have its own unique identifier and you could have multiple sets of simulation options to have very precise control over the data output.
+
+## Current Limitations
+
+Simulation mode is not yet as comprehensively tested as the rest of jsPsych.
+While we are confident that the simulation is accurate enough for many use cases, it's very likely that there are circumstances where the simulated behavior will be inconsistent with what is actually possible in the experiment. 
+If you come across any such circumstances, please [let us know](https://github.com/jspsych/jspsych/issues)!
+
+Currently extensions are not supported in simulation mode. 
+Some plugins are also not supported. 
+This will be noted on their documentation page. 
+
+

docs/plugins/audio-button-response.md

@@ -34,6 +34,12 @@ In addition to the [default data collected by all plugins](../overview/plugins.m
 | rt             | numeric | The response time in milliseconds for the subject to make a response. The time is measured from when the stimulus first began playing until the subject's response. |
 | response       | numeric | Indicates which button the subject pressed. The first button in the `choices` array is 0, the second is 1, and so on. |
 
+## Simulation Mode
+
+In `data-only` simulation mode, the `response_allowed_while_playing` parameter does not currently influence the simulated response time. 
+This is because the audio file is not loaded in `data-only` mode and therefore the length is unknown. 
+This may change in a future version as we improve the simulation modes.
+
 ## Examples
 
 ???+ example "Displaying question until subject gives a response"

docs/plugins/audio-keyboard-response.md

@@ -32,6 +32,12 @@ In addition to the [default data collected by all plugins](../overview/plugins.m
 | rt        | numeric | The response time in milliseconds for the subject to make a response. The time is measured from when the stimulus first began playing until the subject made a key response. If no key was pressed before the trial ended, then the value will be `null`. |
 | stimulus  | string  | Path to the audio file that played during the trial. |
 
+## Simulation Mode
+
+In `data-only` simulation mode, the `response_allowed_while_playing` parameter does not currently influence the simulated response time. 
+This is because the audio file is not loaded in `data-only` mode and therefore the length is unknown. 
+This may change in a future version as we improve the simulation modes.
+
 ## Examples
 
 ???+ example "Trial continues until subject gives a response"

docs/plugins/audio-slider-response.md

@@ -39,6 +39,12 @@ In addition to the [default data collected by all plugins](../overview/plugins.m
 | stimulus     | string  | The path of the audio file that was played. |
 | slider_start | numeric | The starting value of the slider.        |
 
+## Simulation Mode
+
+In `data-only` simulation mode, the `response_allowed_while_playing` parameter does not currently influence the simulated response time. 
+This is because the audio file is not loaded in `data-only` mode and therefore the length is unknown. 
+This may change in a future version as we improve the simulation modes.
+
 ## Examples
 
 ???+ example "A simple rating scale"

docs/plugins/browser-check.md

@@ -63,6 +63,16 @@ In addition to the [default data collected by all plugins](../overview/plugins.m
 
 Note that all of these values are only recorded when the corresponding key is included in the `features` parameter for the trial.
 
+## Simulation Mode 
+
+In [simulation mode](../overview/simulation.md) the plugin will report the actual features of the browser, with the exception of `vsync_rate`, which is always 60. 
+
+In `data-only` mode, if `allow_window_resize` is true and the browser's width and height are below the maximum value then the reported width and height will be equal to `minimum_width` and `minimum_height`, as if the participant resized the browser to meet the specifications.
+
+In `visual` mode, if `allow_window_resize` is true and the browser's width and height are below the maximum value then the experiment will wait for 3 seconds before clicking the resize fail button. During this time, you can adjust the window if you would like to.
+
+As with all simulated plugins, you can override the default (actual) data with fake data using `simulation_options`. This allows you to test your exclusion criteria by simulating other configurations.
+
 ## Examples
 
 ???+ example "Recording all of the available features, no exclusions"

docs/plugins/external-html.md

@@ -24,6 +24,10 @@ In addition to the [default data collected by all plugins](../overview/plugins.m
 | url  | string  | The URL of the page.                     |
 | rt   | numeric | The response time in milliseconds for the subject to finish the trial. |
 
+## Simulation Mode
+
+In `visual` simulation mode, the plugin cannot interact with any form elements on the screen other than the `cont_btn` specified in the trial parameters. If your `check_fn` requires other user interaction, for example, clicking a checkbox, then you'll need to disable simulation for the trial and complete the interaction manually.
+
 ## Examples
 
 ### Loading a consent form

docs/plugins/free-sort.md

@@ -38,6 +38,10 @@ moves | array | An array containing objects representing all of the moves the pa
 final_locations | array | An array containing objects representing the final locations of all the stimuli in the sorting area. Each element in the array represents a stimulus, and has a "src", "x", and "y" value. "src" is the image path, and "x" and "y" are the object location. This will be encoded as a JSON string when data is saved using the `.json()` or `.csv()` functions. 
 rt | numeric | The response time in milliseconds for the participant to finish all sorting.
 
+## Simulation Mode
+
+This plugin does not yet support [simulation mode](../overview/simulation.md).
+
 ## Examples
 
 ???+ example "Basic example"