Simulation mode for experiments

[nikbpetrov]

Abstract (tl;dr)

Want to automatically run through your experiment, simulate a real participant and save the data? The simulate method does exactly this.

To use:

1. Add 3 files to your project:

• Replace your existing jspsych.js with the new one
• Add the get_simulate_method.js and the modify_trial_options_for_simulation.js to your project together in the same directory as the jspsych.js

2. Open the jspsych.js and set the location 2 new scripts (the same source as you would add to your index.html)

Essentially, you will be adding the two new scripts in your index.html but you will do that within jspsych.js. For example, if your folder structure looks like this

---index.html
---js
------jspsych.js
------get_simulate_method.js
------modify_trial_options_for_simulation.js

then the source for the scripts will need to be set to (this is the default):

simulate_method_script.src = 'js/get_simulate_method.js'
simulate_options_script.src = 'js/modify_trial_options_for_simulation.js'

3. Run your experiment in simulation mode

Add simulate: true on the jsPsych.init method like this:

jsPsych.init({
  simulate: true,
  timeline: timeline
});

NB The current implementation of the simulate method is likely not backwards compatible with older versions of jsPsych. Ensure you are using >v6.3 if possible.

For details, additional parameters and more, read on.

1. Functionality

This PR implements a simulate method for all available plugins (see #647). This method simulates random user behaviour on all trials in the experiment and allows to quickly go through an experiment, potentially spotting problems, and generates random data in the end.

1.1. Running the experiment in simulation mode

The simulate method can be run by setting the simulate parameter on the jsPsych.init method to true (boolean), which is a sufficient condition to run the experiment in simulation mode. This would look like this:

jsPsych.init({
  simulate: true,
  timeline: timeline
});

It is also possible to simulate a specific trial by setting its simulate parameter (which is a universal parameter across all plugins) to true. If simulate is set on any specific trial, the trial-specific definition will take precedence (see more about specificity below). For example, in this case the global simulate is set to false (it could very well be left undefined, it does not matter) on the jsPsych.init method, but the trial's simulate parameter is set to true. Hence, this specific trial will be run in simulation mode (example from demo-simple-rt-task.html):

...
var welcome = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin.",
  simulate: true
};
...
jsPsych.init({
  simulate: false,
  timeline: [welcome]
});

1.2. Simulation options

When simulation mode is activated, each trial's parameters can be modified during the simulation. The simulations parameters can be controlled either via the jsPsych.init method or by explicitly defining them on any individual trial. Timeline variables and functions can also be used, just like they would be set outside of simulation mode.

If a trial's parameter is not set to a specific value in simulation mode, the trial-defined value will be used.

1.2.1. Simulation options defined in the jsPsych.init method

Simulation options on the jsPsych.init method can be defined in two different ways, depending on the level of specificity needed.

They can be defined in a simulate_all_trials_opts parameter. This parameter takes in an object as its value that overwrites all trials' parameters during simulation mode. For instance, we can overwrite all trials' stimulus parameter during simulation mode like so:

...
var welcome = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin."
};
...
jsPsych.init({
  simulate: true,
  simulate_all_trials_opts: {'stimulus': '<p>Only this text will display during simulation mode<p>'},
  timeline: [welcome]
});

Thus, our welcome trial will have the text defined in simulate_all_trials_opts.stimulus as its value. The original stimulus will display outside of simulation mode.

Simulations options can also be defined in the jsPsych.init method in the simulate_trial_type_opts parameter. This parameter takes in key-value pairs as its values, whereby the key is a specific trial type (i.e. plugin name, e.g. html-keyboard-response, image-button-response, survey-multi-choice), and the values of the keys are objects that overwrite the parameters of the specific trial type during simulation mode. This would look like this:

...
var welcome = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin."
};
...
jsPsych.init({
  simulate: true,
  simulate_trial_type_opts: {'html-keyboard-response': {'stimulus': '<p>Only this text will display during simulation mode for html-keyboard-response trials<p>'   }},
  timeline: [welcome]
});

1.2.2. Simulation options defined on a specific trial

In addition to defining simulation options within the jsPsych.init method, all trials take a universal simulate_opts parameter, which accepts an object that defines the the trial's parameters to overwrite. For example:

...
var welcome = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin.",
  simulate_opts: {'stimulus': '<p>Only this text will display during simulation mode</p>'}
};
...
jsPsych.init({
  simulate: true,
  timeline: [welcome]
});

Note that this parameter is updated during runtime in order to show specifically which trial's parameters are changed during the simulation mode -- see the "A trial's simulation_opts values" section below.

1.2.3. Simulation response time

An additional parameter can be set during simulation mode, namely simulate_response_time. This is a universal plugin that can be set on any one specific trial. Here are example of how this can be set successfully:

var welcome1 = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin.",
  simulate_response_time: 500
};

var welcome2 = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin.",
  simulate_opts: {'simulate_response_time': 500}
};

var welcome3 = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin."
};
jsPsych.init({
  simulate: true,
  simulate_trial_type_opts: {'html-keyboard-response': {'simulate_response_time': 500}},
  timeline: [welcome3]
});

var welcome4 = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin."
};
jsPsych.init({
  simulate: true,
  simulate_all_trials_opts: {'simulate_response_time': 500},
  timeline: [welcome4]
});

1.2.4. Simulation options precedence

Given that simulation options can be set in a few different ways, it is possible that a trial's parameter is defined in a few different places. In the cases of conflicts, the more specific trial parameter definition takes over. Hence, the simulation trial specificity is as follows:

Simulation trial parameters defined on a specific trial (simulate_opts for any given trial) >>>

Simulation trial parameters defined for a specific trial type (simulate_trial_type_opts on the jsPsych.init method) >>>

Simulation trial parameters defined for all trials (simulate_all_trials_opts on the jsPsych.init method)

1.2.5. A trial's simulation_opts values

During runtime the simulation_opts of an individual trial are overwritten to let the user know which trial's parameters are set for the simulation and where they are coming from. The simulation_opts value is a nested object looking like this:

simulation_opts: {
  'from_simulate_all_trials_opts': {...},
  'from_simulate_trial_type_opts': {...}, 
  'from_trial_simulate_opts': {...}
}

where each key tells you where a specific parameter is set from. Here's an example of a trial object. Note how during trial definition there is conflict between trial durations: the trial parameter is set to 2000, but during simulation, the trial duration is set to 1500 for all trials, to 1200 for all html-keyboard-responses, and to 1000 for the specific trial. In the end, only the trial duration within simulate_opts of the specific trial gets successfully applied during simulation as this is the most specific one. This is then reflected in the trial object during runtime.

var trial = {
  type: "html-keyboard-response",
  stimulus: "Welcome to the experiment. Press any key to begin.",
  choices: jsPsych.ALL_KEYS,
  trial_duration: 2000,
  simulate_opts: {'trial_duration': 1000}
  
};
jsPsych.init({
  simulate: true,
  simulate_all_trials_opts: {'post_trial_gap': 0, 'simulate_response_time': 500, 'trial_duration': 1500},
  simulate_trial_type_opts: {'html-keyboard-response': {'prompt': '<p>My prompt</p>', 'trial_duration': 1200}}
  timeline: [trial]
});

And the evaluated trial object would look like this during simulation mode:

{
  choices: "allkeys",
  post_trial_gap: 0,
  prompt: '<p>My prompt</p>',
  response_ends_trial: true,
  simulate_opts: {
    from_simulate_all_trials_opts: {
      'post_trial_gap': 0,
      'simulate_response_time': 500,
    }, 
    from_simulate_trial_type_opts: {
      'html-keyboard-response': {'prompt': '<p>My prompt</p>'}
    }, 
    from_trial_simulate_opts: {
      'trial_duration': 1000
    }
  },
  simulate_response_time: 500,
  stimulus: "<div>Stimulus during simulation</div>",
  stimulus_duration: null,
  trial_duration: 1000,
  type: "html-keyboard-response"
}

1.2.6. The 'same_as_simulate_response_time' keyword

Simulation options can also be set using the same_as_simulate_response_time keyword. This allows to standardize runtime for some plugins. During runtime, the keyword is evaluated to be the same as the set response time -- see default values below for an example.

1.2.7. Simulation options default values

1.2.7.1. Default values on the jsPsych.init method

The entire experiment will not run in simulate mode by default (simulate: false). But if it does run, the simulated response will occur in 500ms (simulate_response_time': 500). It is possible that the trial duration is shorter than the simulate_response_time -- in this case the trial will continue without a response.

Some trial-type specific parameters are also set to be the same as the simulate_response_time, using the same_as_simulate_response_time to ensure smooth and quick simulation running of the experiment by default, but these can be overwritten by the user. The default post_tral_gap is also set to 0 on all trials (note that it's a universal parameter so it will apply everywhere) to ensure smoother default running.

NB Setting the post_trial_gap parameter means that the default_iti on the jsPsych.init method will never be evaluated. Revert the value of post_trial_gap to null manually if you want the default_iti to be evaluated during simulation mode.

'simulate': false,
'simulate_all_trials_opts': {
  'simulate_response_time': 500,
  'post_trial_gap': 0
},
'simulate_trial_type_opts': {
  'animation': {'frame_time': 'same_as_simulate_response_time', 
                'frame_isi': 'same_as_simulate_response_time'},
  'audio-button-response': {},
  'audio-keyboard-response': {},
  'audio-slider-response': {},
  'call-function': {},
  'canvas-button-response': {},
  'canvas-keyboard-response': {},
  'canvas-slider-response': {},
  'categorize-animation': { 'frame_time': 'same_as_simulate_response_time', 
                            'feedback_duration': 'same_as_simulate_response_time'},
  'categorize-html': {'feedback_duration': 'same_as_simulate_response_time'},
  'categorize-image': {'feedback_duration': 'same_as_simulate_response_time'},
  'cloze': {},
  'external-html': {},
  'free-sort': {},
  'fullscreen': {'delay_after': 'same_as_simulate_response_time'},
  'html-button-response': {},
  'html-keyboard-response': {},
  'html-slider-response': {},
  'iat-html': {},
  'iat-image': {},
  'image-button-response': {},
  'image-keyboard-response': {},
  'image-slider-response': {},
  'instructions': {},
  'maxdiff': {},
  'preload': {},
  'rdk': {},
  'reconstruction': {},
  'resize': {},
  'same-different-html': {'gap_duration': 'same_as_simulate_response_time', 
                          'first_stim_duration': 'same_as_simulate_response_time', 
                          'second_stim_duration': 'same_as_simulate_response_time'},
  'same-different-image': { 'gap_duration': 'same_as_simulate_response_time', 
                            'first_stim_duration': 'same_as_simulate_response_time', 
                            'second_stim_duration': 'same_as_simulate_response_time'},
  'serial-reaction-time-mouse': {'pre_target_duration': 'same_as_simulate_response_time'},
  'serial-reaction-time': { 'feedback_duration': 'same_as_simulate_response_time', 
                            'pre_target_duration': 'same_as_simulate_response_time'},
  'survey-html-form': {},
  'survey-likert': {},
  'survey-multi-choice': {},
  'survey-multi-select': {},
  'survey-text': {},
  'video-button-response': {},
  'video-keyboard-response': {},
  'video-slider-response': {},
  'virtual-chinrest': {},
  'visual-search-circle': {'fixation_duration': 'same_as_simulate_response_time'},
  'vsl-animate-occlusion': {'pre_movement_duration': 'same_as_simulate_response_time', 'cycle_duration': 'same_as_simulate_response_time'},
  'vsl-grid-scene': {'trial_duration': 'same_as_simulate_response_time'}

1.2.7.2. Default values of the trial-specific simulation-related parameters

As already mentioned there are 3 new universal parameters for all plugins.

The first one is simulate, which is a boolean with a default value of null as simulate is, by default, set from the simulate_all_trials_opts of the jsPsych.init method.

The second one is simulate_response_time, which is an integer with a default value of null as simulate_response_time is, by default, set from the simulate_all_trials_opts of the jsPsych.init method.

The third one is simulate_opts, which takes in an object whose default value is {} -- this defines trial-specific parameters to modify during runtime (but also see the "A trial's simulation_opts values" section above).

2. Overview of code changes

Note that the current PR is purposely highly modularised as the goal is to make as FEW changes to the existing jsPsych code base as possible. This not only allows easier code review/troubleshooting, but also allows users to use the functionality by adding it to their own projects while the PR is being reviewed. For each change below, I will suggest where it might land after code review to be integrated within jsPsych but that is up to the reviewers.

2.1. Updates to jsPsych core module

• At the top (lines 4-11) there is additional code that adds the two new simulation-related javascript files to the project. That means that users will not have to change their index.html, but will have to change these paths instead.
  • This is preserved as it is especially useful during code review as if you are going through multiple different .html files, it's more cumbersome to change the index.html of each one. During code review, change the directory filepath to start with .. instead of js to locate the files. This will obviously not be preserved at all in the final product.
• At the top (lines 12-24) there is also a new function added, namely Object_assign_nested which is an upgraded recursive version of the built-in Object.asign. This is then used to assign defaults to the options in the jsPsych.init method on line 206. This is needed because the default simulation-related variables are nested.
  • I expect that this will be integrated within the core module in the final product, but will not be exposed to the global namespace.
• Setting default values for the simulate, simulate_all_trials_opts and simulate_trial_type_opts parameters of the core module (lines 131-191).
• The crucial changes happen in the doTrial function (starts at line 1059).
  • Lines 1073-1079 - After the setDefaultValues function call, the options for the current trial are modified based on the user specifications if simulation is requested. Notably, timeline variables and function parameters are re-evaluated. This will likely need to be modified to prevent 2 repeated function calls but would likely require changes to the infrastructure either of the doTrial function or a new doSimulation function
    • note that it is preferred that the trial method is still executed even in simulation mode as there is a lot of necessary handling within the trial method (even listeners, timeout handling etc.)
  • Lines 1133-1137 - After all trial-related stuff are executed (trial method, on_load functions, extensions etc), then, if we are in simulation mode, the trial's simulate method is set and executed.
• Adding the 3 new universal trial parameters (simulate, simulate_response_time and simulate_opts) - lines 1409-1426

2.2. Adding modify_trial_options_for_simulation.js

• First, specify plugins that do not have a simulate method (preload, call-function, external-html, virtual-chinrest, webgazer-calibrate, webgazer-init-camera, webgazer-validate, free-sort) and send a warn message in the console.
  • of those only free-sort is expected to have a simulate method but does not have one yet.
• If the trial can be simulated, get all of its simulation-related options and save them in an object based on their specificity (i.e. where they are coming from -- see "A trial's simulation_opts values" section above).
• Finally, set the trial's options based on the simulation options, while evaluating the same_as_simulate_response_time keyword and saving the options in the trial's simulate_opts parameter so that the user can see/troubleshoot easier
• I expect this entire file's functionality to be implemented in the core module later.

2.3. Adding get_simulate_method.js

• For each plugin, define the trial.simulate method and return the plugin.
• In essence, what each simulate method does is it sets the trial's values to some random stuff and then tries to continue by either pressing a specific key or clicking a button. If there is no user-specified way to continue, in most cases the simulation mode will wait for the trial duration to trigger the end of the trial. Hence, it is recommended that users specify that too... this can be added as a default value in the simulate_all_trials_opts if preferred.
• There are a few helper funcs/values at the end of the file that can be integrated in any way the reviewers see fit -- it'd perhaps be wise to not expose them to the global scope.
• I expected each plugin simulate method to be integrated within the file of each plugin later on.

2.4. examples-with-simulate folder

This folder is a copy-paste of the examples folder with the only difference that all .html files now have simulate: true in their jsPsych.init calls

3. Some technical notes

There are a lot of notes (in the form of comments) throughout the code on what I think needs to be reviewed, improved etc, but here I make some more big-picture notes.

• Code repetition
  • sometimes code is repeated but this is mostly for readability, e.g. in the get_simulate_method.js it is possible to replace the repetition of the timeout at the end of each simulate method by abstracting it in a global function but would make it less readable
  • some plugin files have exactly the same code for their simulate methods (e.g. same-different-html and same-different-image) but it's better to repeat it as this is likely to go in separate files later (as mentioned above)
• Where to place the modification of trial options for simulation
  • before on_start and after default just to make sure that afterwards on_start and on_load functions can successfully affect the trial

3.1. Potential bugs (mostly with custom setups)

• setting on_start, on_load and on_finish function in the simulation might not work as expected as things are currently set up
• vsl-animate-occlusion offset
• Problems might arise with various simulations if validations/additional functionality is introduced with custom on_start, on_load functions (e.g. custom validation for the survey plugins)
  • One such example is that in any button-clicking plugin, the button might be disabled (disabled attribute set to disabled) in an on_load function. The simulate method is then expected to behaviour weirdly and not to take that into account. This could later be addressed by adding some additional handling.

3.2. Potential list of to-do’s later on

• Documentation -- I am happy to be involved in writing up the documentation for this functionality after the PR is prepared for merge
• Add functionality to overwrite RT data with random values for analysis -- currently the average RT will be approximately equal to whatever the simulation_response_time parameter is and have a very tiny variance. For people who want to quickly simulate data and do mock analysis, that might be suoptimal. A new universal parameter can be added (something like simulate_random_rt_data set to true/false) that will overwrite the data object at the end such that all RT values are random numbers in some range.
• The free-sort of plugin's simulate method is not implemented. It's a confusing plugin...
• Some of the functionality within the simulation method could be parameterized (e.g. how many times to press each key for the reconstruction plugin) but that might add unnecessary complexity

Disclaimer

I am not an expert on using git so hopefully I've done this correctly.

Not a coding expert either, just an enthusiast that googles profusely.

[jodeleeuw]

@nikbpetrov,

This is awesome! I'm hoping to add a simulation mode like this fairly soon (before the end of November at the latest). I'm sorry I missed this PR when you first submitted it. Are you interested in collaborating on updating this for version 7.0, and perhaps tweaking some of the implementation?

[nikbpetrov]

@jodeleeuw ,

Yes, I am happy to collaborate on this. I've continually used this in my own projects (which was the main purpose I developed it) and so far it has been spotless. Even in the most customized projects I am doing right now, it still holds up. The only issue I have encountered, unfortunately, is that it is not fully backwards compatible with older jsPsych versions (due to updates to event listeners) but not difficult to tweak.

I've been keeping an eye out on the work on modernising the code in jsPsych, so hopefully I can be of help. Let me know how you would like to proceed.

[jodeleeuw]

@bjoluc and @becky-gilbert also curious to get your perspectives on what the final API should look like here.

I think so much of this already makes perfect sense. A few things that I thought of after a first read through:

• Now that we have jsPsych.run() we could offer an alternative with jsPsych.simulate(), which could allow passing a timeline plus some simulation specific parameters. That way the experiment options in initJsPsych don't need to be modified during simulation.
• It would be nice to have some kind of simulation JSON that specified particular kinds of actions. Almost like a testing spec for the experiment. That way if you have an experiment that branches in different directions you could write different testing JSON objects. I imagine that these could look very much like the API you've already got, but with the ability to specify particular responses for particular trials without modifying the trial object itself. I can think of a few clunky ways to do this, but not an elegant one...
• I agree that the simulate method should probably be part of the plugin package, but I'm not sure what the elegant way to do this is that can fail gracefully if a plugin doesn't implement it.

[bjoluc]

Hi @nikbpetrov and @jodeleeuw,
here are some thoughts of mine:

• jsPsych.simulate() 👍

• I like the plugin-level simulate method approach.

  but I'm not sure what the elegant way to do this is that can fail gracefully if a plugin doesn't implement it.

  I think that's just checking whether the trial's plugin has a simulate method and otherwise probably skipping the trial / requiring manual interaction, isn't it?

• It would be nice to have some kind of simulation JSON that specified particular kinds of actions.

  One way I could think of: Let the jsPsych.simulate() method take an object that maps strings to simulation-specific configuration objects, and have a "universal plugin parameter" simulation_options (much like @nikbpetrov's version) that can be either an object (to specify options on the trial level), or a string (to reference a config entry from the aforementioned object).

• I wouldn't like to add the simulation-specific parameters to the trial object that gets passed to the trial method, but rather catch them beforehand, hide them from the trial method (separation of concerns) and pass them to the simulate method explicitly.

• A few simulation examples are enough (to prevent duplication). We'll need unit tests to test things in-depth anyway.

• I don't think we should specify default simulation parameters globally (style-wise and maintainability-wise). Maybe we can set sensible defaults as default values to options parameters in the plugins' simulate methods directly. If desired, we can still make the options overridable via the jsPsych.simulate() parameters.

• The only issue I have encountered, unfortunately, is that it is not fully backwards compatible with older jsPsych versions (due to updates to event listeners) but not difficult to tweak.

  I think we can make it fully backward compatible?

[jodeleeuw]

I'm coming back to this with the intent to start diving in more.

One design decision that I noticed in your implementation @nikbpetrov is that the simulation mode actually interacts with the real trial content. That's neat!

My initial instinct was that a simulation mode could just skip the whole trial and call finishTrial() with an appropriate-to-the-parameters set of fake data. This would be like an ultra-fast simulation mode where you get data and can check basic logic like whether a timeline executed the correct number of times for a given set of responses.

An advantage of your approach is that the actual trial methods would be called so folks could observe the experiment directly, which seems useful for checking for visual errors among other things.

When you use the simulation mode in your own work, which mode do you think would be more useful?

[nikbpetrov]

@jodeleeuw That's actually a feature I use quite often when I am still testing. The benefit of this approach -- especially during local testing - is that I can call the simulation mode with a speed of 1s per trial and just watch what would happen in the experiment (making sure that logic runs as expected and everything displays as I want). If I just want the data, I can just run the same thing with a speed of 100ms (or less) and get the data in 3-4 seconds or so.

The simulation mode where the time per trial is set to 1s can also be useful for collaborators who just want to run through and see how the experiment looks like.

Another useful feature is that if I use this visual simulation mode I can set simulate: false to specific trials, e.g. imagine a trial that runs if a participant does not pass an attention check, then I can look at that specific trial and what goes on there. Another use I've encountered is to run through the simulation but stop (again with simulate: false at a specific trial) at an inter-block trial, e.g. every 80 trials, display the PPs' score... when displaying that PPs' score sometimes there might be errors (e.g. SOMEONE did not filter the data correctly) and so those errors can be very easily caught if I can just 'skip' to that trial by setting the simulate duration to 100ms and simulate: false to the inter-block trial -- at that point I can even use the console to make sure I get the right data etc you get the point.

Hope this helps!

[jodeleeuw]

Hey @nikbpetrov I'm going to close this because it can't be merged with the 7.0 changes and #2287 accomplishes this now. However, I think it would be worth posting your guide in the discussions for anyone who wants to enable simulation features with v6. Maybe that will be a little easier to find than a closed PR.

Thanks again for all of the inspiration here that made #2287 possible!