Adding a simulation mode

[jodeleeuw]

work in progress

This adds a simulation mode. It's based on #1886, but updated for 7.0 and integrating some of the features more directly into plugins.

The current plan is to allow simulation through changing jsPsych.run(timeline) to jsPsych.simulate(timeline). The simulate() method will also take an optional simulation_mode argument, which will eventually allow both a data-only mode where the experiment doesn't actually run any trials but does generate appropriate data and follows any logic in the timeline, and also an automated mode, where all of the trials run but the interactions are automated through scripting.

Plugins can support any or none of the modes. If a plugin doesn't support simulation, then the trial will be run.

The plan is to add the data-only simulation mode to all plugins for this release, and then add in the second mode as a medium-term goal. Luckily @nikbpetrov has already figured out most of the automation here, so hopefully we can merge in that work to accomplish the goal. All of the checked plugins in the list below support both data-only and visual modes.

This also makes some changes to the jsPsych.randomization module to add some new functionality for sampling from different probability distributions. It also adds a bunch of methods to jsPsych.pluginAPI to support simulated events and other common simulation-related tasks.

Some todo items:

• Support setting simulation_options via timeline variables.
• Add simulate functionality for all plugins.
  • animation
  • audio-button-response
  • audio-keyboard-response
  • audio-slider-response
  • browser-check
  • call-function
  • canvas-button-response
  • canvas-keyboard-response
  • canvas-slider-response
  • categorize-animation
  • categorize-html
  • categorize-image
  • cloze
  • external-html
  • free-sort
  • fullscreen
  • html-button-response
  • html-keyboard-response
  • html-slider-response
  • iat-html
  • iat-image
  • image-button-response
  • image-keyboard-response
  • image-slider-response
  • instructions
  • maxdiff
  • preload
  • reconstruction
  • resize
  • same-different-html
  • same-different-image
  • serial-reaction-time
  • serial-reaction-time-mouse
  • sketchpad
  • survey-html-form
  • survey-likert
  • survey-multi-choice
  • survey-multi-select
  • survey-text
  • video-button-response
  • video-keyboard-response
  • video-slider-response
  • virtual-chinrest
  • visual-search-circle
  • webgazer-calibrate
  • webgazer-init-camera
  • webgazer-validate
• Allow trial-by-trial skipping of simulation mode.
• Documentation
• New tutorial?

[changeset-bot]

🦋 Changeset detected

Latest commit: 81886f0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 41 packages

Name	Type
@jspsych/test-utils	Minor
jspsych	Minor
@jspsych/plugin-animation	Minor
@jspsych/plugin-canvas-button-response	Minor
@jspsych/plugin-audio-button-response	Minor
@jspsych/plugin-audio-keyboard-response	Minor
@jspsych/plugin-audio-slider-response	Minor
@jspsych/plugin-browser-check	Minor
@jspsych/plugin-call-function	Minor
@jspsych/plugin-canvas-keyboard-response	Minor
@jspsych/plugin-canvas-slider-response	Minor
@jspsych/plugin-categorize-animation	Minor
@jspsych/plugin-categorize-html	Minor
@jspsych/plugin-categorize-image	Minor
@jspsych/plugin-cloze	Minor
@jspsych/plugin-external-html	Minor
@jspsych/plugin-fullscreen	Minor
@jspsych/plugin-html-button-response	Minor
@jspsych/plugin-html-keyboard-response	Minor
@jspsych/plugin-html-slider-response	Minor
@jspsych/plugin-iat-html	Minor
@jspsych/plugin-iat-image	Minor
@jspsych/plugin-image-button-response	Minor
@jspsych/plugin-image-keyboard-response	Minor
@jspsych/plugin-image-slider-response	Minor
@jspsych/plugin-instructions	Minor
@jspsych/plugin-maxdiff	Minor
@jspsych/plugin-preload	Minor
@jspsych/plugin-reconstruction	Minor
@jspsych/plugin-same-different-html	Minor
@jspsych/plugin-same-different-image	Minor
@jspsych/plugin-serial-reaction-time	Minor
@jspsych/plugin-serial-reaction-time-mouse	Minor
@jspsych/plugin-survey-likert	Minor
@jspsych/plugin-survey-multi-choice	Minor
@jspsych/plugin-survey-multi-select	Minor
@jspsych/plugin-survey-text	Minor
@jspsych/plugin-video-button-response	Minor
@jspsych/plugin-video-keyboard-response	Minor
@jspsych/plugin-video-slider-response	Minor
@jspsych/plugin-visual-search-circle	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jodeleeuw]

This is at a stage where some review would probably be useful. The core feature set is implemented, and now it's really just a matter of implementing the simulation functions for all the plugins. This would be a good point to change my implementation if it needs changing...

@bjoluc @becky-gilbert @chrisbrickhouse

[becky-gilbert]

Looks great to me!

It looks like there are lots of great options for generating response times, but for key choices it seems to just be a uniform random selection from "ALL_KEYS" or choices. Were you thinking of implementing simulation options to (1) specify the key (button, slider...) response on a given trial, and (2) allow the response to be specified in other ways, e.g. via a function and/or timeline variables? I'm imagining the case where someone wants to generate a set of responses with a certain proportion of correct/incorrect responses (and possibly combine that with different response time distributions).

Another thing I'm wondering is whether it's possible to move some of the general simulation-related bits of code from the plugin and put these somewhere in the jsPsych package (SimulationAPI?), to minimize the simulation-related stuff that has to be added to each individual plugin. It looks like simulation_create_data will need to be plugin-specific, but some of the other functions look like they might apply to other plugins without any changes? And simulation_visual_mode looks like it could be re-used for any keyboard-response plugin. But I haven't looked at / thought about this as much as you so might be missing something here.

Finally, as far as examples/tutorials, in addition to a very basic example, it might be nice to show how to generate a more complicated, realistically-fake data set. I'm thinking this could be done using the experiment's normal timeline variables, but then adding response and RT values to those trials, but where those values are generated from slightly different proportion correct and RT distributions, based on the experimental condition. This isn't a priority though - just a thought for when everything else is finished.

[becky-gilbert]

Another thing I'm wondering is whether it's possible to move some of the general simulation-related bits of code from the plugin and put these somewhere in the jsPsych package (SimulationAPI?), to minimize the simulation-related stuff that has to be added to each individual plugin. It looks like simulation_create_data will need to be plugin-specific, but some of the other functions look like they might apply to other plugins without any changes? And simulation_visual_mode looks like it could be re-used for any keyboard-response plugin. But I haven't looked at / thought about this as much as you so might be missing something here.

Pretty sure you just did this while I was writing the comment... 😆

[jodeleeuw]

Were you thinking of implementing simulation options to (1) specify the key (button, slider...) response on a given trial, and (2) allow the response to be specified in other ways, e.g. via a function and/or timeline variables?

Yes, both of these!

In fact, both already work (somewhat to my surprise!). I added test cases to cover both scenarios thinking that I would then add the features, but they already work. Sometimes the core library is designed reasonably well 😁 .

Finally, as far as examples/tutorials, in addition to a very basic example, it might be nice to show how to generate a more complicated, realistically-fake data set. I'm thinking this could be done using the experiment's normal timeline variables, but then adding response and RT values to those trials, but where those values are generated from slightly different proportion correct and RT distributions, based on the experimental condition. This isn't a priority though - just a thought for when everything else is finished.

Agreed. This would be great!

[nikbpetrov]

Fascinating! Looks really solid! I am continually surprised at how well maintained, organised and written the library is -- I've never written code in TypeScript, yet I still manage to follow the code (or so I think)!

As far as I can see most, if not all, of the features I originally thought about are implemented -- and more! The data-only mode is definitely interesting. I thought about it myself but originally reckoned that if someone needed a fake dataset, they would probably have an easier time generating it via R. But this ends up looking quite useful.

I am a big fan of the 'tag' idea, where I can provide a string for an individual trial's simulation_options and then specify those in my simulation object. A few additional thoughts spring right away (of course, you might have already considered them but here's my $0.02):

• is there a case where I would want to both 'tag' a trial with a string in its simulation_options but still provide trial-specific options? Imagine that I want to alternate between data- and visual-mode simulation: I would need to change at least two parameters: the simulation_mode and the simulation object I pass to the jsPsych.simulate() call. Even more, it is possible that I want a trial to have different options for the different simulation modes and I am not 100% how this will be best handled in the current setup. Would I need to have the same tag for the trial but specified differently in the different simulation objects? What if I have trial 1 and 3 specified as 'bar' and 2 as 'foo' in data-only but I want 1 and 2 specified as 'bar' in visual mode and 3 as 'foo' in data-only?

• Currently a trial's simulation_options is either a string or an object. I think that's useful for parsimony (no need to have a gazillion universal options for plugins) but I wonder whether a simulation_options_tag and simulation_options wouldn't make more intuitive sense given the goal? This could then allow having a simulation object at the end that specifies options either by global options, by tag, or even by plugin type (currently not implemented I think?)

Would love to contribute further, but as I said, I am not familiar with TypeScript so the best I can do is try to follow the changes and try to make suggestions ^_^

[nikbpetrov]

$0.02 more:

• Just using simulation mode for a highly customized experiment (essentially everything is happening in the on_load function) and realizing that the 'data-only' mode suggested for simulation is very useful because the main data I need to save is handled within my custom on_load function, which has intervals and timeouts. In my case, I had to add some random data in the trial's on_finish function if the experiment is run in simulation mode just so I save something.
• Have you thought about adding an additional parameter for the 'data-only' mode so I can generate multiple datasets? Qualtrics has a similar feature and given the current solid implementation, if I am not mistaken, it would really just be a matter of adding an additional parameter and a loop.

[jodeleeuw]

Have you thought about adding an additional parameter for the 'data-only' mode so I can generate multiple datasets? Qualtrics has a similar feature and given the current solid implementation, if I am not mistaken, it would really just be a matter of adding an additional parameter and a loop.

I think one could just loops calls to jsPsych.simulate(), though I'm not 100% sure. That's a good thing to investigate!

[jodeleeuw]

Even more, it is possible that I want a trial to have different options for the different simulation modes and I am not 100% how this will be best handled in the current setup. Would I need to have the same tag for the trial but specified differently in the different simulation objects?

My idea was that if a developer wants different testing configurations then it probably makes sense to write different simulation_options objects and make heavy use of the string tagging. In an extreme case, you could create a unique tag for every trial, and then have different simulation_options objects in separate files that you swap to get different behavior. I think the more likely scenario is that lots of the trials in an experiment can follow the default options, but some set will be tagged and customized in this way.

[jodeleeuw]

I think this is approaching ready to (squash?) merge.

I'll wait a few days before merging in case anyone wants to take a look before I do!

[bjoluc]

One thing I just noticed: Since the updated plugins need the simulation API and we explicitly told changesets not to update peer dependencies, the "jspsych": ">=7.0.0" peer dependency needs to be manually updated to 7.1.0 in all of these now. Also applies to #2350.

I think this is something we should add to #2314 before merging it (and after merging everything else, because the changesets action force-pushes to that branch).