Trial-level Configuration

[bboudaoud-nv]

As a precursor to a possible staircase design as discussed in #390, we are considering adding per-trial configuration to FPSci. This comes w/ several challenges. This is a place to summarize them.

1. There are several FpsConfig parameters that need to be relocated out of the generic class. Things like sessionFeedbackDuration, sessionCompleteFeedback, showMenuBetweenSessions and sessionFeedbackRequireClick don't really make sense to be specified on a per-trial basis. Having said this allSessionCompleteFeedback can still be specified at the session-level for now, so perhaps this is okay?
2. It may make sense to have a new FpsConfig type structure that is only inherited from the experiment to the session, so that session-level config can be inherited from the experiment, but does not need to be inherited by the TrialConfig as well. This applies in places like (1) where we would still like to support experiment-level specification of parameters that apply to all sessions, but won't use this state for individual trials.
3. The FPSciApp/session state machine will need to be updated to call something like FPSciApp::updateSession() on every trial, this should be fairly easily done, but we should think about where/how the management of the trial config is done and any perf impacts this may imply
4. We will need a trialParamsToLog similar to sessParamsToLog (which is another config parameter like those specified in (1) above). This will be the only mechanism for logging configuration that changes between trials. We will need to update the logger for this accordingly
5. Currently questions can only be asked at the end of a session, we should either move the questions array to the session/experiment level (as specified in (2)) or come up with a model for per-trial questions (which I believe will be needed).

[bboudaoud-nv]

Another issue to discuss here is whether we should recreate what we call "controls" (GUI windows only available in developer mode) on a per-trial basis. Currently there are created (optionally) once per session, but could incur substantial overhead if they are updated by every trial...

[bboudaoud-nv]

A change to results this will imply is that every trial can now specify questions. However, currently questions are only specified at the session level. We likely want to consider a way to concisely refer to the trial in which a question was answered and also disambiguate session-level from trial-level questions in results.

Note that this also implies a fork in the usual inheritance-based configuration model. Typically questions specified at an experiment level would just be asked in every session; however, here questions specified at the session-level would be asked at end of session while questions specified at the trial level would be asked at end of trial.

We should like create a list of parameters we expect to work this way.

[jspjutNV]

Another issue to discuss here is whether we should recreate what we call "controls" (GUI windows only available in developer mode) on a per-trial basis. Currently there are created (optionally) once per session, but could incur substantial overhead if they are updated by every trial...

I wonder if there's a way to just update which parameter is bound to a given GUI rather than recreating the whole UI.

[jspjutNV]

Note that this also implies a fork in the usual inheritance-based configuration model. Typically questions specified at an experiment level would just be asked in every session; however, here questions specified at the session-level would be asked at end of session while questions specified at the trial level would be asked at end of trial.

We should like create a list of parameters we expect to work this way.

I agree that questions may need to be specified differently. I feel like they could be associated with a couple trials and the session in a 2 interval forced choice design for example. I'm sure questions aren't the only thing that works differently either. Thanks for thinking through this.

[bboudaoud-nv]

I wonder if there's a way to just update which parameter is bound to a given GUI rather than recreating the whole UI.

Yes this would be highly useful. Last time I checked there wasn't an elegant solution for this in G3D, but it would be useful if we could fine a way to do it.

I agree that questions may need to be specified differently. I feel like they could be associated with a couple trials and the session in a 2 interval forced choice design for example. I'm sure questions aren't the only thing that works differently either. Thanks for thinking through this.

Yes, this is absolutely required for staircasing to work well (there will be even more requirements there to tie the response to the feedback mechanism). In the short-term I'm thinking about this as a solution to a configuration problem, but it will absolutely interact with #390.

[bboudaoud-nv]

As part of the change to trial-level configuration I realized that the previous state machine relied on presenting the reference target during the PresentationState::trialFeedback state, a confusing detail that resulted in a sort of "off-by-1" error where you presented the next trial's reference target during the "current" trial (before calling Session::nextCondition() when transitioning out of PresentationState::trialFeedback).

This resulted in weird behavior around reference targets, so I added a PresentationState::referenceTarget state to the Session state machine. This explicitly handles the presentation of the reference target as part of the trial, and works around some previous code we had to delineate non-task target spawning mechanics. To me this is a solid conceptual improvement, but may have implications other places in the FPSci code base (could be the source of some possible Session::currentState bugs).

[bboudaoud-nv]

There are some additional (non-logging) issues arising in how we manage menus and game state. For example previously we allowed things like "dropping into" the scene to occur at the start of the session, but since this now happens every trial perhaps we want to do something about it.

The management of the user menu is also a challenge as its not clear the showMenuBetweenSessions field should have any meaning between trials, but the input mode (pointer vs FPM) and menu controls may need to be updated to deal with this. Additionally we should decide whether developer mode GUI controls should be updated on every single trial...

Similarly we should look into whether some audio files will now be loaded on a per-trial basis.