NF: Add fullRandom method to MultiStairHandler

[hoechenberger]

MultiStairHandler now supports a fullRandom randomization method. We do this by sampling with replacement from the list of all running staircases on each "pass".

To make this testable, I also added a randomSeed kwarg to MultiStairHandler, which is used to seed a np.random.RandomState RNG. All randomization operations are then being done through that object.

Closes GH-2635.

[hoechenberger]

So the tests fail because my changes actually exposed another bug in the previous implementation.

The method kwarg was never checked for valid values. In cases where the user wishes to invoke multiple QUEST staircases and to pass down e.g. method='quantile' to the QuestHandler, MultiStairHandler now raises because method='quantile' is not valid for MultiStairHandler. So what we actually need now is an additional method kwarg that gets passed down to the invoked staircase handlers and is separate from the kwarg controlling MultiStairHandler behavior.

Previously, MultiStairHandler(stairType='quest', method='quantile', ...) would invoke the QuestHandlers correctly, but MultiStairHandler would always present them in sequential order.

psychopy/psychopy/data/staircase.py

Lines 1869 to 1877 in fc51f9e

	def _startNewPass(self):
	"""Create a new iteration of the running staircases for this pass.
	This is not normally needed by the user - it gets called at __init__
	and every time that next() runs out of trials for this pass.
	"""
	self.thisPassRemaining = copy.copy(self.runningStaircases)
	if self.method == 'random':
	np.random.shuffle(self.thisPassRemaining)

[hoechenberger]

Ideally, all kwargs that need to be passed down to the invoked staircase handlers would be passed as a separate dict, e.g.,

MultiStairHandler(stairType='quest', method='fullRandom',
                  stairParams=dict(method='quantile'))

[hoechenberger]

Ideally, all kwargs that need to be passed down to the invoked staircase handlers would be passed as a separate dict, e.g.,

MultiStairHandler(stairType='quest', method='fullRandom',
                  stairParams=dict(method='quantile'))

Never mind, we already do that via conditions. Turns it there apparently was a bug in one of the old QuestHandler tests, which is now hopefully resolved.

[codecov-io]

Codecov Report

Merging #2659 into master will decrease coverage by <.01%.
The diff coverage is 84.61%.

@@            Coverage Diff             @@
##           master    #2659      +/-   ##
==========================================
- Coverage   44.28%   44.28%   -0.01%     
==========================================
  Files         235      235              
  Lines       42196    42205       +9     
  Branches     7287     7289       +2     
==========================================
+ Hits        18687    18690       +3     
- Misses      21524    21534      +10     
+ Partials     1985     1981       -4

Impacted Files	Coverage Δ
psychopy/experiment/loops.py	59.2% <ø> (ø)	⬆️
psychopy/data/staircase.py	66.16% <84.61%> (+0.16%)	⬆️
psychopy/visual/window.py	64.57% <0%> (-0.59%)	⬇️
psychopy/app/coder/coder.py	28.26% <0%> (-0.18%)	⬇️
psychopy/visual/textbox/__init__.py	43.5% <0%> (ø)	⬆️
psychopy/app/builder/dialogs/dlgsConditions.py	7.51% <0%> (ø)	⬆️
...ychopy/localization/generateTranslationTemplate.py	0% <0%> (ø)	⬆️
psychopy/tools/wizard.py	74.17% <0%> (+0.65%)	⬆️
psychopy/visual/textbox/fontmanager.py	71.77% <0%> (+0.69%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc51f9e...5737984. Read the comment docs.

[coveralls]

Coverage increased (+0.002%) to 48.987% when pulling 5737984 on hoechenberger:multistair into fc51f9e on psychopy:master.