Added ranking by elimination test #110
Conversation
|
Wow. This is incredible useful. I will look at the paper next week and test the code. |
There was a problem hiding this comment.
looks good so far but i have some question for the method.
- How does this compare to Paired comparison without a reference #88 ?
- Is it part of the test design that a condition has to be removed instead of reordering or actually ranking them?
And it would be nice if you can share the paper with me as I don't have access to AES.
There was a problem hiding this comment.
Testing the code, works nicely. But for some applications it would still be nice to be able to have a reference to be played (which could be optional of course). This has applications e.g. for the waveform renderering as pointed out in the comment on the review.
@kishorkl how does literature comment on references in ranking based tests?
lib/webmushra/pages/RankingPage.js
Outdated
| this.macic = new MushraAudioControlInputController(this.mushraAudioControl, this.pageConfig.enableLooping); | ||
| this.macic.bind(); | ||
|
|
||
| this.waveformVisualizer = new WaveformVisualizer(this.pageManager.getPageVariableName(this) + ".waveformVisualizer", tdRight, this.conditions[0], this.pageConfig.showWaveform, this.pageConfig.enableLooping, this.mushraAudioControl); |
There was a problem hiding this comment.
isn't it an arbitrary decision to select this.conditions[0] as the condition that is used to render the waveform? I feel like overall it would be better to use the reference for this and therefore give experiments control over which condition or even the reference is used?
There was a problem hiding this comment.
@kishorkl can you comment on this one please?
There was a problem hiding this comment.
@kishorkl do you have time to finish this PR, I would like to issue a new version soon
There was a problem hiding this comment.
@faroit you are right that using this.conditions[0] to plot the waveforms is arbitrary. I would need to reread the paper to check what they suggest in terms of using the reference. I will get back on this in two weeks.
There was a problem hiding this comment.
The original paper basically suggests ranking by elimination as an alternative to a paired comparison test - which compares two models, without one marked as a reference or groundtruth (So AB test and not an ABX test). They don't compare to MUSHRA like tests. So there is no mention of the "reference" condition. Since we are looking at an optional feature, I will implement a "reference", which would be used for the waveforms if present (and also with a new optional button). I am still unsure what could be used for the waveforms when the "reference" is absent.
There was a problem hiding this comment.
hi, do you have some update on this one?
There was a problem hiding this comment.
@kishorkl do you have time to finish this?
This is almost entirely different from the mentioned commit. In the current submission one can rank multiple items, whereas in the referred commit, preference between "only" two items is tested. The current submission can find a preference between two items as well.
The paper is available at https://www.mathpsy.uni-tuebingen.de/wickelmaier/pubs/WickelmaierUmbach2009AES.pdf |
Bumps [qs](https://github.com/ljharb/qs) from 6.5.2 to 6.5.3. - [Release notes](https://github.com/ljharb/qs/releases) - [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md) - [Commits](ljharb/qs@v6.5.2...v6.5.3) --- updated-dependencies: - dependency-name: qs dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [braces](https://github.com/micromatch/braces) from 3.0.2 to 3.0.3. - [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md) - [Commits](micromatch/braces@3.0.2...3.0.3) --- updated-dependencies: - dependency-name: braces dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
@kishorkl I would love to merge this, so maybe we can shortcut this and decide on:
This should be fairly easy to do, right? |
Yes. I got occupied with some other tasks, hence could not spend much time, sorry. I will try to implement this during this week. |
…esults for ranking.
…esults for ranking.
|
@faroit Now the changes we discussed should be done. The ranking test does not have the ability to show a waveform and does not have reference. I also did a bug fix which was found in saving the results of the ranking test. |
Added a ranking by elimination test. This is a multi-stimulus test without the need for a reference. The test is similar to paired comparison but much faster to perform when there are many conditions. This test has been compared to paired comparison test in https://www.aes.org/e-lib/browse.cfm?elib=14979