This is the code for Experiment 3 (violations of the triangle inequality in relational similarity judgments, or in other words, violations of transitivity in analogy quality judgments) in "Evaluating vector-space models of analogy" (Chen, Peterson, & Griffiths, to appear in the CogSci 2017 Proceedings). You can download the paper here. The code uses psiTurk.
Here is an example trial:
screenshots folder also contains a screenshot of the instructions page.
As described in the paper, we created stimuli for which people's analogy quality judgments were expected to violate the transitivity property. In other words, we created triads, A:B, C:D, and E:F, for which A:B::C:D holds and C:D::E:F holds, but A:B::E:F does not hold. We obtained ratings of analogy quality for all three analogies from each triad. We created twelve such triads, shown in
get_groups_stimuli.py. This script creates 12 subsets of the analogy stimuli (each participant is randomly assigned to one of the subsets), subject to the following constraints:
- Each subset contains exactly one analogy from each triad.
- Each subsets contains the same number (4) of each type of analogy (A:B::C:D, C:D::E:F, or A:B::E:F).
- Across the subsets, each of the 36 analogies appears the same overall number of times.
- Each analogy does not always appear with the same other analogies.
The stimuli subsets are saved in the file