Randomizing the order of responses in a pair that are shown to participants

[rmovva]

Hi again!

I'm working on a fork of Potato where I am showing participants a pair of responses on each screen, and they pick one out of the two. Each example will be annotated by several participants. To avoid order effects, I'd like the responses to be randomly shuffled for each participant. Is there an easy way to implement this?

This is an example of what my data looks like, this is my custom annotation template html. This is what the annotation screen looks like:

I'm not sure which part of the code I should edit to make this change so that each user receives their own random order. Do you think it would be possible?

Thanks,
Raj

[Jiaxin-Pei]

Hi @rmovva,

I think there is a simple way to make it work. Since it's a paired task, you could simply create two entries for each pair and then set labels_per_instance as N/2 where N is the original number of labels you need.

For example, let's say you have (Response A, Response B) and you want it to have 10 labels, what you need to do is simply create two lines in the data (Response A, Response B), (Response B, Response A) and set labels_per_instance as 5. In this case, half of the labels will be collected in the order of AB and another half in the order of BA.

The only drawback of this solution is, a single user might see the same pair in two different orders. However, I think this would be rare and it actually allows you to test whether users are sensitive to the order of the responses.

Another quick solution is: you can create a JS function in your custom annotation template. This function will randomly swap the position of the two responses on the front end. For example, randomly generate a number, if it's odd, swap the content, if not, keep it the same.

Let me know if this sounds good to you! I'm happy to chat more about this if you have more specific needs

[rmovva]

Thanks Jiaxin! I had this idea too. But isn't there a chance that one annotator would be assigned both orderings of the same example? I want to avoid this situation. Sorry if I am missing something obvious. (But yes, if there is an easy way to avoid the annotators from seeing the shuffled versions of the same example, I am curious to hear that too.)

[Jiaxin-Pei]

@rmovva yeah, you are right about that, please see my edited response for the second solution

[rmovva]

This is very useful, thanks! And yeah, good point that it could actually be a useful data quality metric check. I might try this approach.

Re: the second solution, would it be possible to provide a couple more pointers on how this implementation would work, e.g. with pseudocode? (This is not time-sensitive because I can use the first approach you suggested.) For example, I'm not sure about how I would switch the content in the cells but keep "Response A" and "Response B" headers the same. And then would there be an easy way to track which ones ended up getting swapped and which ones didn't? - I guess maybe the exact content which the user saw gets tracked in annotated_instances.jsonl, so I could use this?

[Jiaxin-Pei]

@rmovva I just pushed a commit to support randomizing the instances when it is defined with a dictionary, you can check the documentation here, please make sure you upgrade potato-annotation to 1.2.1.7

"list_as_text": {
  "randomization":"value" # whether randomizing the list or dictionary when creating displayed_text, 
                          # options: 
                          #     "value": only shuffle the values but keep the order of the keys
                          #     "key": shuffle the order of keys
},

the swapped content will be saved in the output dir

[rmovva]

Thanks so much again, Jiaxin - this is so kind of you! I just updated my fork with your commit, slightly modified to account for the fact that I only wanted a specific set of the fields to be randomized (not all the fields). This is working great. And then, I can figure out what the order shown to the participants was by using the displayed_text field in annotated_instances.jsonl.

Thanks so much again - you are doing so much work to make Potato a good experience for us to use, and I really appreciate it - I will recommend Potato to others who are working on user study research.