Plausible distractors in MCQs

[maneesha]

In the section about formative assessment and MCQs, should we consider that the plausible distractors depend on the strategies that are taught? In our example, we assume that the child was taught to "carry" the number in the tens place. Not everyone is taught this way.

For example, I'm currently working with students who learn to round numbers up/down to make numbers that are easy to add in your head, and then add/subtract the difference.

So for this they would:

• round 27 --> 30 (+3)

Then add:

• 30+15 = 45

Then account for the difference:

• 45-3=42 (-3 that you added before)

So if this was the strategy taught, plausible distractors might actually be

• 48 (you added the rounding difference instead of subtracting it)
• 45 (you forgot to subtract the rounding difference)

I know we can't make this section complicated but I do want to account for differences in early math curricula across generations and geography. To consider the plausible distractors we offer in the curriculum, trainees have to think about arithmetic in a way they may not be used to.

[ceesu]

Hi @maneesha, do you mean that the MCQ example should be updated with 48 or 45 as additional options? Currently, the example only seems to say that "The correct answer is 42, but each of the other answers provides valuable insight." I think this statement seems reasonable for this example.

[tobyhodges]

I know we can't make this section complicated but I do want to account for differences in early math curricula across generations and geography. To consider the plausible distractors we offer in the curriculum, trainees have to think about arithmetic in a way they may not be used to.

Thanks for raising this @maneesha. This highlights a weakness of multiple choice questions and a difficulty with designing/identifying plausible distractors: as lesson designers we may not be able to create a plausible distractor to diagnose every common misconception/error a learner might make. We are limited by our personal experience, and the MCQ format makes it more difficult for us to notice when a common misconception is not covered by our MCQ.

E.g. if the learner thinks the answer here is 48 or 45, what will they do when faced with the four options in this MCQ? In the best case they will be prompted to make another attempt at the challenge and arrive at the right answer, or they will say "my answer is not included in this list." But they may not speak up and instead guess at the right answer (and receive no feedback about what was wrong with their original approach) or give up altogether.

This is a topic for us to consider over at https://github.com/carpentries/lesson-development-training/ but I think there is something relevant to Instructor Training here too: namely, that we should advise Instructors to make sure that they should try to ask (and ask helpers to ask) learners how they arrived at an incorrect answer - they may learn about an important common misconception that a lesson is not currently designed to diagnose and address.

Regarding this issue specifically: I am in favour of adding these two additional plausible distractors to the options in the example MCQ.

[tobyhodges]

Following up, and perhaps changing my mind, after writing carpentries/lesson-development-training#116:

I think including plausible distractors that account for different approaches to solving the same kind of problem is most important for pre-workshop and start-of-lesson assessments. It becomes less of an issue later in a lesson, when we might reasonably expect learners' misconceptions to be limited to the approach taught in the lesson. I.e. for this mental arithmetic exercise, the current set of plausible distractors are sufficient if the exercise is used at the end of (part of) a lesson that teaches the "carrying" approach to addition of two numbers.

[karenword]

Great points all -- how about adding a note like "this example would be appropriate for a class that had been taught to add by carrying numbers. If you were teaching a group to add by a different method, e.g. using rounded numbers, you would use different distractors to diagnose common errors in using that method.