New activity type: multiple choice

[pdawyndt]

The R courses now abuse "exercises" as a poor man's version of multiple choice questions.

This has opened the debate whether we should support a new activity type for multiple choice and to what extent that support should lead us (mockup based on this snippet)

This is a stub to collect good practices and ideas for multiple choice support.

[pdawyndt]

QTI seems to be the leading data exchange format for multiple choice questions. GIFT is an exchange format associated with Moodle. SCORM is much broader in scope.

[bmesuere]

It will be very important to strictly scope this before implementing.

• Which format will we use to store the question, answers and feedback? QTI is a standard but this is very verbose xml which is not user editable without a custom editor.
• Will we allow markdown/html for the questions/answers/feedback? Or restrict it to inline elements for answers?
• Will all questions have a single correct answer or will we allow multiple?
• Will we give feedback to the user after answering?
• Will we allow a user to answer a question multiple times until they get it right? And will we store all answers?
• If we make those last 2 things configurable (e.g. for tests) it probably will not be enough to configure this in question itself because a certain question might be used in multiple courses with different feedback requirements. Those settings should thus be associated with the use of a question inside a series (maybe a setting at series level?). How do we prevent that a student looks up the question in the general listing and hands in multiple times there to retrieve the answer?

I suggest we go for a minimal implementation that mimics the current use of multiple choice questions but with a nicer user interface. We don't want to give the impression that Dodona is suitable for multiple choice exams. It was not built for that and the options to shield questions/answers are inadequate. The multiple choice implementation is merely meant as a study aid. So,

• Only one correct answer per question
• Always give feedback
• Users always can hand in multiple times but we store each result

[pdawyndt]

Maybe explicitly provide “submit”, “try again” and “show answer” options. Features that may possibly be disabled in a later stage, together with the option to show if the answer is correct of not.

[bmesuere]

Some notes after a meeting with stakeholders today:
There are two seemingly similar but conceptually very different ways of looking at this:

1. multiple choice questions as most people know it: question, possible answers, correct answer and feedback.
2. dynamic questions where code is executed to check which answer is correct (or even to generate the question).
   The first is what we had in mind, the second would require an entirely different approach. We decided that the second option is out of scope for now and that such complex questions can be handled using a work around with the current judge. Additional helper functions could be added to the R judge to make such questions easier.

In addition, the timeline has shifted a bit. There is no need to have this implemented by september. It would, however, be nice to have a first prototype by the end of summer on which we can iterate to add more features such as randomly selecting 1 of multiple provided questions.

For a first minimal version, I would:

• Define the format used to describe the questions.
  • Ask for feedback + example from Jan from their current setup for generating questions for Ufora. We don't necessarily want to use the same format, but it would be nice to be able to convert/reuse existing questions.
  • In a future release, we probably will want to allow defining multiple question variants of which one is shown to the student. It would be good to anticipate for this in the format.
• Implement the new activity type in Dodona where the question + answers are rendered on the page. Each question has a single correct answer.
• Students can submit/answer multiple times, each attempt is stored (for now, no restrictions on this).
• Feedback is shown after answering.

To discuss for v1: what about i18n for the question and answers? We will probably want to support this in our format. How do we let this play nice with the multiple question variants.

/cc @lievenclement