Mean of a question across instances

[arthurfc12]

In the instructorAssessmentQuestions tab, a method showing mean across all instances where the question is used will help professors and instructors have a direct comparison between the questions performance in the current assessment and the questions performance in general, making it an easy comparison for professors assessment of the class' academic situation. See image below:

The method will be implemented in the Mean across all instances column

[jonatanschroeder]

I believe that will make the page too crowded, and the calculation required to perform that would impact the loading time of the page. This is usually the landing page for instructor view of an assessment, so the page should be lean and fast. If you feel this is a valid approach, I think a different page should be used for it.

[arthurfc12]

Would the question statistics table or the detailed question statistics table in the question_statistics page be viable?

[mwest1066]

The original intention for this column was to show the "difficulty" of each question, to help instructors balance difficulties within pools, arrange the assessment so that questions start easy and then work up to hard, etc. I still this that this is something we need and that this is the right page for it.

However, I agree that we don't want to be calculating this on the fly. Ideally we'd store this in a question stats table and update this as needed in the background (we should be doing the same for the instance question stats).

Exactly what this column should show is also up for debate. Ideally I think we want to show (at least) two values per question:

1. A pure measure of one-shot difficulty (e.g., predicted probability correct on first attempt on an exam)
2. The predicted points that will be awarded for this question, taking into account the points array. This would also factor in the expected success rate on second (and further) attempts.

The overall goal would be to show a predicted mean score (or maybe distribution of scores) for an exam, to help guide instructors when they are assembling questions into an assessment.

Of course there are lots of unanswered questions above about exactly how to do this. Other factors such as shared questions make this even more complicated, but also more useful. I'm eager to try and work some of these things out.

But as a first step, I think just getting some overall difficulty measure like (1) above would be great.

[arthurfc12]

A simple formula for difficulty estimation in multiple choice questions is P=R/T, where:

P = difficulty
R = correct responses
T = total responses

Further iterations taking into account weighted questions could be seen as P = ∑Res/T*100, where:

P = difficulty
∑Res = sum of all grades in a question
T*100 = total responses * 100 percent (assuming 100% as the weight)

[pmahfuz]

After analyzing the detailed question statistics provided by PrairieLearn we discussed a potential enhancement that could aid instructors in quickly understanding the difficulty level of questions. This feature would automatically categorize questions into "easy," "medium," or "hard" based on their median scores and the percentage of perfect scores achieved by students.

Here’s a brief explanation of the proposed system:

1. Easy Questions:

   • Median score is above a high threshold (e.g., 75% of the maximum possible score).
   • Percentage of perfect scores is above a certain level (e.g., more than 60% of students achieve a perfect score).

2. Medium Questions:

   • Median score is moderate (e.g., between 40% and 75% of the maximum score).
   • Percentage of perfect scores is moderate (e.g., between 30% and 60% of students achieve a perfect score).

3. Hard Questions:

   • Median score is below a lower threshold (e.g., less than 40% of the maximum score).
   • Percentage of perfect scores is low (e.g., less than 30% of students achieve a perfect score).

This feature could be implemented as an additional analytic tool within the instructor dashboard, providing at-a-glance insights that can inform instructional adjustments and future question design. It would also be valuable for identifying outliers and ensuring that assessments are well-calibrated to measure a range of abilities effectively.

Our first idea would be to implement this feature as an extra column in the question stats page, just as we did with the median.

We would like to know what you think of this iteration and any suggestions would be more than welcome.

[pmahfuz]

A few corrections to this previous suggestion:

Categorize the thresholds in order to not have a problem with multiple dimensions, here is a first draft of what they could be:

• Easy Questions:
  Median score is above a high threshold, OR the percentage of perfect scores is very high.
• Medium Questions:
  Median score is moderate, AND the percentage of perfect scores is moderate.
  This can be considered the "default" category if neither "easy" nor "hard" criteria are met.
• Hard Questions:
  Median score is below a low threshold, OR the percentage of perfect scores is very low.

[mwest1066]

After discussion with the core dev group we feel that this issue is too complex for implementation at this point. Instead we recommend focusing on #9699 as originally discussed.