A student's PA mark is always boosted by not responding

[charvey2718]

We have identified a serious flaw in the PA algorithm: A student's PA mark is always boosted by not submitting PA marks. That is, it is always in their interest not to respond. Fortunately, there is an easy fix in the PR that I am about to submit.

To understand the issue, consider the following examples:

1. In the first example, everyone responded, and the calculation is fine. Let's focus on student E, whose PA score is 0.427.

RAW	A	B	C	D	E	Total awarded		PA CALC	A	B	C	D	E
A		4	3	2	1	10		A		0.400	0.300	0.200	0.100
B	3		3	2	0	8		B	0.375		0.375	0.250	0.000
C	4	4		4	4	16		C	0.250	0.250		0.250	0.250
D	3	5	4		1	13		D	0.231	0.385	0.308		0.077
E	5	0	1	4		10		E	0.500	0.000	0.100	0.400
Total received	15	13	11	12	6			PA score	1.356	1.035	1.083	1.100	0.427

2. In the second example, now student E doesn't respond. According to the current algorithm, to deal with this, a "fudge factor" of 5/4 is introduced (since there are five in the team, and four responded). You can see that student E benefits from this because their PA score has gone up to 0.534. This is not isolated to this example, or due to increased sensitivity from having fewer students. This is mathematically provable (as I'll show in a moment). But you can understand it intuitively by considering it this way: If a student doesn't respond, they still receive all the same marks from their peers, but now their PA score will be scaled up by the fudge factor. In short, not responding always benefits a student.

RAW	A	B	C	D	E	Total awarded		PA CALC	A	B	C	D	E
A		4	3	2	1	10		A		0.500	0.375	0.250	0.125
B	3		3	2	0	8		B	0.469		0.469	0.313	0.000
C	4	4		4	4	16		C	0.313	0.313		0.313	0.313
D	3	5	4		1	13		D	0.288	0.481	0.385		0.096
E	0	0	0	0		0		E	0.000	0.000	0.000	0.000
Total received	10	13	10	8	6			PA score	1.070	1.293	1.228	0.875	0.534

The PA algorithm (excluding self scores) is:

$$\text{PA score for student } i=\sum_{j\neq i} \left( \frac{ {}_{j}m_i }{\max{\left( \sum_k {}_j m_k, 1 \right)}} \right) \frac{N}{R}$$

where ${}_{j}m_i$ is the mark awared by student $j$ to student $i$, $N$ is the size of the group, $R$ is the number of respondents, and $k$ is a dummy summation variable. If one student (student $i$) doesn't respond, then they receive all the same marks from the other team members, but now multiplied by $N/(N-1)$, which always increases their mark!

Another issue with this approach is that the $\max()$ function in the denominator doesn't correctly fix the divide-by-zero error caused by $\sum_k {}_j m_k = 0$ if student $j$ doesn't respond, which also invalidates the use of the fudge factor... you can't sum over an undefined object, and you can't just ignore it either!

3. Fortunately, there is an easy fix, which makes use of the facts that:

• marks awarded to others do not influence the grader's own PA result (if not self scoring),
• if you award the same mark to everyone, then you have no effect on anyone's PA result.

The fix is therefore to insert a mark of one against all other team members on behalf of someone who doesn't repsond. (Actually, it could be a mark of two, three, or any number, as long as it's the same for everyone).

Doing this doesn't affect anyone's PA score: giving the same mark to everyone treats everyone equally whatever that mark is. And included in that, the individual's own mark is not affected either.

For the example above, we now replace the marks awarded by student E with 1s. The fudge factor now equals one, since there are five team members and five respondents.

RAW	A	B	C	D	E	Total awarded		PA CALC	A	B	C	D	E
A		4	3	2	1	10		A		0.400	0.300	0.200	0.100
B	3		3	2	0	8		B	0.375		0.375	0.250	0.000
C	4	4		4	4	16		C	0.250	0.250		0.250	0.250
D	3	5	4		1	13		D	0.231	0.385	0.308		0.077
E	1	1	1	1		4		E	0.250	0.250	0.250	0.250
Total received	10	13	10	8	6			PA score	1.106	1.285	1.233	0.950	0.427

Now student E scores 0.427 again, i.e. the score they would have got if they had responded (their score isn't affected by the marks they give).

Fortunately, this easy is very easy to fix with only a couple of modifications to calculator.php.