Q0004_AnSwEr values disappear from form, even with evidence that the student did type in answers

[Alex-Jordan]

I'm posting this, aware that maybe it has already been addressed in develop. I think @drgrice1 will know.

There have been reports in the forum of answers disappearing. I think only with quizzes, but of course it's more likely with quizzes that someone would bother with a report. The description is something like answers were put into answer blanks, and the user hits submit. But the answers are all removed and the student gets a zero for the quiz. The student swears they had correct answers there, but they are just gone.

This happened with a server I manage, and I took a closer look at the transaction log. Below I paste the parts of the log that matter. This is one student's information (anonymized). It's a four-question quiz. They took it one time, got the zero. Later the instructor reassigned it, they retook it, and got 4/4.

The time they got 0/4, there is data like:

"Q0004_AnSwEr0001","","MaThQuIlL_Q0004_AnSwEr0001","73"

For this question, the correct answer was 73. So the MaThQuIlL_Q0004_AnSwEr0001 form field had the student's submitted answer, but the regular Q0004_AnSwEr0001 field was empty. During the second attempt, we have:

"Q0004_AnSwEr0001","73","MaThQuIlL_Q0004_AnSwEr0001","73"

The 73 is in both places, and the student got credit.

I certainly don't know how to explain this. But it makes me wonder if MathQuill failed (in the first instance) to put the student's answer into the Q0004_AnSwEr0001 input of the field when the submission happened. Could that be possible? (And if so has this already been addressed in develop?)

Here are the lines from the log:

[Thu Feb 18 02:29:47 2021] 1	0_4-ReadingQuiz	USERNAME	DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_1.pg	1	-1	3519	0	1	["Q0001_AnSwEr0001","","MaThQuIlL_Q0001_AnSwEr0001","2","Q0001_AnSwEr0002","","MaThQuIlL_Q0001_AnSwEr0002","4","Q0001_AnSwEr0003","","MaThQuIlL_Q0001_AnSwEr0003","5","Q0001_AnSwEr0004","","MaThQuIlL_Q0001_AnSwEr0004","4"]	0	1
[Thu Feb 18 02:29:47 2021] 2	0_4-ReadingQuiz	USERNAME	DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_4.pg	1	-1	2280	0	1	["Q0002_AnSwEr0001","B","MaThQuIlL_Q0002_AnSwEr0001","","Q0002_AnSwEr0002","I","MaThQuIlL_Q0002_AnSwEr0002","","Q0002_AnSwEr0003","N","MaThQuIlL_Q0002_AnSwEr0003",""]	0	1
[Thu Feb 18 02:29:47 2021] 3	0_4-ReadingQuiz	USERNAME	DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_6.pg	1	-1	1682	0	1	["Q0003_AnSwEr0001","","MaThQuIlL_Q0003_AnSwEr0001","16","Q0003_AnSwEr0002","","MaThQuIlL_Q0003_AnSwEr0002","0","Q0003_AnSwEr0003","","MaThQuIlL_Q0003_AnSwEr0003","8","Q0003_AnSwEr0004","","MaThQuIlL_Q0003_AnSwEr0004","0"]	0	1
[Thu Feb 18 02:29:47 2021] 4	0_4-ReadingQuiz	USERNAME	DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_10.pg	1	-1	3901	0	1	["Q0004_AnSwEr0001","","MaThQuIlL_Q0004_AnSwEr0001","73","Q0004_AnSwEr0002","","MaThQuIlL_Q0004_AnSwEr0002","74","Q0004_AnSwEr0003","","MaThQuIlL_Q0004_AnSwEr0003","84","Q0004_AnSwEr0004","","MaThQuIlL_Q0004_AnSwEr0004","214"]	0	1


[Thu Feb 18 16:57:59 2021] 1	0_4-ReadingQuiz	USERNAME	DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_1.pg	1	-1	3519	1	1	["Q0001_AnSwEr0001","2","MaThQuIlL_Q0001_AnSwEr0001","2","Q0001_AnSwEr0002","4","MaThQuIlL_Q0001_AnSwEr0002","4","Q0001_AnSwEr0003","5","MaThQuIlL_Q0001_AnSwEr0003","5","Q0001_AnSwEr0004","4","MaThQuIlL_Q0001_AnSwEr0004","4"]	1	1
[Thu Feb 18 16:57:59 2021] 2	0_4-ReadingQuiz	USERNAME	DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_4.pg	1	-1	2280	1	1	["Q0002_AnSwEr0001","I","MaThQuIlL_Q0002_AnSwEr0001","","Q0002_AnSwEr0002","I","MaThQuIlL_Q0002_AnSwEr0002","","Q0002_AnSwEr0003","N","MaThQuIlL_Q0002_AnSwEr0003",""]	1	1
[Thu Feb 18 16:57:59 2021] 3	0_4-ReadingQuiz	USERNAME	DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_6.pg	1	-1	1682	1	1	["Q0003_AnSwEr0001","16","MaThQuIlL_Q0003_AnSwEr0001","16","Q0003_AnSwEr0002","0","MaThQuIlL_Q0003_AnSwEr0002","0","Q0003_AnSwEr0003","12","MaThQuIlL_Q0003_AnSwEr0003","12","Q0003_AnSwEr0004","0","MaThQuIlL_Q0003_AnSwEr0004","0"]	1	1
[Thu Feb 18 16:57:59 2021] 4	0_4-ReadingQuiz	USERNAME	DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_10.pg	1	-1	3901	1	1	["Q0004_AnSwEr0001","73","MaThQuIlL_Q0004_AnSwEr0001","73","Q0004_AnSwEr0002","74","MaThQuIlL_Q0004_AnSwEr0002","74","Q0004_AnSwEr0003","84","MaThQuIlL_Q0004_AnSwEr0003","84","Q0004_AnSwEr0004","214","MaThQuIlL_Q0004_AnSwEr0004","214"]	1	1

[Alex-Jordan]

I was wrong to say the instructor "reassigned" the quiz. Judging by the random seeds, both instances were the same version of the quiz. The instructor has this quiz set to permit unlimited graded submissions.

But the student clearly tried to submit mostly correct answers at 2:30 in the morning and got 0/4. Then submitted mostly the same answers (with a few changes) again at 5:00 in the evening at got 4/4.

[drgrice1]

I have not seen this occur, and I don't know why it would happen. There is nothing in develop that would address this issue. Clearly something occurred that prevented MathQuill from putting the students answer into the original input, but I don't know how that would happen.

[Alex-Jordan]

This will be frustrating to understand better. The reports of this happening have never indicated that they can reproduce it. I would like to see the console log from an instance when it happened, but good luck getting that from a student, well after the zero was scored.

I confess I have been tempted to write off the reports as user error on the part of the student. But if nothing else, now we can advise people to look in the transaction logs and there is a chance to recover lost answers.

[drgrice1]

It would be useful to see what is in the console, and certainly we would need to be able to reproduce this in order to fix it.

[drgrice1]

One thing that is odd here is that both of the inputs are set at the same time in the javascript. So it really doesn't make sense that the input with id MaThQuIlL_Q0001_AnSwEr0001 would be set and the input with id Q0001_AnSwEr0001 not set.

[Alex-Jordan]

One possibility is that it does successfully set them, but then something else clears them.

I found another clue. In the login log, I find this student timed out for inactivity a few seconds before they got the 0/4 score. Then then logged with a password one second before the submission. Then made the submission:

[Thu Feb 18 02:29:41 2021] LOGIN FAILED inactivity timeout user_id=USERNAME login_type=normal credential_source=params host=136.244.148.252 port=52347 UA=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15
[Thu Feb 18 02:29:46 2021] AUTH WWDB: password accepted user_id=USERNAME login_type=normal credential_source=params host=136.244.148.252 port=52355 UA=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15
[Thu Feb 18 02:29:46 2021] LOGIN OK user_id=USERNAME login_type=normal credential_source=params host=136.244.148.252 port=52355 UA=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15

(The 0/4 submission in the transaction log is at 02:29:47.)

That tells a story that they had answers in answer blanks. Then hit submit, but were rejected for an inactivity logout. Entered password to get back in. Hit submit again, and we see that one in the transaction log. But no real answers, just the MathQuill ones.

So maybe when the entered their password and the screen returned to the quiz, it populated some input forms (like MaThQuIlL_Q0001_AnSwEr answers) but not all (like Q0001_AnSwEr).

Or maybe when they enter the password, that automatically tries to complete what I described as the "first" submission, and it just got logged a second later. But something about the re-logging in dropped the Q0001_AnSwEr type input values.

This would mean it's not MathQuill to blame, but a design flaw with what happens when someone times out for inactivity during a quiz. (The instructor has no time limit on this quiz, so this is more likely to happen than with a timed quiz.)

[Alex-Jordan]

I will try to mock up this situation now to see if it is repeatable.

[Alex-Jordan]

No luck. I repeated the scenario such that I submitted answers after timing out, then had to enter my password to log back in, and then the answers were all accepted. There was no need to push submit a second time. And the logs do indicate one second in between when I logged back in and the answers were written to the transaction log. (But for me, they were all present.)

[dlglin]

Did you try with a failed login in between? On Feb 22, 2021, at 3:42 PM, Alex Jordan <notifications@github.com<mailto:notifications@github.com>> wrote: [△EXTERNAL] No luck. I repeated the scenario such that I submitted answers after timing out, then had to enter my password to log back in, and then the answers were all accepted. There was no need to push submit a second time. And the logs do indicate one second in between when I logged back in and the answers were written to the transaction log. (But for me, they were all present.) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#1208 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AANWEGVLYNS4D3RQOKZIQOTTALMV7ANCNFSM4YBDXXJA>.

[Alex-Jordan]

@dlglin I did not, but the student in the original case here did not have a failed login in between.

My test was on the PCC server, on a flavor of develop. The incident was on a 2.15 server. I'm about to try it on the same server, same quiz, same course config..(except I'll cut the inactivity time to like 30 seconds). I'll try it with a failed login too, just to see what can be learned. It would be good to know if the data persists through a failed login attempt.

There is the larger issue of holding on to quiz answers in local storage when something goes wrong. Some student may not even have passwords, and need to go in through the LMS.

[Alex-Jordan]

Still no luck. Same server, same configuration, same quiz settings. Timed out for inactivity and then submitted the quiz to grade. Had to enter password, so I did, and immediately am told I got 4/4.

Tried again, resetting the quiz. This time using an incorrect password once when I have to log back in. But then after using the correct one on the next try, I get in and am told I got 4/4.

So I'm back to having no ideas. Other than speculation that it has something to do with peculiarities of the student's web browser.