Grading and regrading failing - copy to VM failing and timeouts #169
Comments
|
(Lead TA here) Here's the stack dump from the 500 I got when I tried to regrade all – |
|
Wow - the original problem reported is weirder than I expected: the first task on the assignment lists completely different autograder feedback than the remaining four tasks: different job number and and everything. In the case of the student below, who I renamed BOVIK, neither job 442 nor job 407 were listed in the autolab feedback from any other one of their handins, and all the other handins appeard to have consistency of job number between the various tasks, as I would expect. Regrading individually does seem to make this problem go away, but that's a scary bug! Autolab feedback for the first task (named "handout"), student BOVIK, handin 5 Autolab feedback for other tasks, student BOVIK, handin 5 |
|
Could the broken handins I made while poking at @126 have broken the "regrade all" script? I've now deleted those two broken handins. |
|
Not sure if it's related, but the timestamp on the latest feedback seems to be a time in the future (unless I'm reading it wrong). From Rob's comment above – Doesn't that timestamp indicate 9 a.m. and 10 a.m. today morning (which hasn't happened yet)? |
|
Never mind about my comment 71189933 above - the broken handins are not at fault, without them "regrade all" still fails. Sorry for conflating two separate bugs (the mysterious zeroes + inconsistent autolab feedback within a submission and the bulk regrade failure) on one issue, I didn't fully understand these were separate things this morning. |
|
Wait... I thought I saw the regrade all fail but now I'm not sure. It appears to have succeeded with this error message... |
|
Okay, it looks like I did manage to regrade all, so the command-line-handed-in submissions may have been causing Nivedita's failure. However, I'm still seeing the same failure to grade the first assignments: everything except for the first task works But when I regrade by hand the right score shows up. I don't know if it's a heisenbug - I don't know if the people that have 9/10 now are the same people that had 9/10 before - or what. |
|
Oh: @niveditc reports that there is now a different group of people with failures, so this appears to be some sort of nondeterminstic bug, great. |
|
This sounds like a bug that we spent 6 hours on Monday squashing, and that we were very certain had been fixed. @mihirpandya and @yrkumar should probably get involved in this. |
|
So I did regrade all and diffed the exported grades, and sure enough, 17 students had different grades from on regrade all to the next. |
|
Were the changes in multiple directions? (0 -> 1 and 1 -> 0)? |
|
Looked to be that way. |
|
But it is a little bit strange that only the first problem ever changes. Is it possible that there's some non-determinism in your autograder? |
|
Look at the example autograder feedback above: if you click on the other problems, you seem that my autograder did give people a point for the "handout" task. But if you click on the handout task it reports a completely different job for which autograder never even got to the point of running my python code. |
|
@niveditc the timestamp issue of the autodriver has to do with autolab/Tango#13 Looks like the bug we fixed on Monday is resurfacing. Might be a race... Digging deeper. |
|
Looks like a bunch of consecutive submissions or regrades is causing scp to fail erratically. As a result, the files necessary for a job to run are not copied over to the grading VMs, which is why the feedback file says copyIn failed. Still not sure if this is an Autolab issue or an scp issue. A temporary fix would be to retry scp-ing until the file is copied over. Gonna investigate more and if nothing works out by tonight, I'll just make the temporary fix and investigate further later. |
|
What's the status here? Can I regrade all my initial homework submissions and have confidence that all the grades will be recorded? |
|
This is an issue with Tango. So moving it over there. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Why is copy to VM failing with nondeterministic "unexpected token at...."? The error message makes me think it's maybe related to @132? I don't think that looks like anything I could have put in my scripts, though I could be wrong.
Anyway, my lead TA tried a bulk regrade, but apparently that is failing too.
I need to release a number-of-submissions-limited assignment in the next couple of days, so in addition to @121 I'm pretty nervous about the prevalence of these kinds of errors.
==== New TA, 11:30pm ====
(Referring to @ 230 on Piazza)
After I fixed this student's issue, I went to the scoreboard and randomly picked two people who got perfect scores except for the handout category and hit regrade, and their scores also went up to 10.
The error messages usually look like this:
I think there may be more students who need regrading. Is there a way to do a bulk regrade?
==== Lead TA, 5am ====
I just did a regrade all. Thanks for noticing this!
==== Lead TA, 6am ====
In response to Grant’s email, I tried to regrade all, and there’s something wrong with the autograder. I get a 500 (internal error) with this error message –
The text was updated successfully, but these errors were encountered: