Container Autograder grading function

[keiferms3]

Figured I'd get this issue started as I've run into some questions regarding it while creating a mock function for the grading endpoint.

• How will we relate the filepaths in a submission's content to the autograders that need to be run on them? In other words, when iterating through the files included in the submission's content, how will we know which container autograder to run on each file?

• Am I correct to assume each file will have a single autograder run to produce all of its SubmissionProblemScores? If yes, in what format will we return the scores from the autograder's grading function to the grader service? How will we associate the returned results to each assignmentProblem? The non-container autograders can do this quite easily by finding an assignmentProblem with the same problemName as the grader's question, but how will we accomplish this with container autograders?

• Currently, all the grader service is doing is iterating through the files listed in the submission's content, finding the ContainerAutoGrader associated with each file, and calling the grading function with the submission filepath and the grader object as parameters. Actually obtaining the file is left up to the autograder's grading function and isn't directly handled by the grader service. The grading function is assumed to just return scores and feedback and creating the SubmissionProblemScores is left up to the grader service. Is this the way we want to structure this?

@SantarinX @jessehartloff

[jessehartloff]

We have a bit of design work to do here. Here are my current thoughts and we can discuss our exact approach moving forward.

• All files should be used for all ContainerGraders. That way each grader has the entire submission. The most common use case will be a student submitting a programming project that spans multiple files and each grader might check a different aspect of the program so they'll all want every file.
• [As a side note] It just hit me that we'll need a way to preserve the original filenames and directory structures if we are sticking with multiple file submissions. In our last conversation I said that files will be renamed to whatever the instructor specified in the assignment, but that only works if there's a single file for the submission.. Maybe we compress multi file submissions on the backend?.. Need further discussion
• For Autolab compatibility, we'll support the same way ContainerGraders give scores in Autolab. We capture stdout while the grader is running and expect the last line to be a json sting in the format '{"scores": {"q1":0, "q2":3.5}}'. Then the question is, who is responsible for parsing this string between the grading path and the ContainerGrader function. In either case, the grader will result in one giant string that has all the feedback with the scores on the last line and this has to be parsed at some point
• We'll have a few special cases to handle: 1) If a problem is not graded at all after all the graders finish, it doesn't get a score and will presumably be manually graded later, or regraded with a full grader. In either course, the score should be blank, not 0. In Autolab, these scores appear as dashes 2) Problems that are scored by multiple graders. This really shouldn't happen by design, but we should have some logic in place just in case. I think taking the max score is appropriate. I'm thinking that the grading path runs all graders, collects all the scores and runs this logic to combine them if there are duplicates, then add them to the DB
• We should have default ways of determining the submission score and the assignment score. Submission score as the sum of the problem scores seems fine. Assignment score has 2 main options, max or latest, and if we can be so bold.. I think max should be the default. We can also [eventually] add an option on the assignment where instructors can choose between the 2. And if someone doesn't like either option, they can use a WebHook and the API (Maybe a third option of "don't populate the assignment score" for people who want to use the API without risking students seeing a temp score)