FR Processing of Evaluation Requests

GOAL: Each evaluation request is a transaction

Processing of evaluation requests feature deals with the processing of incoming evaluation requests. In the normal course of events, an evaluation request comes in to main server, goes through load balancer to the execution node and gets evaluated in the execution node. The evaluation results flow back from execution node to load balancer to main server to user front end.

Exception cases to the normal course of action are as follows.

1. User submits multiple evaluation requests
   When a user submits multiple evaluation requests, these requests get can be treated in one of two ways: give the pending evaluation status message of the previous evaluation request, or take the new evaluation request and push the same to job queue. Depending on context, both choices make sense. It is better to make the choice a configuration parameter for the course administrator.

2. AutolabJS fails to evaluate submitted code properly
   This may happen due to bugs in AutolabJS or clever code trying to sabotage AutolabJS evaluation workflow. From past experience, we know that most of these errors occur in execution nodes. We need a mechanism for clean recovery from such faults. Exception handling at right locations in the execution nodes makes for a clean solution to this problem.

3. AutolabJS takes too long to evaluate a submission request
   Sometimes an evaluation request takes too long to process. We have inherent timeout mechanism for each test case of an evaluation. We need a recovery mechanism to timeout evaluation requests and inform the user of failed evaluation. We can put the timeout mechanism either in load balancer or in main server. The load balancer is responsible for job scheduling. Thus authority and responsibility for cancellation of a job also rests with load balancer. Load balancer can put a timeout on pending evaluation requests, send a message with null comment array and null score array. The load balancer can specify Evaluation is taking too long. The request has been cancelled. as the log message.

4. Revaluation after update of solution by instructor
   Sometimes, course instructor updates solution during the course of the lab. Then, the student may need to reevaluate all her(his) submissions against the updated solutions. In such a case, bulk evaluation request makes enormous sense. In fact, for cases where instructor updates solutions, we need to make a UI choice available to the user to submit evaluation requests for all the commits in the user repository.

5. Bulk evaluation / revaluation
   After the lab closes, sometimes instructor may need to update the solution and trigger a reevaluation of all the student submissions. After the assignment is closed, the instructor should be able to click on reevaluation button on /admin section of the site to trigger bulk evaluation of all commits / latest commit of the submitted code. The instructor also needs a guarantee that the student codes have not been changed after deadline of an assignment. We need to preserve a snapshot of all the student repositories at the end of each assignment. For more details on a way to do this, see Time Capsule. All revaluations should happen on code base preserved in time capsule.
   A crucial choice to be made is the source of users list. When an instructor chooses to revaluate an assignment, (s)he needs to provide AutolabJS with a list of users to evaluate. The list can come from the instructor herself or from the AutolabJS DB. The instructor can provide a list of students to evaluate in a csv file or she can paste the list in a text box. The way piazza takes the list of users as input is an inspiration here. Evaluating against the list of users in the DB is easier to achieve. Evaluating for users in the DB also limits the availability of AutolabJS service to enrolled users. First implement the evaluation for list of enrolled users from DB and then gradually scale up to custom list from instructor.
   A crazy possibility is the BITS makeup case. Here student takes the assignment after its due, gets evaluated like a normal assignment. We will not cater to makeup as revaluation for now. For now, all makeups have to be fresh assignments.

6. Collect and evaluate
   An instructor may wish to collect submissions on Gitlab and evaluate after assignment is closed. In such a case, students do not interact with the website portion of mainserver during the course of the assignment. Actually, this case is a special case of Bulk evaluation / revaluation.

7. Complex build pipelines
   In the normal course of programming labs, one evaluation request makes sense. But in complex programming assignments where multiple branches of git repository need to be evaluated, submitting multiple evaluation requests for a repository makes enormous sense. In this case though, we are coming very close to the usecase of Gitlab CI-runners. We can take advantage of those components to service complex build pipelines.