forked from autolab/Autolab
-
Notifications
You must be signed in to change notification settings - Fork 0
/
howto_autoresults.txt
executable file
·49 lines (40 loc) · 1.65 KB
/
howto_autoresults.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
=========================
HowTo: Autoresult Strings
=========================
Autoresult strings are a short easily parseable output of an autograder when
run against a student's submission. Autoresult strings (AR) can be delivered
to Autolab in two ways:
1) Remotely: via an HTTP POST request to unofficial.fish.ics.cs.cmu.edu
2) Locally: via a result from an official autograder.
AR strings represent all the possible information that an autograder can
return about a student's submission, including performance and correctness
metrics. AR strings can be parsed into scores for the submission as well as
parsed into data for a scoreboard ranking for that student. Assessments using
AR strings should define the following function
# parseAutoresult(autoresultString,isOfficial)
# @param autoresultString The autoresult string to parse
# @param isOfficial true if the autoresult string is to be considered official
#
# @return
def parseAutoresult(autoresultString,isOfficial)
Add autoresult as a field of submissions.
createScoreboardEntry(scores,autoresult)
returns array of column=>value
called per student
scoreboardHeader()
parseAutoresult(autoresultString,isOfficial)
returns map of problemName=>score
submissions.sub_score(scores)
returns a single value representing the sum of the scores
submissions.total_score
sub_score + tweak + late_penalty
driver submits autoresult string via
1) unofficialSubmit.rb on port 80
->parseAutoresult(AR,false)
->store as submission version 0
->Append to log.txt for assessment
2)autogradeDone.rb
->AR = the last line of the results
->parseAutoresult(AR,true)
->store as submission version n+1
->Append to log.txt for assessment