Berkeley SAAS w/ edX
- Berkeley folks
- There is an AMI (id ami-df77a8b6) that includes latest version of the autograder.
- Other folks
- the autograder code is at github:saasbook/rag.
Launch an EC2 instance (micro is fine) with autograder AMI
hw6Grader1 has the latest version of rag, so move the content from it to the new instance.
- You might want to move the logs before you copy as they take up a lot of space.
NOTE: Somebody should make a new AMI with the updated connection code, and enabled to pull from the saasbook repos
There are two files which control the autograder:
conf.ymlincludes the following:
live: queue_uri: 'uri' autograders_yml: ./config/autograders.yml django_auth: username: 'username' password: 'password' user_auth: #may be unnecessary user_name: 'username' user_pass: 'password' halt: false # default: true, exit when all submission queues are empty sleep_duration: 30
live: the name that will be given on the command line when calling the autograder command
- The rest of the information should be filled in appropriately.
autograders_ymlgives the path from the
autograders.ymldescribes the actual autograder that will be running. An example is:
assign-6-queue: name: "BerkeleyX-cs169x-p3" type: HW5Grader due: 20121217120100 grace_period: 0 parts: assign-2-part-1: uri: ./hw6.yml type: MigrationGrader
name: the name of the queue that will be pulled from, this must match what edX is submitting assignments to
due: in YYYYMMDDHHMMSS format
parts: used to describe the parts of each assignment
uri: a path to the solutions from the rag base directory
type: the type of autograder that will be used to grade the assignment
grace: period is used to give late credit, this should be refactored further to allow specification for late penalties in this config file
while true ; do ./run_edx_client.rb live ; done
The grader can crash, this starts it up again.
./grade ../hw/ruby-calisthenics/skeletons/part1.rb ../hw/ruby-calisthenics/sanity_specs/part1_spec.rb ./grade ../hw/ruby-calisthenics/skeletons/part2.rb ../hw/ruby-calisthenics/sanity_specs/part2_spec.rb ./grade ../hw/ruby-calisthenics/skeletons/part3.rb ../hw/ruby-calisthenics/sanity_specs/part3_spec.rb ./grade ../hw/ruby-calisthenics/skeletons/part4.rb ../hw/ruby-calisthenics/sanity_specs/part4_spec.rb ./grade ../hw/ruby-calisthenics/skeletons/part5.rb ../hw/ruby-calisthenics/sanity_specs/part5_spec.rb ./grade ../hw/ruby-calisthenics/skeletons/part6.rb ../hw/ruby-calisthenics/sanity_specs/part6_spec.rb ./grade ../hw/ruby-calisthenics/skeletons/part7.rb ../hw/ruby-calisthenics/sanity_specs/part7_spec.rb
./grade ../hw-solutions/ruby-calisthenics/lib/part1.rb ../hw-solutions/ruby-calisthenics/spec/part1_spec.rb ./grade ../hw-solutions/ruby-calisthenics/lib/part2.rb ../hw-solutions/ruby-calisthenics/spec/part2_spec.rb ./grade ../hw-solutions/ruby-calisthenics/lib/part3.rb ../hw-solutions/ruby-calisthenics/spec/part3_spec.rb ./grade ../hw-solutions/ruby-calisthenics/lib/part4.rb ../hw-solutions/ruby-calisthenics/spec/part4_spec.rb ./grade ../hw-solutions/ruby-calisthenics/lib/part5.rb ../hw-solutions/ruby-calisthenics/spec/part5_spec.rb ./grade ../hw-solutions/ruby-calisthenics/lib/part6.rb ../hw-solutions/ruby-calisthenics/spec/part6_spec.rb ./grade ../hw-solutions/ruby-calisthenics/lib/part7.rb ../hw-solutions/ruby-calisthenics/spec/part7_spec.rb
There is also checking the skeletons against the full specs and the sanity specs against the full solutions.
At a high level HW3 and others that use
FeatureGrader work by running
student-submitted Cucumber scenarios against modified versions of a
The Following Diagram roughly describes the flow of the autograder :
Each step defined in the .yml file can have scenarios to run iff that step passes.
Example from hw3.yml:
- &step1-1 FEATURE: features/filter_movie_list.feature pass: true weight: 0.2 if_pass: - &step1-3 FEATURE: features/filter_movie_list.feature version: 2 weight: 0.075 desc: "results = [G, PG-13] movies" failures: - *restrict_pg_r - *all_x_selected
In this case if step1-1 passes, Step 1-3 will be run. If step1-1 fails then step1-3 will not run and the student will not receive points for it. It is important that the outer step be less restrictive than the inner step (If the outer one fails, there should be no way that the inner one could pass).
Step1-3 has two scenarios specified as failures; this indicates that when the cucumber features are run, both of those scenarios should fail. In other words, when the mutation for this step is added to the app, the student’s tests should detect the change and fail. (Example: If the test is to ensure that data is sorted, and the mutation is that the data is in reverse order, the student’s test should fail because the app is not behaving as expected)
In order to add a new step the following must be done:
Add an entry to the .yml file.
The new entry should be a hash with the following properties:
FEATURE, a relative path to the Cucumber feature that will be run for this step.
weight, the fraction of total points on this homework represented by this feature
version: This sets an environment variable that the mutation-test app can use to add any modifications desired to the app before the feature is run.
desc: A string describing the step, used when providing feedback
failures(list): scenarios that should fail in this step
if_pass(list): steps to run iff this step passes.
To define a new scenario add a new entry to the "scenarios" hash in the .yml file. It is a good idea to set an alias for the scenario so it can be referenced later inside of steps.
The entry should contain:
match: A regular expression that will identify the name of this scenario. (Used when parsing cucumber output to see if this scenario passed or failed)
desc: A description of the scenario. (Used to give feedback to the student)
When a feature is run, the environment variable
version will be set to
the value of the
version property for that feature. Use this as a
feature flag in the app (by checking
ENV["version"]) to trigger a
"bug", e.g. reversing sort order/not returning all data.