Autograder development
Python HTML CSS Shell
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

CS111 autograder and test cases

Autograding scripts for Wellesley's CS111 (Introductory CS) course. The program evaluates student Python code on a suite of test cases, and generates HTML outputs with error hints, as well as json logs. It supports grading for a variety of functions, including those with print statements and some support for graphics.

Currently in active development. The codebase is adapted from the Berkeley AI autograder.

How to create a new problem set skeleton

Clone this repo and pull for latest changes before starting work.

  1. Create a new directory for the problem set somewhere on your machine. Let's refer to it as /path/to/ps0x

  2. Copy,, casefile_creator.txt, and inspector/ to this new directory:

$ cp /path/to/ps0x $ cp /path/to/ps0x $ cp /path/to/ps0x $ cp casefile_creator.txt /path/to/ps0x $ cp -R inspector /path/to/ps0x ```

  1. Change to the the problem set directory

$ cd /path/to/ps0x/ ```

  1. Delete the sample inspector/test_cases and inspector/pickled contents (but leave the directories).

$ rm -r inspector/test_cases/* inspector/picked/* ```

  1. Edit casefile_creator.txt with your desired test cases, descriptions, and grade weights. For complex arguments or results, create a pickle file with a .pickle extension, drop it in the pickled/ directory, and put the filename in casefile_creator.txt. If you pickle arguments, the entire argument list should be pickled as a tuple.

    • Note that for functions with print side-effects, you must create an object of type ReturnPrint (defined in inspector/, pickle it, and set the "print" column in casefile_creator.txt to True.

    • For functions with randomness, set the seed to 0 before you generate your test cases.

  2. Run

    $ python

    This populates the inspector/test_cases directory with files readable by the otterInspect.

  3. You can now delete and casefile_creator.txt since the students don't need these.

  4. Edit inspector/urls.txt where each line has a task id (like "task1") and the URL to the problem set description of the task.

  5. Edit the first few lines of the main function in by specifying the filenames for the student code, project name, and due date.

  6. Place the solution files for the problem set, and all supporting files in your /path/to/ps0x directory. Test that running


    works as expected.

  7. Delete all *.pyc files and the files generated by your testing runs of otterInspect:

$ rm *.pyc $ rm your_result.html $ rm -r .logs ```

  1. Scrub the solutions to create skeleton files. Voila! The problem set is ready for release.