Otter-grader is a new, open-source, local grader from the Division of Data Science, External Pedagogy Infrastructure at UC Berkeley. It is designed to be a scalable grader that utilizes temporal docker containers in order to remove the traditional overhead requirement of a live server.
Otter-grader can be installed using pip:
pip install otter-grader
Otter also requires you to have its Docker image installed, which is where it executes notebooks. The docker image can be installed in two ways:
Pull from DockerHub
To pull the image from DockerHub, run
docker pull ucbdsinfra/otter-grader.
Download the Dockerfile from GitHub
To install from the GitHub repo, follow the steps below:
- Clone the GitHub repo
- Build the Docker image with this command:
docker build . -t YOUR_DESIRED_IMAGE_NAME
Note: With this setup, you will need to pass in a custom docker image name when using the CLI (see below).
Notebook Design: PDF Generation
Otter-grader uses nb2pdf to generate manual submissions as PDFs. When generating the PDF, the following cells will be included by default:
- Markdown cells
- Code cells with images in the output
- All cells tagged with
If a cell falls within one of the 3 categories listed above but you do not want to include it in the exported PDF, tag the cell with
ignore. This cell will then not be added to the manual graded PDF.
You can use https://oktests.herokuapp.com/ to generate your tests in the ok format. Place all generated test files in the tests directory.
Instantiating the Autograder
Otter has functionality to allow it to be used in Jupyter Notebooks to allow students to check their progress against tests distributed with the notebook. This functionality is encapsulated in the
otter.Notebook class which allows you to run tests as you go through the notebook. When this class is initialized, it automatically assumes that the tests are located in the
tests subdirectory of the directory that contains the notebook, e.g.
lab01 | - lab01.ipynb | tests | - q1.py | - q2.py ...
To initialize the class, simply import it and create an instance:
import otter grader = otter.Notebook()
To use a custom directory of tests, pass it as the argument to the
import otter grader = otter.Notebook("/path/to/your/tests")
Checking Questions and Exporting
To run a test case, use the
Notebook.check function, which takes the test name as its argument. For example, if I wanted to run the test located in
tests/q1_1.py, I would call
Note the lack of the
Notebook.check automatically adds this when finding the test case.
Although it is possible to export manual submission PDFs from the command line usage of otter, the
Notebook API includes a method that exports the notebook for each student so that they can submit these themselves. This functionality is included because generating all PDFs at once requires a significant amount of time. To have students export their own notebooks, add the cell below to the notebook.
For example, if I was working in a notebook called
lab01.ipynb, my call to
Notebook.export would be
This will create a file called
lab01.pdf in the notebook's directory which students can submit. For more information on how nb2pdf filters cells, see the Notebook Design section.
Command Line Usage
The help entry for the
otter command is given below.
usage: otter [-h] [-g] [-c] [-j JSON] [-y YAML] [-n NOTEBOOKS-PATH] [-t TESTS-PATH] [-o OUTPUT-PATH] [-v] [-r REQUIREMENTS] [--containers NUM-CONTAINERS] [--pdf] [--image IMAGE] optional arguments: -h, --help show this help message and exit -g, --gradescope Flag for Gradescope export -c, --canvas flag for Canvas export -j JSON, --json JSON Flag for path to JSON metadata -y YAML, --yaml YAML Flag for path to YAML metadata -n NOTEBOOKS-PATH, --notebooks-path NOTEBOOKS-PATH Path to directory of notebooks -t TESTS-PATH, --tests-path TESTS-PATH Path to directory of tests -o OUTPUT-PATH, --output-path OUTPUT-PATH Path to which to write output -v, --verbose Flag for verbose output -r REQUIREMENTS, --requirements REQUIREMENTS Flag for Python requirements file path --containers NUM-CONTAINERS Specify number of containers to run in parallel --pdf Create PDFs as manual-graded submissions --image IMAGE Custom docker image to run on
Using otter requires a metadata file, which can be set for Gradescope (
-g) or Canvas (
-c) exports, JSON format (
-j), or YAML format (
-y). The JSON and YAML flags require an argument corresponding to the name of the metadata file relative to the
NOTEBOOKS-PATH argument. Note that you can only specify ONE (1) of the metadata flags. The defaults for certain flags are given below.
Otter executes the notebooks in parallel Docker containers. Depending on the number and size of the notebooks you are grading, you may wish to change the default number of containers (4) so that your grading can be executed quicker. To do this, pass the number of containers you want created to the
An example command would be
otter -g -n ./notebooks -t ./tests -o ./outputs --containers 6 -v
This command tells otter that we have a Gradescope export located in the
./notebooks directory with tests in the
./tests directory and that we want output in the
./outputs directory. It also tells otter to spin up 6 (rather than 4) docker containers and to include verbose output with the
The output of the command line tool is a CSV file which contains one row for each student and lists the student's identifier, the filename of their submission, their score breakdown by test case, and their final grade. This CSV file is output as
OUTPUT-PATH is the value passed to the
-o flag (defaults to
The Docker image for the grader comes with the following Python packages installed along with their dependencies:
If you require packages other than these, place them in a
requirements.txt file and pass the path to this file to the
-r flag, e.g.
otter -j meta.json -r requirements.txt
Notebook API includes a method to generate manual-graded PDF submissions, but the command line interface also provides this functionality. To generate PDFs of all notebooks run through the grader, pass the
otter -j meta.json --pdf
This will create a subdirectory
manual_submissions/ of the current working directory and put the PDF exports of each notebook into this directory, leaving the filename unchanged (with the exception of the extension); e.g.,
lab01-123456.ipynb would be
Given the time and execution cost of generating all of the PDFs at once, if you are distributing the in-notebook checker from otter, it is recommended that you have students generate and upload their own PDF submissions.
Custom Docker Images
It is possible to use a custom Docker image to execute the notebooks, but it must include the Python packages listed above, along with pandoc and xetex (if you intend to generate PDFs). To use a custom image, pass the image name to the
--image flag when calling otter. For example,
otter -y meta.yml --image my_docker_image
Otter is compatible with the Gradescope autograder, and has a command line tool to generate the zipfile that is used to configure the Docker container that Gradescope's autograder usage. The base command for this utility is
otter gen, and its help entry is given below.
usage: otter gen [-h] [-t [TESTS-PATH]] [-o [OUTPUT-PATH]] [-r [REQUIREMENTS]] [files [files ...]] Generates zipfile to configure Gradescope autograder positional arguments: files Other support files needed for grading (e.g. .py files, data files) optional arguments: -h, --help show this help message and exit -t [TESTS-PATH], --tests-path [TESTS-PATH] Path to test files -o [OUTPUT-PATH], --output-path [OUTPUT-PATH] Path to which to write zipfile -r [REQUIREMENTS], --requirements [REQUIREMENTS] Path to requirements.txt file
otter gen command creates a zipfile at
OUTPUT-PATH/autograder.zip which contains the necessary files for the Gradescope autograder:
run_autograder: the script that executes otter
setup.sh: a file that instructs Ubuntu on how to install dependencies
requirements.txt: Python's list of necessary dependencies
tests: the folder of test cases
files: a folder containing any files needed for the notebooks to execute (e.g. data files)
The requirements file create automatically includes the otter dependencies (see Installing Requirements), but you can optionally include your own, other ones by passing a filepath to the
-r flag. Any files included that are not passed to a flag are automatically placed into the
files folder in the zipfile and will be copied into the notebook directory in the Gradescope container.
An example command would be
otter gen -t ./hidden-tests -r requirements.txt my_data.csv my_other_data.json utils.py
otter gen to copy the tests from
./hidden-tests, add the requirements from
./requirements.txt, and to include the files
utils.py. All of these files will be made available to the notebook when it is executed.
Relative Imports on Gradescope
Because of the way that the Gradescope autograder is set up, our script must change relative import syntax in order to ensure that the necessary files are loaded. Because we cannot differentiate between package imports and relative imports,
otter gen automatically assumes that if you are importing a .py file that it is called
utils.py. The regex used to change the import syntax will only change the syntax if you're using the import statment
from utils import *
For this reason, if you have any functions you need in a file that is going to be imported, make sure that it is called