maketest

1. Automated testing for command-line projects in GitHub Classroom
2. maketest can clone repos, build them, run with predetermined input and output, and score the results vs. your rubric
3. To get started, clone maketest onto your machine and cd into it

Requirements

1. Student projects must be callable from the command line, and write results to stdout or another file you specify (see altactual below)
2. Student projects must have a Makefile to build them
3. The Makefile must generate an executable with the same name as the PROJECT (see below)
4. If written in C, your projects must take input via argc and argv, not via stdin
5. You must use GNU make 4.2 or later. This is current on Raspberry Pi OS, but macOS installs make 3.6 by default. If you're running this on macOS, you might need to brew install make but I haven't tested maketest on macOS

Usage

1. Remember to run the maketest commands from the maketest/ directory
2. You may want to use the make flags -s (silent) and -k (keep going past errors). You can combine the flags, e.g. make -ks
3. maketest expects to find make variables for ORG and PROJECT as they are shown in GitHub Classroom. In this README I use examples from the Spring 2020 class but you should substitute the ORG and PROJECT values for your class. You can specify these variables in one of three places, based on your preference
   1. On the command line

      make -ks test ORG=cs315-20s PROJECT=project02

   2. In the Makefile by editing the variables manually

      make -ks test

   3. In your shell environment (or .bashrc). You can use the -e flag to tell make to get its variables from the environment

       export ORG=cs315-20s PROJECT=project02 STUDENTS="phpeterson-usf gdbenson"
       make -eks test

Usage for Students

1. In addition to the ORG and PROJECT variables, students must define a DIR variable which contains the local filesystem path to your project
2. To test your local repo (without cloning)

   make -ks PROJECT=project04 DIR=../project04-phpeterson-usf

3. You should see output which looks like this

    build: ../project04-phpeterson-usf/
      run: ../project04-phpeterson-usf/ PASS for: -f quadratic -a 2,4,8,16
      run: ../project04-phpeterson-usf/ PASS for: -f fib_iter -a 20
      run: ../project04-phpeterson-usf/ FAIL for: -f find_max -a 2,-4,28,16
    score: ../project04-phpeterson-usf/ 10

4. To test the project with a new/clean repo (as the instructor will), you can run the clone and test targets

   make -ks clone test PROJECT=project04 ORG=cs315-20s STUDENTS=phpeterson-usf

Usage for Instructors

1. In addition to the ORG and PROJECT variables, instructors must define STUDENTS which contains a list of students' GitHub IDs. You may wish to do that in the Makefile or your environment, since the long list is static throughout the school term.
2. To clone all the student repos, use the clone target (assuming your are logged in to GitHub Classroom as a teacher for the Organization)

   make -ks clone ORG=cs315-20s PROJECT=project02 STUDENTS="phpeterson-usf gdbenson"

3. To build, run, test, and score the repos, use the test target (pro tip: test is the default target so you can omit test if you like)

   make -ks test ORG=cs315-20s PROJECT=project02 STUDENTS="phpeterson-usf gdbenson"

4. To pull new changes since you cloned the repos, use the pull target

   make -ks pull ORG=cs315-20s PROJECT=project02 STUDENTS="phpeterson-usf gdbenson"

5. To checkout changes at a timestamp, use the checkout target with a DATE

   make -eks checkout DATE="'main@{2020-08-25 00:00:00}'"

Testing and Scoring

For Students

1. When you run the test cases for your project02 solution

   make -ks test PROJECT=project02 DIR=../project02-phpeterson-usf

   maketest will generate a .actual file for each test case and one .score file in your directory.
2. To get credit for passing a test case, your output must match the expected output. Case and whitespace are not significant (it's diff -b -i)
3. The overall project rubric contains some portion for automated testing. The automated testing portion of the rubric is the sum of the scores in the .score file
4. Since a clean repo does not contain build artifacts (like your .o files), you should not commit .actual or .score files to your repo. You may wish to remove them before committing

   rm *.actual *.score

   Alternatively, maketest can remove them using the clean target

   make -ks clean PROJECT=project02 DIR=../project02-phpeterson-usf

For Instructors

1. In maketest/tests/ you should create a directory for each PROJECT
2. In each PROJECT directory, you should create test cases. A test case is defined by three text files. For test case foo, you should create:
   1. foo.input containing the command line arguments (not including the executable name) you wish to test. The format of the arguments is up to you, but students must parse the format you specify. I'm using getopt style arguments

      -f quadratic -a 1,2,3,4

   2. foo.expected containing the output you expect for foo.input
   3. foo.rubric containing the number of points earned when the student's foo.actual output matches (via diff -i) your foo.expected
3. If your project writes output to a different file other than stdout, you can specify that file in the contents of foo.altactual

How does it work?

1. maketest is itself a Makefile and everything it does is a list of targets
2. The approach is to build lists of unique target names using a unique suffix for each phase (clone, pull, clean, build, run, score)
3. There is a target for each test case for each phase for each student, so the target lists are pretty long
4. However, since make loops over the target lists automatically, the recipes for each phase can be relatively simple
5. There is a log file in ./$(PROJECT).log with all the gory details. Can be useful for finding bugs, either mine or the students!