Standalone autograders, autodetecting file names, package, etc

[dbp]

For whatever reason, I've found it difficult to wrap my head around making autograders with this: I think it's the copy repo, edit files, etc, workflow that for whatever reason is confusing me.

Working backwards, I started from wanting a single file to be an autograder for an assignment. They live in the repo for the course, and so should be self-contained, depending on the gradescope code in a library (and depending on shared pre-built docker images seems as bad / worse, given how poorly they seem to be maintained as infrastructure).

This is easy --- if you move the lib-grade.rkt code into a package (which I tentatively named autogradescope, though haven't published, as I wanted to open this issue first):

#!/usr/bin/env racket
#lang racket
(require autogradescope)

(require rackunit) 
...
;; (define-var ...
;; (define-test-suite ...

The next thing that caused trouble for me was the hardcoded file names -- students would often submit slight variations (capitalized, etc), and while one could say this is a learning opportunity, it doesn't strike me as an important one. So, rather than (define-var foo from "file.rkt"), I changed it so that it finds whatever file in the submission directory has the right extension (which is configurable, but defaults to .rkt). If students are submitting multiple files, obviously this doesn't work (and perhaps that was the original motivation for the current design?).

(set-submission-extension! ".rkt") ; this is the default, so not actually needed.
(define-var my-function) ; these are found in the first file in the submission with the correct extension. 
(define-var other-function)    
...

The last thing that I changed was how testing of the autograders is done (i.e., on our own computers, before we send them to Gradescope). While there is certainly value in running on the same image (or, hopefully the same image) that runs on Gradescope, I suspect most of us rely on the fact that Racket 8.9 is Racket 8.9, across platforms (as, indeed, we need our testing to match what our students are doing), and so running natively should be "good enough", and can be a lot lighter weight: an environment variable can hardcode the path to the file where definitions are loaded from (and, when it is present, the results are printed to stdout):

$ SUBMISSION=./reference-soln.rkt ./run_autograder
{"score":"96.42857142857143","tests":[{"output":"Execution error in test named «blah»"}]}

Obviously, these are a bunch of changes, and some actively conflict with how this library currently works. At the same time, they aren't mutually exclusive (e.g., there can easily be two forms of define-var), and if you were interested in turning this into a library (to support my first goal: having the autograders be single files, not depending on prebuilt docker images), I could certainly do a more careful job merging. On the other hand, I'm also fine with having a fork (which is why I haven't called the library gradescope-racket). One thing though: could you stick a license on this code? :)

[jasonhemann]

@shriram @wilbowma : Any objections to an MIT license?

@dbp Just FYI: https://pkgs.racket-lang.org/package/khoury-gradescope, https://github.com/northeastern-khoury/gradescope-racket/.

[wilbowma]

No objection to MIT license.

FWIW, some of these design decisions don't work in my context. I'm using this package (or my fork of it) in a 4th year course, and students need to submit multiple files, so I can't use a system that searches for files with the right extension.

I'm also using it in a compilers course, and we rely on particular version of external binaries such as nasm and ld, and on the OS, binary format, etc, which are related to the underlying image, so I really need the ability to test to the image.

Maybe I'm an outlier and I can continue to maintain my fork (https://github.com/cpsc411/cpsc411-gradescope), but I'll be less and less able to contribute upstream.

[dbp]

@wilbowma The former isn't too big of a deal (as I said, having multiple define-var forms should be straightforward).

To the latter -- to some extent, the docker infrastructure always has to be able to be built (it's what Gradescope is going to do), so perhaps there is a way of separating out the parts that are relevant to that (mostly, the Makefile & conventions?) from code that could go on pkgs.racket-lang.org? Since everyone seems to have their own fork anyway, maybe there are just templates (of common patterns of use) & a library?

(With that said, I'm obviously perfectly happy to have a fork, but would prefer to avoid if possible, hence opening this in the first place).

[wilbowma]

To the latter -- to some extent, the docker infrastructure always has to be able to be built (it's what Gradescope is going to do), so perhaps there is a way of separating out the parts that are relevant to that (mostly, the Makefile & conventions?) from code that could go on pkgs.racket-lang.org?

That makes sense to me; something similar to raco pkg new could setup a new template directory?

[shriram]

I'm just fine with an MIT license.

We too have some more complicated uses for our upper-level PL course so I don't want to over-simplify it: I really don't want to fork and maintain that. The more @dbp can make a conservative extension, the better.

[dbp]

Okay -- I'll work on a minimal pull request, with the aim of avoiding forks.