This is an x86-64 simulator designed to autograde C code safely and less bafflingly. Students should focus on learning C, not making sense of autograder output.
- Unicorn for x86-64 simulation
- elfutils for libelf, used to load the student ELF into the simulator, and libdw, used to parse DWARF debug symbols in the student ELF (for backtraces)
To try out the sample assignment, first build the dependencies and the astro library with the following commands:
$ ./update-deps.sh $ make
libastro.so as well as dependency libraries, so
cp -L the whole directory wherever you need it. To try an
example grader which links with astro, run:
$ cd example $ make $ ./tester
For reference, the Old Way of C autograding works as follows:
- Autograder starts up on baremetal
- Autograder jumps into student code
- Student code jumps back into autograder code
- Autograder asserts that student code worked properly
- For next test, goto 2
(For checking for memory leaks, repeat the whole process inside valgrind.)
Problem: Student Can Jump Wherever
When the autograder and student code share an address space, to pass tests, a student can jump back into the grader past any assertions: https://austinjadams.com/blog/acing-a-c-homework/. With astro, only the student code runs in the simulator.
Problem: Student Can Trash Autograder Memory
Imagine you're a freshman and writing C code for the first time. Just to
be safe, you
free() the same pointer three consecutive times. When you
run the autograder against your code, you see only this message:
../../src/check_pack.c:121: Bad message type arg 1937074548
Student code should not be able to corrupt autograder memory; the results are too confusing. In astro, student code has no access to autograder memory.
Problem: Student Can Run Arbitrary Syscalls
This grants too much power to break the autograder and possibly connect to the network to upload the autograder source or worse. In astro, there is no support for syscalls.
Problem: Simulating Memory-Mapped Hardware Features Requires Dark Magic
Whereas currently autograding access to memory-mapped registers requires
(https://austinjadams.com/blog/autograding-gba-dma/), astro could
simply monitor the special addresses. When a student writes to them,
astro could log the access and then simulate the hardware before
restarting the simulation.
Problem: Checking for Memory Leaks Is Fundamentally Broken
The Old Way for autograding data structures works as follows:
valgrind, start the autograder
- Allocate some memory (maybe, depends on the test)
- Jump into student code, passing it a pointer to the memory
- Student code jumps back into autograder
- Autograder asserts that code behaved correctly, and explodes if not
- Autograder free()s memory it or the student code allocated
- Autograder exits
valgrindcomplains if any memory was not freed
This setup works well enough, but it has several big limitations.
Sub-problem: False "Leak" Reports on Assertion Failures
When an assertion fails, the whole autograder exits. This happens before step #6 happens, since step #5 needs to inspect data structures before freeing them. This means that when this inspection fails, memory does not get freed, and valgrind reports a leak. Current autograders mitigate this by only running valgrind on individual tests that have passed, but when students use the valgrind task in the Makefile and have failing tests, they find memory leaks reported for other tests confusing.
free()ing Uninitialized Pointers in Data Structures
When testing student functions that allocate data structures, as part of
#6, the autograder must traverse the data structure allocated by the
student and free the entire thing. However, if the data structure
contains a bogus pointer, namely if a student forgets to initialize the
pointer, then the autograder calling
free() on it can abort the
autograder. The GNU C library prints a gigantic scary exception which
students don't understand ("what am I freeing wrong?").
Solution: Managed Heap
astro will manage the simulated heap outside the simulator, so it knows what has been allocated and what hasn't.
Problem: GNU/Linux Error Messages Are Cryptic
A common question from students is "what does 'Segmentation Fault'
mean?" If astro prints a line number and stack trace, that could help
student confusion significantly. Same applies to when
free() gets an