Clone this wiki locally
vmchecker is a grading platform built by students from University POLITEHNICA of Bucharest. vmchecker requires two systems to run:
- A storer system: where assignments are uploaded, stored, submitted for testing and graded by teachers.
- A tester system: where assignments are tested/checked using a virtual machine infrastructure.
Contributing to vmchecker
Make your changes and check Python source code using pylint then push your code. You may request code review on the mailing list or create a pull request. Wait for your code to be pulled into the main repository and a thanks message from the maintainers. Grab yourself a beer or a cup of coffee and celebrate.
We recommend using our virtual machine in order to test your code before submitting a pull request.
Behind the Scenes
vmchecker Folder Structure
bin: A directory that should be contained by the storer machine. It stores the following Python scripts :
vmchecker-download-external-files: A script to download files (that are not stored on the storer) required for testing from another system through SSH.
vmchecker-init-course storer: Initializes the directory path for a class storer machine. Its actions are:
- Creates a sample config file named
configin the current directory. It gets the sample config from
- Creates a sample config file named
- Creates the repo for the assignments on the storer.
- Create the implicit db if it does not exist.
- Copy the example auth file from
- Copy build and run example scripts into the root path
- Run initialization tasks for the tester machine.
- Run initialization tasks for the storer machine.
vmchecker-init-course tester: Initializes the directory path for a course tester machine.
vmchecker-queue-manager: Waits for assignments to appear and invoke
vmchecker-vm-executorto handle them.
vmchecker-submit: Submit assignments from the command line.
vmchecker-update-db: Manually update the db storing info from the repo.
vmchecker-view-grades: Generates a HTML table containing student grades.
vmchecker-vm-executor: Starts a vm, reverts it to a known snapshot, tests a submission bundle (submission + tests), and closes the vm.
bin folder must also reside on the tester machine, where
vmchecker-queue-manager runs as a deamon. For each submission
vmchecker-vm-executor is started.
The vmchecker WebUI authentication is achieved through the university LDAP server. If the user isn't on the LDAP server, then the user is searched in a plain text file, where the user names and the passwords are saved.
Configuring a New Class User on a Storer System
In order to add a new class on vmchecker, you need to add a corresponding user on the tester system and configure vmchecker for that user. You may then configure assignments, request students to submit assignments and then grade assignments.
pc (for "Programarea Calculatoarelor") as the username to describe the steps to follow. The steps are show below. We assume the tester system is
- First you'll need to create the actual user:
$ ssh email@example.com elf:~# adduser pc # don't set a password; use ssh keys elf:~# su - pc pc@elf:~$ ssh pc@localhost # creates .ssh directory with default permissions pc@elf:~$ emacs .ssh/authorized_keys # put your id_rsa.pub key here and log-out
- Create and initialize the vmchecker class folder:
$ ssh firstname.lastname@example.org pc@elf:~$ mkdir vmchecker-storer pc@elf:~$ cd vmchecker-storer/ pc@elf:~/vmchecker-storer$ vmchecker-init-course storer
- Create ssh key to connect to tester system:
pc@elf:~/vmchecker-storer$ ssh-keygen -f ~/.ssh/id_rsa-vmchecker -N '' # creates a key with no password
- Add the SSH key to the corresponding user on the tester system (we'll assume it is calleed
checker.cs.pub.ro; you will use whatever you have configured):
youraccount@yourhostname:~$ ssh email@example.com pc@sanctuary:~$ emacs .ssh/authorized_keys
- Test connection and also save the fingerprint to
pc@elf:~$ ssh -i .ssh/id_rsa-vmchecker firstname.lastname@example.org The authenticity of host 'checker.cs.pub.ro (184.108.40.206)' can't be established. RSA key fingerprint is 8a:0a:51:86:46:85:17:85:b6:84:3d:e1:b5:68:45:01. Are you sure you want to continue connecting (yes/no)? yes
- Add email@example.com's public key in the
pc@elf:~$ emacs .ssh/authorized_keys # paste-in firstname.lastname@example.org's password
- Make private key readable by
www-dataread access to the file):
pc@elf:~/.ssh$ setfacl -m u:www-data:r id_rsa-vmchecker
- Edit vmchecker config file:
pc@elf:~/vmchecker-storer$ emacs config
You'll find a sample config file in the repository.
Add tests to
<asg-name>is the name of each assignment as specified in the
Add new class to global list:
$ ssh email@example.com elf:~# emacs /etc/vmchecker/config.list Add this to the file: PC:/home/pc/vmchecker-storer/config
The class folder is generally stored in
class_name is the name of the class:
programare) and will look something like this:
/home/class_name/vmchecker-storer |--tests |--asg1_tests.zip |--build.sh |--run.sh |--config |--auth_file.json |--vmchecker.db
auth_file.json is a plain text file that contains the user name and password for the users that are not in the ldap server.
vmchecker.db is an SQLite3 database that contains the paths to the assignment archives that will be graded.
Adding a New Assignment
In order to add a new assignment the user must:
- Add the assignment in the config file.
- Upload the assignment tests.
- Upload the
- Upload the
Each class specific folder (
vmchecker-storer/) resides on the storer system contains a repository where assignments are saved. The repository follows a hierarchical structure consisting, on each level, of:
- 1st level: assignment name
- 2nd level: student name that submitted assignment for the 1st level
- 3rd level: submission id of submission by student for the given assignment
- 4th level: three items
archive.zip: a compressed archive of the submission
archive/: a folder representing the uncompressed submission
results/: a folder that stores assignment test results and grading
results/ folder is created after the homework is sent back from the tester machine after being tested/checker. It stores the entries below:
submission-config: contains assignment details.
grade.vmr: contains the grades given by the teacher
run-stdout.vmr: contains the results of the automatic tests.
build-stdout.vmr: contains the results of the build.
Uploading an Assignment File
In order to upload an assignment a user must log in the WebUI, browse for the assignment archive and click submit.
On the click of the submit button a new directory is created in the course name with the user name and the assignment archive, if it does not exist.
Resubmitting an Assignment from CLI
It may happen that due to an error on the tester system side, an assignment needs to be resubmitted; to preserver timestamp and other. You may use the
submit.py script for this, while located in the class folder. A sample run is:
./bin/submit.py 1-minishell-linux Alexandru\ Moșoi path_to_some_archive.zip