Weldon is mainly a system built on top of any OS and editor allowing students to practice TDD, aggressive testing and more generally Python.
Weldon also allow a battle mode, where teams of coders confront against teams of testers.
Players are the main user of Weldon, and generally named students or coders (rooters are the admin/teachers). Rooter starts a server, to which he can add problems. Problems are retrieved by coders that can then work as they wish on it, and when wanted submit their source code, that constitute their solution to the problem.
Coders answers are collected by the server, and tested against the unit tests defined in the problem. Unit tests comes in 3 flavors : public, hidden and community. Public tests are accessible by coders, so they can see what to expect from tests. Hidden tests are not accessible by coders. This ensure that they do not code a 1-to-1 mapping of I/O. Community tests are uploaded by coders. These allow the students to perform aggressive testing against each others.
Testers are a special flavor of players that have for only job to upload tests that fail in other players submissions.
Weldon is currently in POC stage : all codes here are mainly here to show that the goals are realistic and reachable. These codes should not be used in production.
Run or read module problem01.py, that contains a linear use case where a teacher create a problem and a student send solutions and tests then retrieve the results.
Module webclient.py is exactly the same story, but using the over-the-network API of weldon. Therefore, an instance of the webserver.py must run during webclient execution.
Create your virtualenv with pytest inside. Run the modules you want to. Hack around.
Currently weldon is not packaged, but that's something that will naturally arise when Weldon will quit the POC stage.
- two user level: player and rooter. A rooter is a player.
- a user is identified by a unique secret token sent by server after registration.
- problem definition: have description, title, and tests.
- tests are divided into 3 category: public, hidden and community. The second contains tests that are known only by teachers.
- users can register using a password defined by Server's admin.
- submissions (source code for a given problem) is tested using pytest.
- submission result (output of pytest + parsed information) is returned to user.
- submission result are kept in a local database. Further exploitation of these data is possible.
- users with last submitted source code meeting all the tests can upload community tests for everyone.
- basic permission management: only teachers can access hidden tests of a problem.
- new role: tester, a player that do not have to submit a solution to upload new tests. Testers gain points by uploading tests that are failed by players.
- in pytest return result, filter out lines that expose the content of hidden tests.
- allow teacher to provide at problem definition a reference implementation. Any community test that fails against reference implementation is rejected.
- allow arbitrary number/hierarchy of file for user submissions
- allow arbitrary number/hierarchy of file for test submissions and definition
- allow arbitrary number/hierarchy of file for test problem definition (that could include public & community tests, readme, instructions, code template,…)
- basic gui allowing student to interact with the server without knowledge of the protocol
- gui: better printing of the submission results
- gui: printing of the report (will need works on server code to get a formal representation of reports, or a very direct printing of the report as a multiline label)
- gui: make a distributable, dinopython compatible, single-file, version of the gui, so it can be used on any machine having python.
- no third-party module
- cross python compliancy OR two living versions at the same time (using 3to2https://pypi.python.org/pypi/3to2)
- single file packaging (students just have to run the file, that can easily be sent by mail or retrieved from the forge) ()
- ALTERNATIVE: compilation for main OSs to a unique executable file.
- automatic generation of a single file module for client side of the webinterface so they can use the functions by just pluging it into their project.
- management of tests as a list of complex objects having implementation and author
- provides more insight into tests using static analysis (function name, fixtures,…)
- human readable serialization of server, so it can be edited and reloaded easily
- server configuration by command line or from any rooter account
- provides way to control student names (could force them to use their official mail)
- configuration of testers by problem (how many ? randomly choosen ?).
- system to avoid naming collisions leading one player to hide a community test by providing a new test with the same name (thank you, ast analysis !).
- perform report for each player, providing him results about number of passed tests (using bashplotlib) and pylint rate and messages.
- perform post-session report for each player, providing him insight of the data (graphic of number of passing (hidden) test and regression according to time, for instance), and the hidden tests.
- perform during-session report for all players, providing insight about best players/testers.
- allow teachers to know which players are working on a particular problem.
- allow teachers to access players report.
- allow teachers to close submissions to a particular problem.
- allow teachers to add new public and hidden tests.
- cut this implementation into dedicated files.
- perform a real sandboxing to avoid attacks by players on the server (firejail ? apparmor ?).
- provides one supplementary problem example, with different use-case and more complex story (involving multiple problems and students).
- over the network discussion between server and players, allowing the program to be used by multiple players without access to the server machine, and also ensure that players do not access internal data of the server (that includes hidden tests and everyone's users).
- verify multi-user handling of the WebInterface
- network discussion needs encryption: using wireshark its easy to intercept token, source and submissions of others players.
- found how to perform asymmetric encryption using python
- integrate it into code
- allow clients to perform encryption if the python module is available.
- extend weldon to work with arbitrary command for testing result generation (this leads to arbitrary language support).
- arbitrary language and unit test framework support.
A sandboxing solution for testing is a necessity before employing Weldon in production.
chrootbuilder Use OS related data for fake a root hierarchy and sandbox a script.
Cons: needs to be read because of many linux distribution-dependant paths It's an automatic chroot jail generator.
Cons: not fully secure
firejail have simplicity in mind.
Used to quickly discard some codes for community tests.
Over the long-term, could be used to detect missing functions into the
submitted code, or unexpected behavior of the code itself
(access to some libraries like
importlib, usage of particular builtins like
Currently, the community tests, at submission, are tested in the following way:
- it must have a correct syntax.
- it must have only one function, eventually decorated.
- function name must starts with
- it must hold at least one assert or one call to pytest.
- it must not take any parameter.
- it must not import some modules (shutil and importlib for instance)
- it must not use some builtins (locals and globals for instance)
These restrictions are somehow violent, because they prevent users to use pytest fixtures system, and not enough against a really motivated attacker. All these verifications are deactivable if necessary.
Data exchange protocol
Server and client exchange data using the following protocol.
Both client and server send a JSON-formatted dictionary with two keys:
encryption_key: contains the encryption key (null when no encryption)
payload: contains the (possibly encrypted) data
Server also provides
status, which is either
In fail case, the payload is not encrypted and contains the error
message describing the origin of the failure (for instance: bad parameters for a command).
The first step of communication between a client and a server, if the client wants its communications to be encrypted, is the retrieval of server's public key. This method is integrated in the API itself. The second step is made during registration, where the client may communicate its public key.
Server and client expect payloads to be encrypted with the public key they communicate.