Phobos – Sandboxing for Programming Exercises

Overview

Phobos is a sandboxing solution for the Artemis e-learning platform, designed to securely run student code tests by restricting filesystem and network access to only what’s necessary. It works in two phases:

• Resource Discovery (Pruning) Phase: An offline phase where Phobos automatically discovers which files, directories, and external network hosts are truly needed for the tests to run. It does this by running sample exercises in an instrumented environment and “pruning” away unnecessary resources, producing a whitelist of required paths and allowed network endpoints.
• Application (Sandboxing) Phase: The runtime phase where Phobos is applied to student submissions. The tests execute inside a confined environment (using Linux namespaces via Bubblewrap that only exposes the whitelisted files (mostly in read-only mode) and blocks all other files by mounting empty directories (tmpfs). Network access is similarly constrained using a custom preload library (“netblocker”) that intercepts network calls and only allows connections to whitelisted hosts. This ensures student code cannot access undeclared resources or external services beyond what’s been approved.

By integrating Phobos with Artemis, instructors and administrators can run student tests with greater security and academic integrity. Phobos extends Artemis’s existing test infrastructure (e.g. Maven/Gradle, pytest, etc.) with an additional layer of sandboxing without requiring changes to the tests themselves, by wrapping the normal test execution inside the sandbox.

Repository Structure and Components

The project is organized into components for the two phases:

• Pruning (Resource Discovery) Tools: Located primarily under the docker/prune_phase directory (and associated scripts in the repository). This includes the orchestrator script and any Docker configurations needed to run the discovery phase. The repository also contains example output files under a var/tmp structure (e.g. path whitelist files) which illustrate the results of pruning.
• Phobos Core (Sandbox Runtime): Located in the core/ directory. These are the scripts and binaries used during the sandboxing phase. Notable contents include:
  • phobos.sh: The main entry-point script to apply Phobos sandboxing when running tests. It sets up the restricted environment (file binds, network rules) and then invokes the actual build/test commands inside that sandbox.
  • libnetblocker.so: The compiled netblocker library used for network call interception. This library is preloaded into the test process to hook network-related system calls (e.g. DNS resolution and socket connect). It ensures only allowed hostnames/IPs can be contacted, blocking others by failing DNS lookup or connect calls.
  • Whitelist Files: The allow-list of file system paths that were generated by the pruning phase. These might be provided as *.paths files (e.g. java_union.paths, python_union.paths, etc.) under a directory (e.g. var/tmp/path_sets/ in the repo). They enumerate directories with required access mode – e.g. lines starting with r for read-only binds or w for writable binds. Phobos uses these to know which directories to mount in the sandbox.
  • Dockerfiles: The repository contains Dockerfile(s) for building the Phobos Docker images. For example, a Dockerfile extends Artemis’s base test image (such as the ls1tum/artemis-maven-template for Java) by adding the Phobos core files and any needed dependencies (like Bubblewrap). The Docker image published (e.g. ajayvir/phobos on Docker Hub for Java) includes all Phobos components and config pre-installed on top of the standard Artemis test environment.
• Miscellaneous: Additional scripts or configs for advanced usage, e.g. an orchestrator script (orchestrate.py) to coordinate multi-language pruning (explained below), and configuration hooks for special cases (skip patterns, forced binds, etc., as discovered during development).

Resource Discovery Phase (Pruning)

Before using Phobos in production, you should run the resource discovery (prune) phase to generate the sandbox configuration for your particular programming language and test framework. This phase is typically done once per language environment or whenever the environment changes (e.g. upgrading to a new JDK or new testing library version). The goal is to determine the minimal set of files and network resources that must be accessible for any student’s code to compile and pass the tests in that environment.

How Pruning Works

Phobos uses a top-down “prune-by-hiding” algorithm to automatically find unnecessary vs. required files on the filesystem:

• It begins with an environment where all filesystem locations are initially accessible (writable by default). Then it iteratively hides entire directories by overlaying them with an empty tmpfs (this is denoted as mode n for “not accessible”) and reruns the tests. If hiding a directory does not break the build/tests, that means nothing in that directory was needed – so Phobos leaves it hidden (excluded). If hiding a directory does cause something to fail, then that directory (or file) is needed. In that case, Phobos restores it and marks it as read-only (r) or writable (w) depending on whether the tests need to write there (it falls back progressively: from hidden to read-only to fully writable).

• This process recurses into subdirectories: Phobos only descends into a directory if that directory was identified as required. Large unused subtrees can be dropped in one shot, whereas required branches are explored further. This top-down approach drastically reduces the number of iterations compared to discovering one file at a time (a bottom-up approach). In practice, the algorithm can converge in tens of test runs even for complex projects, rather than hundreds, by hiding entire branches of the filesystem that turn out to be irrelevant.

• Throughout this process, Phobos observes the outcome of each test run. It treats a test run failure as an indicator that something necessary was blocked, and success as indication that blocked items were truly unnecessary. It has logic to distinguish real test failures from failures caused by missing resources. For example, Gradle can report a build with "NO-SOURCE" (no tests or code to run) as a success (exit code 0) even though it means something went wrong (nothing executed). Phobos accounts for such cases by defining pattern-based rules, e.g. treating compile NO-SOURCE as a failure condition despite the zero exit code. Similarly, it ignores certain expected test failures (like failing tests) if they are known not to be related to resource access, ensuring it only reacts to failures caused by missing resources.

• By the end, the pruner produces a list of all directories/files that remained non-hidden. These are precisely the ones needed for compilation and tests to run. Everything else can be safely hidden in the sandbox. The needed paths are recorded with their required access mode (read-only or read-write) in output files (the “*.paths” files). For example, after running on a Java/Maven exercise, you might get a java_union.paths containing lines like /usr/lib/jvm/java-17-openjdk/... -> r (which means bind this JDK directory read-only) and perhaps a target build directory as -> w (writable). The term "union" here indicates it may combine results from multiple runs (see below).

Multi-Language or Multi-Run Orchestration: If you want to generate configurations for multiple language environments (say Java, Python, C) or multiple sample projects, Phobos provides an orchestration mechanism. You can run pruning in parallel or sequence for each environment and aggregate the results:

• We typically set up one container per environment (for instance, a Java env container with Maven/Gradle, a Python env container with pytest, etc.), each mounting a shared host directory for results. Each container runs its own prune script (e.g. prune:java for the Java container, prune:py for Python) against a reference exercise in that language. All containers share a host folder (bind-mounted at /var/tmp) to collect outputs, so they all write their resulting *_union.paths files into the same location.

• After all prune runs complete, a coordinator script (e.g. orchestrate.py) can be executed (on the host or in a lightweight container with the results mounted) to gather and finalize the whitelist data. The orchestrator essentially sees a directory full of e.g. java_union.paths, python_union.paths, etc., and can merge or format them as needed for the final config. (In many cases, each language’s output is separate, but the orchestrator ensures the workflow is the same without cross-container data copying – everything was shared via the mounted folder.)

• Output: For each environment, you will have a <lang>_union.paths file that lists all required paths and their access modes. You may also get accompanying JSON summaries or a combined config, depending on the tools used (the repository includes an emit_artifacts.py script that can produce .json records of the pruned paths and also a combined configuration file for tail flags, etc., but these are mainly for informational purposes).

Running the Pruning Phase: To perform the pruning phase for a given language, follow these general steps (assuming the repository already contains necessary scripts and Docker configurations):

1. Prepare Reference Projects: Choose one or a few reference exercises that are representative of the environment’s needs. Ideally, these should collectively cover the typical resources needed (e.g. one simple project that compiles and runs tests, possibly another that triggers network usage or other special functionality if applicable). Organize them in the var/tmp/testing-dir directory (each in its own subfolder if multiple). For example, testing-dir/java/assignment and testing-dir/java/tests could hold a Java exercise’s code and test files.

2. Build and Run Pruner Environment Images: The repository provides Docker build files for the pruning environments. For instance, there may be docker/prune_phase/java/Dockerfile (for Java/Maven) and similarly for other languages. These Dockerfiles likely extend the base Artemis image for that language and add tools like Bubblewrap and the Phobos core scripts. An orchestrating docker-compose file is located in the root directory of the project to spin up the containers for pruning for each programming language. No direct inter-container communication is needed; each container works independently on its language and just writes to the shared path_sets volume.

3. Monitor and Iterate: Each prune run will likely execute the tests multiple times, hiding different paths, until it converges. This may take a little while (on the order of tens of test runs). The process is automated by the prune script, so you can let it run. If a run fails due to a legitimately needed resource being hidden, the script will adjust and try again. If it fails due to something else (like a real test failure or misconfiguration), you may need to investigate logs. The prune script logs its decisions; look for lines about mounting n/r/w and any warnings about patterns. For example, it may log that it is hiding /home/user/.m2 (just as an example) and then see a test failure, etc. 4. Finalize Outputs: Once pruning is done, each container will have dropped a <lang>_union.paths file in the shared path_sets directory (and possibly intermediate files or logs). If you used multiple reference exercises for one language, there might be multiple files (e.g. java_proj1.paths, java_proj2.paths which the orchestrator can union-merge). Now run the orchestrator script to finalize. The orchestrate.py --out path_sets is then run by docker-compose (the exact invocation might vary) on the host with access to the path_sets folder. This script will read all *_union.paths and potentially combine them or simply verify them. In many cases, the *_union.paths might already be the merged result of multiple runs for that language (depending on how the prune script was invoked for each exercise). The orchestrator ensures the final whitelist is ready. The final output by the orchestrator is stored into the core/config directory is:

• BaseLanguage-<lang>.cfg: language-specific configuration file which is used to run Phobos Sandboxing
• TailPhobos.cfg: file containing common parameters needed to run the Bubblewrap Sandbox 5. Review and Tweak: Open the resulting .cfg files and review them. They should list directories with r or w. You can manually edit these if needed (for example, if you realize something else should be allowed). Also, check if any network host requirements came up during the run (e.g., did the build attempt to download something and get blocked?). If so, ensure those hosts are accounted for in the network whitelist (see next section). It’s often useful to include any obviously needed external domains at this stage if they weren’t automatically captured.

6. Push Phobos Image: You can manually place the configuration files for the respective languages and the already existing bash scripts from core/ into the respective Docker images used for test case execution (e.g. for Java see file docker/run_phase/java/Dockerfile). We can push the resulting image to Docker hub for public use.

At this point, you have an up-to-date sandbox policy for the environment. The next step is to use it during actual test runs on Artemis.

Sandbox Application Phase

Once the necessary config files and binaries are prepared (either by using the pre-built Phobos image or building your own with the generated whitelist), you can integrate Phobos into Artemis’s test execution for student submissions.

Building/Obtaining the Phobos Docker Image Phobos is packaged as a Docker image that Artemis can use in place of the standard test image. For example, for Java exercises, the image ajayvir/phobos:latest might be provided, which extends the official Artemis Maven template image. This Phobos image includes:

• Bubblewrap: Bubblewrap: The tool used to create unprivileged sandboxes. Bubblewrap is installed in the image (usually under /usr/bin/bwrap). It is invoked by phobos.sh to actually spawn the sandboxed process.
• Phobos Core Scripts: The phobos.sh script and any helper scripts (as well as the path whitelist files) are present in the image, typically under /var/tmp/opt/core or a similar path. In our setup, we place them in /var/tmp/opt/core inside the container, and also use /var/tmp/path_sets for the whitelist data. These paths align with what we used in the prune phase for consistency.
• Netblocker Library: The libnetblocker.so shared object is included (for the correct architecture of the container). The phobos.sh knows the path to this library and will set LD_PRELOAD accordingly at runtime.
• Allowed Network Config: If there are known hostnames that should be reachable by the tests (e.g. a package repository or an external service required for the build), the image or config will include those as well. For instance, if using Gradle and the Gradle wrapper, the host services.gradle.org (which hosts the wrapper distribution) would be whitelisted. The Phobos script can be configured with rules like allow services.gradle.org:443 for network access. These rules can be baked into the image or passed via the .cfg files. We’ll discuss how to supply these to Phobos below.

Configuring Artemis to Use Phobos

1. Use the Phobos Docker Image: In Artemis, update the exercise’s build plan to use the custom Docker image. This means editing the job configuration to point the Docker runner to ajayvir/phobos:latest (or your built image name) instead of the default (e.g. ls1tum/artemis-maven-template:java17-XY). In Artemis’s UI, this can be done by going to Configure Build Plan for the exercise, then navigating to the job’s configuration and setting the image name. If the exercise was just created, you might need to manually adjust this because Artemis by default picks the standard image.

2. Adjust the Test Command to Invoke Phobos: By default, the Docker container will run the test commands (for example, in the Java Maven image, it might run something like mvn clean test or similar). We need to wrap this call with phobos.sh so that the tests execute under the sandbox: phobos clean test.

Once the Docker image and command are configured, Artemis will use Phobos for all subsequent student submissions for that exercise. Each submission’s tests will run inside the sandbox as follows:

1. Phobos Initialization: When the test job starts, it invokes phobos.sh. This script first prepares the Bubblewrap arguments. It reads the whitelist of paths and builds a list of --ro-bind and --bind options for Bubblewrap (read-only and read-write binds, respectively). Every required path from the whitelist is bound to the same path inside the sandbox. If the whitelist marked it as r, it uses --ro-bind, if w then --bind. Anything not in the whitelist will not be bound, which means it will fall under a higher-level tmpfs mount. The script adds --tmpfs for directories that should be hidden entirely (the prune phase implicitly decides these by not marking them as needed). Some special paths like /proc and /dev are mounted by default for basic functionality. The script also handles mount ordering: it sorts binds by depth to ensure that if a parent directory is marked as hidden but a child is needed, the child’s bind isn’t overridden by the parent’s tmpfs.

2. Network Setup: Before launching the sandbox, phobos.sh sets up the network restrictions. As described, it takes the allowed host rules and writes them to an allowedList.cfg in the core directory. Then it sets NETBLOCKER_CONF to point to that file and enables the LD_PRELOAD of libnetblocker.so. At this point, any new process started will have the netblocker library injected from the very beginning of its execution. This is important: the script uses the execve system call via Bubblewrap to launch the test process with LD_PRELOAD set. The dynamic linker will load our library before the program’s main function begins. Netblocker’s constructor will initialize and from then on, any call to getaddrinfo or connect in the program goes through our interception. Netblocker checks the hostname against our whitelist; if it’s not allowed, the call is made to fail as if the host is unknown. If it is allowed, the real getaddrinfo is called and its result (IP addresses) are recorded in an approved list in memory. Later, when a connection is attempted, the library checks if the destination IP was one of those approved (or matches an allowed CIDR range, if we specify ranges) and only then permits the real connect system call. Otherwise, it blocks the connection by forcing an error (setting errno EACCES, meaning “permission denied”). This effectively sandboxes network access to only the hosts you’ve explicitly allowed. Everything else will behave as if the network is unreachable or the host doesn’t exist.

3. Entering the Sandbox: With all mounts and environment ready, phobos.sh then uses Bubblewrap to spawn the sandboxed build process. The final assembled bwrap command will look something like:

bwrap --proc /proc --dev /dev \
      --ro-bind /usr/lib/jvm/java-17-openjdk /usr/lib/jvm/java-17-openjdk \
      --ro-bind /usr/lib /usr/lib \
      --bind /home/artemis/app/target /home/artemis/app/target \
      --tmpfs /home/artemis/.m2 \
      --tmpfs /etc  \
      ... [more binds/tmpfs as per *.paths] ... \
      --share-net \
      -- /path/to/your/buildScript.sh mvn clean test

The build script (Artemis uses something like a build_script.sh internally to set up environment and call the build tool) is executed inside the sandbox with all the constraints in place.

4. Test Execution: Now the tests run as normal, but if they attempt to access something outside the whitelisted set, they will encounter an error. For example, if a test tries to read a file in /home that wasn’t allowed, that path is mounted as an empty tmpfs – the file will simply appear missing (or the directory empty). If code tries to write somewhere not allowed, it writes into a tmpfs that is thrown away at the end (and it wouldn’t affect the host or persist between runs). If the code tries to connect to an unapproved URL, the connection will fail (the code might get a network error exception). Typically, well-behaved tests won’t notice anything because they don’t try to do disallowed actions; if they do, it usually indicates either a misconfiguration (we pruned too much) or potentially a malicious attempt or unintended access, which is exactly what we want to catch.

5. Tear-down: After the tests complete (or timeout), the Bubblewrap process exits. All mounts and the sandbox namespace are destroyed, automatically cleaning up any tmpfs (so any data written to disallowed areas is gone). The phobos.sh script then exits, passing through the same exit code that the test process returned. This ensures Artemis receives the correct result of the tests. If tests failed due to a sandbox violation (e.g., a SecurityException or a resource missing), that will show up as a test failure. As an Artemis admin, you should interpret such failures carefully – if it’s because of a missing resource that should have been allowed, you may need to update the Phobos config and re-run (or consider if the test was trying to do something it shouldn’t).

Conclusion

Phobos provides a two-step approach to secure programming exercise evaluation: first automatically determining what system resources are needed, and then strictly enforcing that policy during test execution. By integrating Phobos with Artemis, administrators can ensure that students’ code only uses permitted files and cannot reach out to the internet or host system in unintended ways.

To get started, run the pruning phase for your target environment, build the Phobos image with the discovered configuration, and configure Artemis to use the image and phobos.sh for test runs. Once set up, Phobos runs transparently during testing, enhancing security without altering the student or test code.

With Phobos, you gain a minimal, reproducible test sandbox – if a submission passes in the sandbox, you can be confident it didn’t rely on anything outside the specified resources. This preserves academic honesty and makes test results more reliable across different environments. Happy sandboxing!