Improve isolation, execution limits and execution metrics by using Isolate

[Brikaa]

Summary

This patch integrates Isolate to manage isolation, set execution limits and report execution metrics for each submission. I use a fork (with very simple changes) of Isolate that we made for our graduation project that does not require systemd.

Improvements

1. Fine-Grained Execution Limits

• CPU-Time Limiting: Submissions can now be restricted by both CPU time and wall time. Previously, only wall time (total elapsed time including IO, context switches, etc.) could be limited.
• Memory Management with cgroups: Memory limits are now enforced on all processes spawned by a submission using cgroups. Previously, with prlimit, child processes inherited the memory limit rather than contributed to it, leading to the possibility of memory exhaustion.

2. Enhanced Execution Metrics Reporting

• The response now also includes detailed metrics for each stage (compile, run) of the submission:
  • Peak consumed memory
  • CPU time
  • Wall time

3. Enhanced isolation

• Isolate uses namespaces and chroot for isolation leading for better sandboxing. For instance, submissions can't now cheat via reading and writing files to the shared /tmp directory. They also can't view all of the processes running on the system now (e.g, via ps aux)

4. Other improvements

• The Docker container now runs as an unprivileged user.

Backward Compatibility

1. Self-Hosting (might be breaking)

• Compatible with Linux environments where cgroup2 is enabled and cgroup v1 is disabled. This is the default configuration on most distributions.
• The privileged flag is needed if using docker run. The patch adds the flag to the docker-compose files and the scripts using docker run

2. API Compatibility (compatible)

• The API remains backward-compatible. Only new fields are added to requests and responses.
  • Request: New fields compile_cpu_time and run_cpu_time are added for CPU time limits. compile_timeout and run_timeout continue to represent wall-time limits.
  • Response: New fields cpu_time, wall_time, and memory are added for each stage.

Testing

The patch has been tested using the following NodeJS script:

const assert = require('assert');
const WebSocket = require('ws');
const BASE_URL = 'http://localhost:2000/api/v2';
const WS_URL = 'ws://localhost:2000/api/v2';

const sendRequest = (method, url, body, signal) => {
  const opts = {
    method,
    signal,
    headers: {
      'Content-Type': 'application/json'
    }
  };
  if (method.toLowerCase() !== 'get' && method.toLowerCase() !== 'delete')
    opts.body = JSON.stringify(body);
  return fetch(url, opts);
};

(async () => {
  {
    console.log("Testing 'Hello world' in an interpreted runtime (bash)");
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: 'echo Hello world'
        }
      ]
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.stdout, 'Hello world\n');
    assert.ok(!!body.run.memory);
    assert.ok(!!body.run.cpu_time);
    assert.ok(!!body.run.wall_time);
  }

  {
    console.log("Testing 'Hello world' in a compiled runtime (C++)");
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'c++',
      version: '*',
      files: [
        {
          content: `#include <iostream>
int main() {
  std::cout << "Hello world\\n";
  return 0;
}`
        }
      ]
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.stdout, 'Hello world\n');
    assert.ok(!!body.compile.memory);
    assert.ok(!!body.run.memory);
    assert.ok(!!body.compile.cpu_time);
    assert.ok(!!body.run.cpu_time);
    assert.ok(!!body.compile.wall_time);
    assert.ok(!!body.run.wall_time);
  }

  {
    console.log('Testing memory limit disabled in bash');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: 'echo Hello world'
        }
      ],
      run_memory_limit: -1
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.stdout, 'Hello world\n');
    assert.ok(body.run.memory !== null);
    assert.ok(body.run.time !== null);
  }

  {
    console.log('Testing over memory limit in C++');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'c++',
      version: '*',
      files: [
        {
          content: `const int N = 14e6;
char mem[N];

int main()
{
	for (int i = 0; i < N; ++i)
		mem[i] = 1;
	return 0;
}
`
        }
      ],
      run_memory_limit: 13000000
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.status, 'RE');
  }

  {
    console.log('Testing under memory limit in C++');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'c++',
      version: '*',
      files: [
        {
          content: `const int N = 11e6;
char mem[N];

int main()
{
	for (int i = 0; i < N; ++i)
		mem[i] = 1;
	return 0;
}
`
        }
      ],
      run_memory_limit: 13000000
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.code, 0);
  }

  {
    console.log('Testing over-cpu-time limit in bash');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: `while :; do
    # Do nothing
    :
done`
        }
      ],
      run_cpu_time: 100
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.status, 'TO');
  }

  {
    console.log('Testing under cpu time bash');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: 'echo Hello world'
        }
      ],
      run_cpu_time: 100
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.code, 0);
  }

  {
    console.log('Testing over-wall-time limit in bash');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: `sleep 0.5`
        }
      ],
      run_timeout: 300
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.status, 'TO');
  }

  {
    console.log('Testing under wall time bash');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: 'sleep 0.1'
        }
      ],
      run_wall_time: 300
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.code, 0);
  }

  {
    console.log('Testing over environment cpu time bash');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: `while :; do
    # Do nothing
    :
done`
        }
      ]
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.ok(body.run.cpu_time < 1150);
    assert.equal(body.run.status, 'TO');
  }

  {
    console.log('Testing interactive bash execution');
    const ws = new WebSocket(`${WS_URL}/connect`);
    await new Promise((res, rej) => {
      ws.on('open', () => {
        ws.send(
          JSON.stringify({
            type: 'init',
            language: 'bash',
            version: '*',
            files: [
              {
                content: `echo hello world`
              }
            ]
          })
        );
        let received_data = false;
        ws.on('message', (data) => {
          const body = JSON.parse(data);
          console.log(body);
          if (body.type === 'exit') {
            ws.close();
            if (received_data) res();
            else rej();
            return;
          } else if (body.type === 'data') {
            received_data = true;
            assert.equal(body.data, 'hello world\n');
          }
        });
        ws.on('close', (code, reason) => {
          console.log({ code, reason: reason.toString() });
        });
      });
    });
  }

  {
    console.log('Testing over overridden cpu time C++');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'c++',
      version: '*',
      files: [
        {
          content: `int main() {
  int i = 0;
  while (true) {
    ++i;
  }
}`
        }
      ]
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.ok(body.run.cpu_time < 1000);
    assert.equal(body.run.status, 'TO');
  }

  {
    console.log('Testing internet access in bash');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: `echo > /dev/tcp/8.8.8.8/53`
        }
      ]
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.code, 0);
  }

  {
    console.log('Testing killing parent process in bash');
    const res = await sendRequest('POST', `${BASE_URL}/execute`, {
      language: 'bash',
      version: '*',
      files: [
        {
          content: `kill -9 $PPID`
        }
      ]
    });
    const text = await res.text();
    console.log(text);
    const body = JSON.parse(text);
    assert.equal(body.run.signal, "SIGKILL");
  }

  {
    console.log('Running 128 bash submissions in parallel (should take over double the time of a single submission)');

    const promises = [];
    const before = new Date();
    for (let i = 0; i < 128; ++i) {
      promises.push(
        sendRequest('POST', `${BASE_URL}/execute`, {
          language: 'bash',
          version: '*',
          files: [
            {
              content: `sleep 0.3`
            }
          ]
        })
      );
    }
    await Promise.all(promises);
    const time = new Date() - before;
    console.log(`time taken: ${time} ms`);
    assert(time >= 600 && time < 10000);
  }
})();

The patch was tested on Piston's development environment with bash and gcc runtimes installed. The following docker-compose.dev.yaml was used:

version: '3.2'

services:
    api:
        build: api
        container_name: piston_api
        privileged: true
        restart: always
        ports:
            - 2000:2000
        volumes:
            - ./data/piston/packages:/piston/packages
        environment:
            - PISTON_DISABLE_NETWORKING=false
            - PISTON_RUN_CPU_TIME=1000
            - PISTON_LIMIT_OVERRIDES={"c++":{"run_cpu_time":700}}

    repo: # Local testing of packages
        build: repo
        container_name: piston_repo
        command: ['--no-build'] # Don't build anything
        volumes:
            - .:/piston

[HexF]

LGTM