Distributed Judging

Distributed judging and web/judge separation

English | 简体中文

Architecture and deployment layouts

For full AI execution, use --role all on the web node, keeping its local judge/AI worker, then add remote judges for ordinary submissions. For complete separation, use --role web; it does not build local rootfs, so remote judges handle ordinary submissions.

flowchart LR
    Browser[Browser] --> Nginx[Nginx]
    Nginx --> Frontend[Frontend assets]
    Nginx --> Backend[Backend]
    Backend --> DB[MariaDB]
    Backend --> Redis[Redis]
    Backend --> MinIO[MinIO]
    Judges[Local or remote judges] <-->|Authenticated WebSocket /api/socket| Backend
    Judges --> Sandbox[Per-node simple-sandbox and rootfs]
    Judges -->|Signed /storage/ downloads| Nginx
    Backend -->|all mode: local UNIX socket| AIWorker[Local AI sandbox worker]

Loading

Judges initiate connections and need no inbound web port. They do not hold database, Redis or MinIO administrator credentials. They receive tasks, download/cache test files through signed URLs and return results. This deployment uses a single backend, not a highly available backend/database cluster.

Judge queue

Judge tasks are kept in the database (table judge_job), not only in Redis: a submission, Hack or Self Eval and its task are saved in one transaction, so restarting the backend or Redis loses no queued work. Redis only wakes up waiting judges; they also look for tasks every few seconds.

• A task is sent to one judge connection at a time, under a lease that the backend renews while the judge stays connected. The result is stored together with finishing the task, and a result from an older lease (for example after a reconnect) is ignored.
• When a judge disconnects, its unfinished tasks are queued again at once. When the backend itself stops, its leases expire after 90 seconds and another backend process, or the restarted one, queues them again. A task can therefore be judged more than once, but its result counts once.
• A task is given up on after 5 interruptions, or after 10 failures to prepare it (for example missing test data). Its submission then shows System Error with the reason and can be rejudged.
• Site administrators see the queue at the bottom of Judge Machine (/judge-machine): tasks waiting or being judged, the judge machine holding each, retries, and the tasks given up on in the last 7 days. The libreoj_judge_jobs metric reports waiting and running tasks.
• Every lease is recorded in judge_job_attempt with how it ended. Finished tasks and their records are deleted after 30 days.

Drain, revoke and capabilities

These are on the Judge Machine page for site administrators, and every change is recorded in the audit log (judge_client.add, delete, reset_key, drain, resume).

• Drain (pause icon) stops sending new tasks to one judge machine; it finishes the tasks it has. Its status shows Draining while it still judges, then Drained: it can now be stopped, updated or resized without interrupting judging, while the other judge machines take the queue. It stays drained after reconnecting or a backend restart until you Resume it (play icon). Draining doesn't affect local AI validation, which doesn't use the queue.
• Reset key and Delete take effect at once: every backend process disconnects the connections using the old key, and their unfinished tasks are queued again for other judge machines. Reset the key when it may have leaked, then put the new key on the judge machine.
• Capabilities: when a judge machine connects, it announces its protocol version, the task types it can run, its task threads, execution slots, slots for AI validation and memory budget; the Capacity column shows them. The backend sends it only the task types it announced. A judge machine of an older version announces nothing and is marked Legacy: it is sent only submissions, so Hacks and Self Evals wait until an updated judge machine is online.

Web node

sudo bash /tmp/llmoj-install.sh --repository houyhlouis/LLMOJ \
  --role web --public-url http://oj.example.com

For public remote judging, configure the final HTTPS origin first, or use a controlled private network. HTTP sends judge credentials without transport encryption.

The installer configures WebSocket Upgrade at /api/socket and sets services.minio.forJudge.urlEndpoint to the public origin's /storage/. Nginx restores the internal Host used in MinIO signatures. Database and MinIO remain on loopback.

Register each judge

An administrator registers a node with POST /api/judgeClient/addJudgeClient and body {"name":"judge-01","allowedHosts":[]}.

In the browser Console on the site while logged in as admin, run this request. It uses the current session without printing or copying its token:

if (!window._appState?.currentUser?.isAdmin) throw new Error('Sign in as an administrator first');
fetch('/api/judgeClient/addJudgeClient', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer ' + window._appState.token
  },
  body: JSON.stringify({name: 'judge-01', allowedHosts: []})
}).then(response => response.json()).then(result => {
  console.log(result.judgeClient ? 'Created; save the key privately from this response' : result);
});

In the Network panel, save judgeClient.key from that response to this node's private key file. Register each machine separately. A new connection using the same key replaces the previous session; do not share keys between judges.

The gateway authenticates keys. Although the API stores allowedHosts, the current connection logic does not enforce source-IP checking through that field. Use a reverse proxy, VPN or firewall for source restrictions.

Deploy and verify

Follow Remote-Judge using the same source version/rootfs recipe as the web node. After “Successfully authorized as …”, query status from the administrator Console:

fetch('/api/judgeClient/listJudgeClients', {
  headers: {'Authorization': 'Bearer ' + window._appState.token}
}).then(response => response.json()).then(result => {
  console.table(result.judgeClients?.map(({id, name, online}) => ({id, name, online})));
});

Submit a simple C++ program and confirm downloading, compilation, execution and Accepted. Test compilation errors, time/memory limits and cancellation on each node. Restart a node to inspect recovery; multiple clients can consume the queue, but recovery is not an exactly-once execution guarantee.

AI limitation

Backend model calls, editing, translation and search can still be configured. AI reference-solution validation and test-data generation currently use a local UNIX socket and private shared directories on the web server; there is no cross-machine AI worker protocol. Adding remote judges for ordinary submissions does not move these actions off the web server. They need the local worker retained by all and are unavailable with web. Do not replace the private socket with an open TCP service or uncontrolled shared directory.

Troubleshooting

• Connection fails: check final origin, TLS, /api/socket Upgrade and the node's key.
• Online but downloads fail: inspect signed URLs for localhost/private endpoints and check /storage/ Host handling.
• Nodes replace one another: check for shared keys.
• cgroup errors: check systemd delegation, cgroup v2 and kernel support.
• AI execution fails despite remote judges: check whether the web node retains a local worker.
• Hacks or Self Evals stay waiting: check that an online judge machine that isn't drained runs the current version (not Legacy) and lists them under Capacity.
• A judge machine gets no tasks: check whether it is drained, and resume it.

Real multi-machine deployment has not yet been validated end to end; complete the on-site tests above.

CPU and existing-node capacity

See Judge configuration for every YAML field, consumers versus execution/download queues, the 8 CPU/8 GiB example, and resizing/rollback while retaining keys. Maintain and verify one node at a time. More remote slots do not increase the web node's local AI worker capacity. Do not change installer metadata or rerun with a different --judge-slots as a substitute.