Coding Runner

Coding runner

Coding questions are graded by @assessment-os/runner via the API.

Mock vs Judge0

Mode	When
Mock (default)	USE_MOCK_RUNNER=true or JUDGE0_URL unset — local processes (pytest, Jest, PHPUnit, JUnit, g++, etc.)
Judge0	JUDGE0_URL set and USE_MOCK_RUNNER=false

Production (Kubernetes)

Prod uses the mock runner with tools baked into docker/Dockerfile.api. ConfigMap: USE_MOCK_RUNNER=true. Rebuild/roll the API image after changing those packages (make update).

Unit mode on Judge0 (optional)

Unit questions submit a multi-file job (language_id 89): zip of solution + tests + compile / run scripts. Output is parsed like the mock runner.

AssessmentOS unit image (pytest, Jest, PHPUnit, JUnit jar, GoogleTest):

docker compose -f docker-compose.yml -f docker-compose.judge0-unit.yml up -d --build
# .env: JUDGE0_URL=http://localhost:2358  USE_MOCK_RUNNER=false
./scripts/smoke-judge0-unit.sh

Dockerfile: docker/judge0-unit/Dockerfile. Overlay: docker-compose.judge0-unit.yml. Cluster manifests: k8s/07-judge0.yaml (not applied by default).

The stock judge0/judge0 image may not include those tools — use the overlay above or keep the mock runner for local/CI.

I/O mode uses normal per-language Judge0 IDs.

Unit config fields

Field	Purpose
language	python → pytest; javascript/typescript → Jest; php → PHPUnit; java → JUnit 5; cpp → GoogleTest
framework	Derived if omitted
entryFile	Main solution path
starterCode	Entry file seed
starterFiles	Extra { path, content }[] (max 20 files / 256KB with workspace)
visibleTestCode	Suite candidates can run
hiddenTestCode	Scoring suite — never sent to candidates
timeLimitMs / memoryMb	Runner limits
scoring	proportional (default) or all_or_nothing
checkerCode	Optional Python I/O checker (exit 0/1)

Run uses the visible suite; submit grades from the hidden suite.

stdin/stdout (mode: "io")

{
  "language": "python",
  "mode": "io",
  "starterCode": "...",
  "visibleTests": [{ "id": "v1", "stdin": "2 3\n", "expectedStdout": "5\n" }],
  "hiddenTests": [{ "id": "h1", "stdin": "10 20\n", "expectedStdout": "30\n" }]
}

Questions without mode behave as I/O for backward compatibility.

Multi-file workspace

• Entry file content comes from starter code.
• Additional starter files ship as a path→content map; candidates edit in multi-tab UI.
• Save / run / submit send the full file map to the runner.

Agent authoring

Via MCP add_coding_question, prefer mode: "unit" with both visible and hidden test files. Do not put scoring assertions only in visible_test_code.