Architecture

ACE (Agentic Coding Evaluation) is a Java 21 / Spring Boot backend plus a Vaadin 25 web front end. It drives a coding agent through a fixed, non-trivial build task (a small trading platform — Spring Boot

PostgreSQL + Flyway + React + Docker Compose microservices), then scores the result with an oracle that never trusts the agent's own word for anything.

This page covers how the system is actually built, with citations into the real source. For the why behind the choices described here, see Design Decisions. For a screenshot-driven walkthrough of the UI, see the User Manual.

1. The two modules

Both are Gradle subprojects of one root build (./gradlew :ace-service:..., ./gradlew :ace-ui-vaadin:...):

ace-service — the harness itself: job queue + worker, benchmark runner, scoring oracle, recording proxy, provider registry, and a REST API. Runs on port 8765.
ace-ui-vaadin — a Vaadin 25 web UI that is a pure REST/SSE client of ace-service. ~8,700 LOC across 14 routed views. Runs on port 8800 and talks to ace-service over HTTP.

ace-service persists its own operational state (jobs, providers, experiments) in a local SQLite database, migrated with Flyway (src/main/resources/db/migration/V1..V8__*.sql), queried through jOOQ's DSLContext. This replaced an earlier PostgreSQL-backed store partway through development — there is no external DB server or Testcontainers container behind ace-service's own data anymore (PostgreSQL only reappears as something the agent's own generated stack uses, and as a real sidecar the build-check container talks to — see §6).

Source layout (ace-service/src/main/java/com/strgmai/ace/service/):

Package	Contents
`agent/`	`ReferenceAgent` (in-process, LangChain4j) and `PiAgent` (external subprocess) — the two selectable harnesses; `AgentTools`, `AgentSession`, `CompactionStrategies`, `ProviderStore`
`config/`	`BenchProperties` (`@ConfigurationProperties(prefix="ace")`), `JsonColumns` (jOOQ JSON binding)
`docker/`	`DockerService` — passive Docker-running probe, CLI-plugin mirroring
`metrics/`	`Collect`, `Trajectory`, `StatsService` — cost, trajectory analysis, k-run statistics
`oracle/`	`CheckId` (the 30-check registry), `CheckStatus`, `CheckResult`, `Scorer`, `RunOracle`
`oracle/checks/`	`StructureChecks`, `MoneySafetyChecks`, `BuildChecks`, `ComposeChecks`, `BlackboxScenarios`
`pack/`	`Packs` — stable pack, task pack, parallel-plan pack, review/trajectory-review packs, fix pack
`plan/`	`PlanParser`, `PlanTask`, `PlanError` — plan-text parsing into topologically ordered tasks
`proxy/`	`RecordingProxy` (journals every LLM request/response), `SseAssembler`
`reference/`	`TradingService` (+ 9 seeded bugs), `TradingController`, `ReferenceServer` (calibration twin)
`runner/`	`RunBench` (orchestrator), `RunBenchSupport`, `ContextProbe`, `Validity`, `VerifyTask`, `JournalFacts`, `Reviews`, `Judge`, `Intervene`, `PauseAfterTask`, `TaskMatcher`
`service/`	`JobQueue`, `WorkerService`, `ExperimentsService`, `ImporterService`, `BenchController`, `Preflight`, `RunSpec`, `Tailer`, `TreatmentPin`, `JobCfgFactory`, `TaskTranscript`

2. What is being measured

The agent is driven to build a small trading platform to a frozen spec (src/main/resources/task/PROMPT.md): Java 21 / Spring Boot 3 / Gradle, PostgreSQL + Flyway, a React frontend, OpenAPI 3.1, Docker Compose, at least 3 microservices, with a frozen HTTP contract (account creation, HALF_EVEN-rounded deposits, idempotent order repeats, asOf-scoped point-in-time holdings queries, fixed price table, health endpoints). The task ladder (src/main/resources/tasks/ladder.json) defines graduated rungs — L1_migration_entity … L7_full_platform, plus a plan-first variant L3p_point_in_time — each a strict subset of the full 30-check, 77-weight registry. See Features for the full check table.

3. Job queue and worker model

A job is a single benchmark run (or one arm of an experiment), submitted via the UI's Jobs → New Job form (POST /api/jobs, handled by BenchController) and tracked by service/JobQueue.java and service/WorkerService.java:

JobQueue owns the persisted job rows (status: queued → running → succeeded/failed/cancelled, plus paused) and the enqueue/cancel/pause/resume/requeue operations the REST layer and UI call into.
WorkerService is the piece that actually turns a queued row into a live benchmark: it picks the next queued job, builds the run configuration, and hands it to runner/RunBench.java to execute. WorkerService's own guard enforces Preflight's pinned-JDK check as a fatal gate — a job is refused outright, not just flagged in the UI, if the operator's environment doesn't have a usable JDK 21.
RunSpec.java is the typed representation of a job's configuration — the --flag=value argv-style list seen on every job detail page's "Run flags" line (model, harness, mode, task, budgets, judge, lesson-recording, compaction strategies, context-window, etc.) — and JobCfgFactory.java is what turns the raw form/API input into that spec.
Preflight.java runs a real gradle test against the positive-control fixture in the agent's own environment before any run is trusted to work at all — see Features for what it checks and User Manual: Providers & Preflight for the UI.
Tailer.java streams a running task's live log/transcript to the UI over SSE; TreatmentPin.java pins the comparability-relevant fields of a run's configuration (the "treatment") so later statistical comparisons can verify two runs are actually comparable.

Jobs run one at a time per ace-service instance (a single-run lock, see §4) — the queue serializes work rather than running benchmarks in parallel on the same machine, since a benchmark run already saturates the local model server and/or the Docker daemon on its own.

4. The runner — `RunBench`

runner/RunBench.java (~2,500 lines) is the orchestrator. Two harness modes, selected by the mode config value:

Monolithic — one growing agent session handles planning and implementation together, with structural compaction triggered by a token threshold as context accretes.
Orchestrated — the plan is split into independent tasks, each its own session, composed from a stable pack (byte-identical system-prompt prefix across every task, so the model server's prefix cache serves it) plus a deterministic per-task pack (the task entry, its dependencies' interfaces only, repo tree, previous task's diff, test verification, and progress notes). A fixed INTEGRATION task always runs last. The class-level Javadoc also claims parallel task waves are supported (Packs.parallelPlanPack() plus merge/conflict-handling code in RunBench) — its correctness under load has not been independently verified.

Provenance and isolation, from RunBenchSupport.java:

A scrubbed environment is built for every agent subprocess, with ACE_RUN_ID set into it.
The workspace is git-snapshotted after every phase/task, so a run's artifact trail is reconstructible.

Separately, WorkerService.guard() acquires a single-run lock at ~/.cache/agentbench/run.lock before a job is allowed to start, so concurrent runs never race for the same local model server. (RunBenchSupport.acquireLock() exists in source but is currently unused — the real lock is the one in WorkerService.)

Validity (runner/Validity.java) decides whether a finished run is even eligible to be scored or pooled: a run is invalid (recorded, but never ranked) on a non-budget agent crash, a length finish with no artifact, a dead recording proxy, an error rate above maxErrorRate, zero recorded LLM requests, a request referencing another run's workspace, an unparseable plan, or — in orchestrated mode — the INTEGRATION task never running. Budget exhaustion is explicitly not an invalidation; it's treated as a real result (see Design Decisions).

Two selectable harnesses exist side by side, both constructor-injected into RunBench and picked per run via --harness=:

agent/ReferenceAgent.java (AGENT_VERSION = "jls-ref-1.0") — an in-process LangChain4j agent against an OpenAI-wire-compatible endpoint. Six tools: read, write, edit (exact-match replace), bash (zsh, with timeout caps), plus two independently toggleable additions — recall_tool_output (recall an earlier tool call's output from the session journal after compaction dropped it) and record_lesson (report a cross-cutting finding — see §9). A turn with no tool call ends the session; the recording proxy's budget-exhaustion refusal ends it with rc=3; a hard 400-turn cap is the backstop.
agent/PiAgent.java — drives an external agent subprocess, owning its lifecycle (start, HOME seeding, process-tree kill on stop) and folding its output into the same journal/session machinery.

Context compaction

CompactionStrategies.apply() runs whichever strategies are enabled, in order, once the last prompt exceeds the configured compaction_strategies trigger. Four mechanisms exist (CompactionStrategies.KNOWN); three are on by default:

Strategy	Default	What it does
`tool_result_mask`	on	Stubs the oldest tool outputs once they fall outside a token-budgeted protected window; also stubs a `write` call's own full-file argument once its result is masked.
`error_line_extraction`	on	When a `bash` output's head/tail cap would let the omitted middle go dark, scans it for FAILED/Exception/assertion-diff/stack-frame lines and keeps the newest matches verbatim.
`tool_output_recall`	on	Makes `recall_tool_output` a real behavioural change — without it, a masked result's stub text just says "re-run the command" instead of pointing at a tool the model can call.
`structured_eviction`	opt-in	A one-shot LLM summary (LANDED/STATE/INVARIANTS/TASKS/DO NEXT/DECISIONS) of the evicted range, updated in place on later passes — the same shape as the inter-task handoff notes, applied within one session.

5. Context-window probing — `ContextProbe`

runner/ContextProbe.java is step 0 of a benchmark run against a local model: it measures the context window the current setup actually offers (min(server cap, memory), neither of which is knowable from config alone) by sending real streamed chat-completion requests of growing length (each a prefix-extension of the last, so the prefix cache keeps every step cheap), refining near the first failure, and recording prefill/decode rate at every step. It owns the server's max_model_len for the duration of the probe (raising it before measuring, restoring it after), and caches a result keyed by checkpoint + settings + server version + host. The derived usable_context number then flows directly into Packs.setWindow()/setScale() and the run's token budgets.

This step is oMLX-specific — prepareCap()/finalizeCap()/restartServer() shell out to a local ~/.omlx/bin/omlx binary and rewrite ~/.omlx/model_settings.json; against a remote provider there is no such binary, so these become no-ops, and the probe is pointless anyway (it would just spend a remote provider's paid tokens to rediscover a window size the operator already knows). --no-context-probe plus an explicit --context-window=<N> is how every job against an external provider is run — see User Manual: Jobs.

6. The oracle — scoring without trusting the agent

oracle/RunOracle.java is the single entry point (score(ws, task, systemBaseUrl, manifest)):

Looks up the task's rung in tasks/ladder.json — an unrecognized task throws, it never silently falls back to the full 77-point denominator.
Runs every offline check unconditionally (S1–S9, M1–M4, P1–P3) — no Docker needed. These are file-existence/size checks, directory-structure checks, and regex/text greps over (comment-stripped) source — none of S1–S9/M1–M4 does an AST parse.
Probes Docker availability once (docker info), as a single gate for every Docker-dependent check, rather than gating separately per script.
If Docker is reachable: runs BuildChecks (B1/B2/B3) in a pinned JDK container with a real PostgreSQL sidecar, then — if any wanted id is C1/C2/F* — ComposeChecks (C1/C2 plus the black-box F-suite) against the agent's own docker compose-built stack.
Any id that's still missing after that is SKIPPED ("docker unavailable") — excluded from the denominator but still blocking the headline weighted_score_pct/functional_score_pct (see oracle/CheckStatus.java). A checker that crashes fails every id it owns; it can never shrink the denominator by omission.

Docker-outside-of-Docker (DooD)

ace-service itself runs as a container (per docker-compose.yml) but never runs its own Docker daemon. It reaches the host's real daemon by bind-mounting the host socket:

# docker-compose.yml
- /var/run/docker.sock:/var/run/docker.sock

with the container's user added to the socket's GID via group_add: ["${DOCKER_GID}"]. This is why a managed container runtime (ECS/Fargate/App Runner) doesn't work as a deploy target — see DEPLOY.md. From inside that container, docker/DockerService.java shells out to the real docker/docker compose CLI on PATH (reached through the ordinary scrubbed-environment PATH, no shim) to:

B1 — gradle assemble inside a pinned gradle:8.14-jdk21 image, via docker run with the scratch-copied workspace and a shared Gradle-cache volume bind-mounted in.
B2 — gradle test the same way, then verify results by parsing the real JUnit XML (**/build/test-results/**/*.xml) rather than trusting Gradle's exit code — a gradle test that exits 0 having run zero tests is not evidence of anything.
B3 — a seeded mutation test: one of five syntactic mutations (e.g. .subtract( → .add() applied to the arithmetic nearest a SELL/point-in-time-replay token in the agent's own source, requiring the agent's own test suite to catch it (a passing suite under a broken invariant means B3 fails).
C1/C2 — docker compose build/up -d the agent's stack (from a source-only scratch copy — a jar built on the host could never legitimately ship in an image built from source), poll service health, then an HTTP GET /health against it.
F1–F9 — oracle/checks/BlackboxScenarios.java drives real HTTP scenarios against the agent's live stack: buy/sell symmetry, point-in-time queries, insufficient-balance 422, illegal-transition 409, exact decimal round-trips, OpenAPI conformance, HALF_EVEN deposit rounding, an exclusive asOf boundary, and an idempotency-key repeat. Every scenario creates its own account (so scenarios are statistically independent) and reads time from the server's own response timestamp, never the oracle's clock.

docker_shim.sh — an older script that auto-started Docker Desktop on the agent's first docker call — was removed entirely in a commit that also dropped all Docker-lifecycle management from DockerService: ACE no longer starts or stops Docker Desktop itself, the operator owns that. DockerService today only probes whether Docker happens to be running and mirrors the Docker CLI plugins (e.g. docker compose) into the agent's fresh HOME.

Calibration

reference/ReferenceServer.java is a plain JDK HttpServer wrapping the same TradingService the real Spring controller serves, carrying nine seeded bugs (float, selladd, no409, no422, asym, halfup, inclusive, noidem, hollow). A dedicated BlackboxCalibrationTest proves, per bug, that exactly the scenario claiming to catch it fails, and that zero bugs means every scenario passes — this calibrates the oracle's own scenarios against themselves rather than scoring any agent.

7. Scoring

oracle/Scorer.java turns a collection of CheckResults into the headline numbers:

denom = Σ weight over every counted (non-SKIPPED/INFRA) check; got = Σ weight where PASS.
pct = 100 * got / denom, rounded with HALF_EVEN.
The same ratio restricted to the functional ids (F1–F9) gives functional_score_pct.
If any id was SKIPPED/INFRA, the run gets partial_score_pct/partial_functional_score_pct instead of the real weighted_score_pct/functional_score_pct — a partial run is never silently presented as a full score.

8. Judge — mid-task trajectory checkpointing

runner/Judge.java is a separate LLM session, run every N agent turns within a task (judge_every_n_turns, default 10), that reads the task's goal, budget, diff-so-far, and recent trajectory, and replies with exactly one verdict line (ON TRACK / OFF TRACK) plus guidance injected as the agent's next turn when off-track. Consecutive off-track verdicts tighten the check-in cadence; near budget exhaustion the judge switches to a "triage mode" notice. Every checkpoint is written to judge/<taskId>-<attempt>-turn<N>.md — this is what the UI's per-task Judge tab reads (see User Manual). judge_enabled defaults to on; task-scoped sessions only (plan/handoff/review sessions never run it).

9. Cross-task lessons — `lessons.md`

The per-task handoff mechanism only propagates a task's notes along its own dependency edges, so a host/toolchain quirk one task discovers has no channel to reach a later, unrelated task — every task was independently rediscovering the same facts. record_lesson(text) is offered to the main task session, parallel-wave sessions, and both handoff paths; RunBench.recordLesson() is the single canonical write path (Judge's own off-track recommendations and a budget-rescue reconnect's reason are recorded through it too) into one run-wide lessons.md, folded into every downstream task's pack. The file is size-capped (lessons_max_chars, default 6000); crossing the ceiling triggers a deterministic, zero-LLM dedup pass that keeps the most recent of any duplicate group — an LLM-based tier-2 compaction was deliberately deferred, since an LLM pass risks corrupting the exact kind of factual record this feature exists to make trustworthy.

10. Operator intervention and pause-after-task

runner/Intervene.java is an in-memory mailbox, keyed by (runId, taskId), letting an operator inject a live steering message into a running task's next turn — delivered through the same per-turn check Judge uses, but with no LLM call and no enablement gate. It isn't persisted, but delivery is durably logged into the task's own session JSONL. runner/PauseAfterTask.java is a graceful stop request: unlike the immediate /pause endpoint (which can discard a task's in-progress turns), it waits for the current task to finish, then calls the same JobQueue.pause() the immediate path uses. Both live on the Job-detail page's Live tab.

11. The provider registry

agent/ProviderStore.java backs the LLM Providers tab: a Provider record (name, providerKey, baseUrl, apiKey, defaultModel) persisted in a providers table (V7__add_providers.sql), seeded with five external rows (openai, openrouter, anthropic, gemini, nebius). The local oMLX server is not one of the seeded rows, but it's expected to be registered as an ordinary row too — ProviderModelPicker.java (the UI's cascading model-selection control) hardcodes the literal key "omlx" for it: a bare, unprefixed model string is treated as "omlx/" + itself (the same "no prefix means local" assumption the backend already made before the registry existed, preserved for pre-registry historical runs via V8__backfill_omlx_model_prefix.sql's one-time backfill), and if no provider row with key omlx exists, the UI's main model field has no way to pick a local model at all. Routing uses two distinct conventions by design:

route(raw) — the main coding-model field's convention: a bare string with no / is always a local model id; a prefix/rest string whose prefix matches a registered providerKey routes to that provider; any other slash-containing string (unknown prefix, or a local id that just happens to contain /) stays local. Never throws.
resolveModel(raw, localFallback) — the reviewer/judge/task-matcher slot's convention: blank resolves to the local fallback; a bare known provider key expands via its defaultModel; a bare unknown key, or one with no default model, throws — a clear operator error rather than a silent fallback.

This is also the registry Preflight and job submission consult when deciding which provider a --model string's prefix should be checked against — see User Manual: Providers.

12. Recording and metrics

proxy/RecordingProxy.java sits between the agent and the model server, journaling every request/response (SseAssembler.java handles streamed-response reassembly) — this is what makes every downstream claim ("what sampler parameters were actually sent," "how many tokens were generated") verifiable against a recorded fact rather than configuration. metrics/Collect.java, metrics/Trajectory.java, and metrics/StatsService.java compute cost/trajectory analysis and k-run statistics: bootCi() (bootstrap confidence intervals), permutationTest(), and requireMatchedBudgets() (harness-effect comparisons refuse unless budgets actually match, verified numerically).

13. The Vaadin front end

ace-ui-vaadin is a pure client of ace-service's REST API plus SSE for live state. 14 routed views under com.strgmai.ace.ui: RootView (redirects to runs), RunsView/RunDetailView, JobsView/JobNewView/JobDetailView, ExperimentsView/ExperimentNewView/ExperimentDetailView, GroupsView, CompareView, PreflightView, LlmProvidersView, FileViewerView. DiffView is not a route — it's the per-task dialog's diff-rendering helper, opened from JobDetailView rather than navigated to directly. Live state on the Jobs/Job-detail pages is SSE-driven (JobEventLoop.java, TaskTranscriptLoop.java), not polling. CompareView doesn't do a naive id-for-id task match between two independently-decomposed plans — runner/TaskMatcher.java is an LLM-based matcher that semantically aligns them first. Full route table, form fields, and tab-by-tab breakdown are in the User Manual.

14. Data flow, end to end

Operator (UI: New Job form)
   │  POST /api/jobs
   ▼
BenchController ──▶ JobCfgFactory (raw input → RunSpec) ──▶ JobQueue (persisted, queued)
   │
   ▼
WorkerService (picks up queued job; Preflight gate)
   │
   ▼
RunBench.runOnce()
   ├─ ContextProbe (local provider only) ──▶ Packs.setWindow()/setScale()
   ├─ Packs.stablePack() / taskPack()  ──▶  ReferenceAgent / PiAgent (LLM turns via RecordingProxy)
   │        │                                        │
   │        │                                        ├─ Judge (every N turns)
   │        │                                        ├─ Intervene (operator steering)
   │        │                                        └─ record_lesson ──▶ lessons.md
   │        ▼
   │   per-task git snapshot, session JSONL journal
   ▼
RunOracle.score() ──▶ StructureChecks / MoneySafetyChecks (offline)
                   └─ BuildChecks / ComposeChecks (Docker-gated, via DockerService DooD)
                        └─ BlackboxScenarios (HTTP against the agent's live compose stack)
   │
   ▼
Scorer ──▶ manifest / oracle / metrics / trajectory written to results/<run-id>/
   │
   ▼
ace-ui-vaadin (RunDetailView, GroupsView, CompareView, ExperimentDetailView) ── reads via REST/SSE

Navigation Menu

Repository navigation

Architecture

Architecture

1. The two modules

2. What is being measured

3. Job queue and worker model

4. The runner — `RunBench`

Context compaction

5. Context-window probing — `ContextProbe`

6. The oracle — scoring without trusting the agent

Docker-outside-of-Docker (DooD)

Calibration

7. Scoring

8. Judge — mid-task trajectory checkpointing

9. Cross-task lessons — `lessons.md`

10. Operator intervention and pause-after-task

11. The provider registry

12. Recording and metrics

13. The Vaadin front end

14. Data flow, end to end

See also

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Architecture

Architecture

1. The two modules

2. What is being measured

3. Job queue and worker model

4. The runner — RunBench

Context compaction

5. Context-window probing — ContextProbe

6. The oracle — scoring without trusting the agent

Docker-outside-of-Docker (DooD)

Calibration

7. Scoring

8. Judge — mid-task trajectory checkpointing

9. Cross-task lessons — lessons.md

10. Operator intervention and pause-after-task

11. The provider registry

12. Recording and metrics

13. The Vaadin front end

14. Data flow, end to end

See also

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

4. The runner — `RunBench`

5. Context-window probing — `ContextProbe`

9. Cross-task lessons — `lessons.md`