Repository navigation
Design Decisions
This page collects the why behind ACE's non-obvious choices — mined from the codebase's own
explanatory comments (this codebase has a strong convention of "why, not what" comments at exactly the
places a reader would otherwise be confused) and from
docs/DESIGN.md.
See Architecture for how these decisions are implemented.
docs/DESIGN.md §18 lays out 18 foundational design rules as "methodology, not code" — the
standard ACE's implementation is held to. They're worth stating up front because almost
every decision below is one of these rules applied to a specific piece of code:
- Fix the denominator; absence is failure; a crash can never raise a score; partial runs are never quoted.
- The only correctness evidence is an oracle-owned test against the running system; self-authored green is not evidence; mutation-test the suite; freeze the contract or there is nothing to test against.
- Build a positive control before trusting a negative gate: it finds the oracle's own bugs.
- Separate infrastructure failure from agent failure, and never charge the machine to the model.
- Pin what you claim: sampler parameters via the proxy, packages by version, prompts by hash; record what was sent, not what was configured.
- Kill process trees, mark run processes in the environment, never read a child's stdout through a pipe its grandchildren can inherit.
- Compose the agent's context; do not let it accrete. Stable prefix first, per-task pack next, reachable stop conditions, a harness-owned status trail.
- One run is one draw: k ≥ 5, CIs, pass^k, and never pool runs whose harness, budgets, model, oracle or contract differ.
- Every state-changing scenario must assert the intermediate state, or a hollow server passes; a mutation test needs a green suite, or a red suite "catches" everything.
- Match total budgets before claiming a harness effect; a session that died on a stop-policy artefact is not an observation.
- Trust harness-verified completion, never the model's own "done".
- A self-review is a self-report: score it by calibration against an oracle or an objective index, never add it to the correctness score; an independent reviewer's judgment may count directly, with its identity on the record.
- Build the runtime under test from a clean checkout — anything the agent left in its working tree is not the code — and keep build outputs out of the snapshots too.
- Pin the toolchain the agent and the harness share (JDK, Gradle) and prove it with a real smoke test before the clock starts; a verification that could not run is inconclusive, never the agent's RED.
- Censor both arms of a comparison the same way; the runner is part of the treatment, so its hash is part of the comparability key; a matched budget is checked numerically, not assumed.
- Score every step, not just the end state — but attribute honestly and measure a step in isolation from a fixed prior state before choosing a model for it.
- Measure the context window the setup actually offers before anything else, and derive every context-dependent knob from that measurement, never from a number typed into a config.
- When the grader's contract changes, the specification the agent reads changes in the same commit — and the positive control must be able to fail the new rule, or add a fixture that does.
A single Docker gate, not per-script gating. The oracle probes Docker availability once
(docker info) and shares the result across every Docker-dependent check, rather than gating
separately per script. Simpler, and avoids five checks disagreeing about whether infrastructure is up.
SKIPPED/INFRA exist specifically so infrastructure is never charged to the agent (rule 4). Both are
excluded from the scoring denominator but still block the headline weighted_score_pct — a run
that couldn't reach Docker is never allowed to present as a clean 100%, but it also isn't punished with
zeroes for checks the machine, not the model, failed to run. NOT_ATTEMPTED (a feature simply absent)
is deliberately not given the same treatment — it counts as a real FAIL, because "the agent never
tried" is not infrastructure's fault.
Missing-record-is-FAIL, twice over. RunOracle synthesizes a FAIL for any wanted id a checker
didn't emit at all, and CheckStatus.parse() fails closed on a null/unparseable status. Both exist
for the same reason: a checker that silently produces nothing must never be able to raise a score by
omission (rule 1).
JUnit-XML verification instead of trusting Gradle's exit code. gradle test exiting 0 having run
zero tests is not evidence of anything — B2 parses the real **/build/test-results/**/*.xml output
and requires executed > 0, failed == 0, and no neutered/disabled test task.
Mutation testing targets the arithmetic nearest a SELL token specifically, not a random line in
the codebase — candidates are ranked by proximity to a SELL/BUY/point-in-time-replay anchor within
a fixed character window. A random mutation anywhere in the codebase would mostly land on code no test
exercises at all; anchoring near the exact invariant the task is supposed to get right makes B3 a real
test of whether the agent's own suite would catch a broken money calculation.
The compose stack is built from a source-only copy, specifically so a host-built jar can never ship
in an image "built from source." Two anti-gaming guards run before any build attempt: Dockerfiles
may not COPY/ADD a prebuilt artifact without --from= (which would legitimately be a multi-stage
build), and compose files may not use absolute context/mount paths that would let the stack escape the
source-only copy.
F6 must run strictly after F7/F8/F9, not in enum order — found live as bug #281. F6 grades every
HTTP exchange observed so far against the contract; dispatching it before the other scenarios had made
their calls meant it graded an incomplete exchange set, silently missing the very response shapes
(like the idempotency-key-repeat's second 200) those later scenarios are the only ones that ever
produce.
Every black-box scenario creates its own fresh account and reads time from the server's own
response, never the oracle's wall clock. Independent scenarios make pass^k statistically
meaningful (rule 8); server-sourced timestamps mean a drifting oracle clock can never fail a correct
implementation.
Budget exhaustion is a result, not an invalidation. A run that ran out of budget is recorded and scored, not thrown away — the same "censor both arms the same way" rule (rule 15) that governs harness-effect comparisons: if budget exhaustion silently invalidated a run, an experiment comparing two configurations could end up quietly dropping exactly the runs where the weaker configuration struggled, biasing the comparison.
Compare refuses a harness-effect comparison unless the two groups' budgets match — numerically
checked, not assumed (rule 15, rule 10). The UI's "Allow budget mismatch" checkbox exists precisely
because this is sometimes a deliberate, acknowledged departure from the default — not because the
check itself is soft.
TaskMatcher exists because two independently-decomposed plans don't share task ids. Comparing two
--plan-source=agent runs task-by-task would otherwise require the exact same granularity and
numbering, which two separate planning passes have no reason to produce — so an LLM-based semantic
matcher aligns "the same goal, different task id/granularity" before any task-level comparison can
happen at all.
BudgetExhausted is matched against a literal string in the proxy's own error body, not by HTTP
status code alone, because a real upstream 429 and the recording proxy's own budget refusal are
otherwise indistinguishable over HTTP — treating every 429 as budget exhaustion would misclassify a
genuine rate limit from the model server as the harness's own budget policy.
Budget-exhaustion rescue is a reconnect with trimmed history, not a cold restart or a raw
continuation. When a task's session ends via BudgetExhausted, ContextWindowExceeded, a task-wall
timeout, or an unresolved "unfinished" status, the opt-in rescue step (cfg "budget_rescue"."enabled",
same never-silent-default philosophy as Judge) gives it exactly one more dedicated session on its own
separate budget before accepting that final state — but deliberately neither of the two obvious
designs: not a cold continueSession=true reconnect to the full raw history (that's the task with the
largest accumulated history in the whole run — a known stall mechanism on its own), and not a fresh,
trajectory-free pack either (a live judge checkpoint once had to stop a session oscillating on the same
fix four times in a row; starting over loses exactly the context that would prevent repeating it). The
middle ground: reconnect, but compact the loaded history down to its most recent turns first, and name
the harness's own just-computed verification output explicitly in the instruction — the model's last
tool call before running out of budget was almost certainly not "run the tests." Scoped to the ref
harness only; PiAgent has no in-process turn loop to hook a cold reconnect into.
Compaction is token-budgeted, not turn-count-only, and the protected window itself is capped. The
default tool_result_mask strategy was found live (#325) to also need to mask a write call's own
full-file argument, not just its result — the argument is often the larger of the two and was
previously never compacted at all, silently defeating the point of masking in exactly the case where
it mattered most.
Judge and lesson-recording are both deliberately not members of CompactionStrategies, despite
lesson-recording briefly living there during its own first implementation — found not worth the
conceptual mixing it caused. Compaction strategies decide what to drop from context; Judge and
lesson-recording decide what the agent does and reports. Collapsing them into one toggle list made the
distinction harder to reason about for no real benefit.
lessons.md's dedup pass is deterministic, not LLM-based, on purpose. An LLM-based tier-2
compaction (for when simple dedup isn't enough) is a known, deliberately deferred gap — because an LLM
pass risks corrupting the exact kind of factual, cross-task record this feature exists to make
trustworthy. A dynamically-populated, run-scoped lessons.md is also kept entirely separate from the
static, operator-curated KNOWN_GOTCHAS list baked into every run's stable pack — one is live
operational memory for this run, the other is accumulated institutional knowledge that ships from day
one.
Handoff notes are scoped to a task's own declared dependencies, with a hard cap on their total
size — found live as two separate bugs (#304, #309). Without dependency scoping, an early,
size-heavy, unrelated task's handoff note could starve out the notes a later task actually depends
on, just because it happened to come first in plan order. Without the total-size cap, the final
INTEGRATION task (which has no declared dependencies, so it legitimately needs every note) could
receive an unbounded sum of individually-capped notes.
structured_eviction's own one-shot summarization prompt is appended as the last user message,
not prepended as a second system message. This followed a specific finding (tracked in issue #425):
under KV-cache compression, instruction order is one of three factors (alongside compression method
and eviction bias) that determines whether an instruction gets silently ignored, per "The Pitfalls of
KV Cache Compression" (arXiv 2510.00231). Placing the eviction instruction closest to generation,
rather than ahead of the agent's real system prompt, is a concrete mitigation for a documented failure
mode — one the codebase explicitly flags as a lever worth re-checking if ACE ever adopts KV-cache-level
eviction (H2O/StreamingLLM/PyramidKV) inside oMLX itself, which is out of scope today since that's a
third-party app's own internals, not something this codebase controls.
Two harnesses exist side by side — ReferenceAgent and PiAgent. ReferenceAgent exists
specifically to control reasoning effort, compaction behaviour, and loop version, not to replace
PiAgent: both are constructor-injected into RunBench and selected per run via --harness=.
ACE no longer starts or stops Docker Desktop itself — the operator owns that entirely. An earlier
on-demand Docker-lifecycle shim (docker_shim.sh) that auto-started Docker Desktop on the agent's
first docker call, plus an idle-teardown window monitor, were both removed: the idle-teardown was
found to be shutting Docker down mid-operation, and the whole up-down dance around the oracle's own
build phase was fighting the operator's own usage rather than helping it. DockerService today is
deliberately passive: it only probes whether Docker happens to be running and mirrors CLI plugins into
a fresh HOME.
Docker-outside-of-Docker, not Docker-in-Docker, and not a managed container runtime. ace-service
reaches the host's real Docker daemon by bind-mounting its socket, rather than running a nested
daemon inside its own container. This is also the reason DEPLOY.md rules out ECS/Fargate/App
Runner-style managed container services as a deployment target — they don't expose a usable host
socket, and the build/compose oracle checks have no other way to drive a real docker/docker compose
from inside a containerized ace-service.
The operational SQLite store moved off PostgreSQL partway through development. ace-service's own
job/provider/experiment state now lives in a local, Flyway-migrated SQLite database queried through
jOOQ — no external DB server or Testcontainers container backs it anymore. PostgreSQL still appears
elsewhere (as the sidecar the build-check container talks to, and inside the agent's own generated
compose stack), but ACE's own bookkeeping no longer needs an external database server to run at all,
simplifying both local development and the single-VM deployment story.
The provider registry replaced a hardcoded ProviderConfig enum with an operator-editable database
table specifically so adding a new provider, or rotating a key, doesn't require a code change and a
redeploy. The migration comment is explicit about the resulting risk model: api_key stores the raw
secret value directly, not an env-var name, "since this is a single-operator local tool (same risk
class as today's .env/shell-profile convention), not a shared multi-tenant service needing key-vault
machinery" — a deliberate, stated trade-off rather than an oversight.
Two different routing conventions exist on purpose, not by accident. The main coding-model field's
route() is tolerant by construction and never throws — a local model id that happens to contain a
/ (e.g. an org/Model-name-style id) must stay local, not get misinterpreted as a provider prefix,
whether it arrives through the UI's picker, the REST API directly, or a historical run's own recorded
string. The
reviewer/judge/task-matcher slot's resolveModel() is the opposite: a bare, unrecognized provider
key throws a clear operator error rather than silently falling back to something the operator didn't
ask for — those slots have no "custom value" tradition to preserve, so failing loudly is strictly
better than guessing.
ContextProbe is deliberately oMLX-specific and designed to no-op gracefully everywhere else,
rather than being generalized to probe arbitrary remote endpoints. Probing a remote provider would mean
spending that provider's paid API traffic purely to rediscover a context-window size the operator
already knows and can simply pin with --context-window; the probe's entire mechanism (rewriting
~/.omlx/model_settings.json, restarting a local omlx server process) only makes sense against a
model server ACE itself can control.
A plain VM, not a managed container service — driven by the same two hard requirements that recur
throughout this document: ace-service needs the host's own Docker socket for Docker-outside-of-Docker
(no managed runtime exposes one), and everything that should survive a redeploy (the SQLite DB, run
results, workspaces) needs real persistent disk, not ephemeral container storage. See
DEPLOY.md for the
full provisioning walkthrough.
No CI is configured in this repository, and that is a stated, known limitation, not an oversight.
The gaming-resistance fixture suite (positive control + adversarial cheats + SelftestIT) and the
Docker-gated integration tests are real and present, but nothing re-runs them automatically — they only
run when a developer runs the test suite locally with Docker up.
- Architecture — the implementation these decisions shaped
- Features — the resulting capability list
- docs/DESIGN.md — the fuller design reference