Skip to content

Features

Jump to bottom
Andrei Andriiuk edited this page Oct 27, 2026 · 5 revisions

Features

A comprehensive list of what ACE actually does, grounded in the source. See Architecture for how these pieces fit together, and the User Manual for how to use each one from the UI.

Scoring: 30 checks, 6 categories, fixed denominator

oracle/CheckId.java is the single source of truth: every run must produce a record for every applicable check id — a missing record is scored as FAIL, never silently dropped.

Category ids weight each What's checked
Structure S1–S9 1 task docs present, compose file present, ≥3 real Gradle services, real Gradle wrapper, parseable OpenAPI spec, React in package.json dependencies, DB migrations present
Planning P1–P3 2 definition sections present; subtasks carry id/goal/deps/criterion; no source written during phase 0/1
Lint M1–M4 1 money never float/double (Java/Kotlin/SQL); BigDecimal imported where money is handled; no .equals() on money; point-in-time replay correctly subtracts sells
Build B1–B3 3 compiles in the pinned JDK container; >0 tests executed and passing (JUnit-XML verified, not just exit code); the agent's own suite fails on a seeded mutation
Runtime C1, C2 3, 2 ≥3 non-DB services report healthy; the declared readiness endpoint returns 200
Functional F1–F6 5,5,5,5,5,4 buy/sell symmetry; point-in-time query; insufficient-balance → 422; illegal-transition → 409; exact decimal round-trip; responses conform to the OpenAPI contract
Bespoke F7–F9 5 each HALF_EVEN deposit rounding; exclusive asOf boundary with echo; Idempotency-Key repeat → same order, 200, no re-execution

See the Check Registry for every id's own detailed mechanics, grounded in its checker's own implementation. Total weight: 77. CheckStatus has five values: PASS, FAIL, NOT_ATTEMPTED (feature simply absent — still a FAIL), SKIPPED (infra unavailable before the check ran), INFRA (infra failed mid-check). SKIPPED/INFRA are excluded from the denominator but still block a full-score headline — a run missing Docker never gets to claim a clean 100%, it gets a partial_* score instead.

The task ladder

tasks/ladder.json defines graduated rungs, each a strict subset of the full registry — e.g. L1_migration_entity (denominator 11, no C1/C2 compose/runtime checks — though it still includes B1/B2, which run inside Docker via Docker-outside-of-Docker), L3p_point_in_time (plan-first variant, denominator 36), up to L7_full_platform (the full 77). A rung also fixes the task prompt text and a default wall-clock budget — a per-job parameter, overridable from the New Job form, not a cap baked into the rung. Choosing a rung there immediately shows its default budget and check list (see User Manual: Jobs). See the Task Ladder for what every rung actually asks the agent to build.

Build checks, made evidential

oracle/checks/BuildChecks.java never trusts the agent's own Gradle wrapper or exit code:

  • B1 — every discovered Gradle root is built with gradle assemble inside a pinned gradle:8.14-jdk21 container image, via Docker-outside-of-Docker (see Architecture §6).
  • B2 — gradle test, verified by parsing the real JUnit XML output (executed > 0, failed == 0, no neutered/disabled test task) — a green exit code with zero tests executed is not evidence.
  • B3 — a seeded mutation test: one of five syntactic mutations (.subtract( → .add(, a dropped unary minus, -= → += on a money line, etc.) is applied to the real arithmetic nearest a SELL/point-in-time-replay token, and the agent's own test suite must fail under it. If no such arithmetic exists to mutate, B3 is NOT_ATTEMPTED, not FAIL.

Compose/runtime checks

oracle/checks/ComposeChecks.java builds and runs the agent's own docker-compose.yml from a source-only scratch copy (build artifacts stripped — a host-built jar can never legitimately ship in an image built from source), with two anti-gaming guards checked before any build even starts: Dockerfiles may not COPY/ADD a prebuilt artifact without --from=, and compose files may not use absolute context/mount paths that would escape the source-only copy. C1 counts non-DB services reporting healthy (≥3 required); C2 polls the declared readiness endpoint over real HTTP.

The black-box functional suite

oracle/checks/BlackboxScenarios.java is, in its own words, "THE correctness score" — real HTTP scenarios driven against the agent's live stack, each creating its own account (so pass^k across scenarios is statistically meaningful) and reading time from the server's own response timestamp (never the oracle's clock, so clock drift can't fail a correct implementation). F6 grades every response observed by F1–F5/F7–F9 against both the frozen contract and the agent's own shipped OpenAPI spec — it always runs last, specifically so it grades a complete set of exchanges.

Gaming resistance

Three legs, all present:

  1. Fixed-denominator scoring (above) — absence is failure, a crash can never raise a score.
  2. A positive control (ace-service/fixtures/positive/) — a real, substantive known-good solution that must score ≥95% on every required check, used to find the oracle's own bugs.
  3. Adversarial "cheat" fixtures (ace-service/fixtures/adversarial/cheat{A-F}_*) — six fixtures, each named for the specific gaming pattern it exercises (comment-only changes, bugs hidden in the test package, renamed fields, a directory merely named java, stub implementations, a prebuilt jar substituted for source), each asserted to stay below 70% and/or trip the specific check it targets.

SelftestIT.java is the two-sided gate running both legs 2 and 3. There is no CI configured in this repository, so this only runs when a developer runs the Docker-gated test suite locally.

The provider registry and model routing

The LLM Providers tab (LlmProvidersView, backed by agent/ProviderStore.java) is an operator-editable list of OpenAI-wire-compatible providers — name, the stable provider/model routing prefix, base URL, API key, and an optional default model. Seeded with five external rows (OpenAI, OpenRouter, Anthropic, Gemini, Nebius); only Nebius ships a seeded default model (zai-org/GLM-5.3-Flash). The local oMLX server isn't one of the seeded rows, but it's expected to be registered as an ordinary one too, under the literal key omlx — the main model picker hardcodes that key to resolve a bare, unprefixed model string as local. See Architecture §11 for the routing rules and User Manual: Providers for the UI.

Preflight

A one-click readiness check (service/Preflight.java, UI: PreflightView) that runs a real gradle test against the positive-control fixture in the agent's actual environment before any benchmark run is trusted. It reports per-check badges (ok / degraded / fatal) and an overall runnable/blocked verdict. A job's own --model (possibly provider-prefixed) is checked against whichever provider it's prefixed with — the generic Preflight view (no model selected) showing the local model-server check as "unavailable" on a remote-only deployment is expected and harmless; it doesn't block a job targeting a configured external provider.

Judge — mid-task trajectory checkpointing

A separate LLM session, run every N turns within a task (default 10), that reads the task's goal, budget, diff-so-far, and recent trajectory, and returns ON TRACK/OFF TRACK plus guidance injected as the agent's next instruction when drifting. Consecutive off-track verdicts tighten the check-in cadence automatically; near budget exhaustion the judge switches to a "triage mode" notice. Every checkpoint is visible per-task in the UI's Judge tab. On by default. See Architecture §8.

Cross-task lessons (lessons.md)

A tool (record_lesson) the agent can call at any point to report a cross-cutting finding — a host/toolchain quirk, a discovered constraint — that should reach every later task, not just ones that happen to depend on the task that found it. Judge's own off-track recommendations and a budget-rescue reconnect's reason are recorded through the same path. Size-capped with deterministic dedup once the cap is crossed. On by default. See Architecture §9.

Budget-exhaustion rescue

An opt-in second chance for a task whose session ended via budget exhaustion, a context-window overflow, a task-wall timeout, or an unresolved status: one more dedicated session, on its own separate budget, before that final state is accepted — a trimmed-history reconnect, not a cold restart or a raw continuation. Configurable (own wall-clock/token budgets, how many recent turns survive the trim) and scoped to the ref harness. See Design Decisions for why it's built this specific way, and User Manual: Jobs for the toggle.

Operator intervention and pause-after-task

Two live controls on a running job's Live tab:

  • Intervene — inject a steering message directly into a task's next agent turn, at any time, regardless of that task's own Judge setting. Not persisted as state, but durably logged into the session transcript.
  • Pause after task — a graceful stop request that waits for the currently-running task to finish before pausing, as opposed to the immediate /pause action which can discard in-progress turns.

Context compaction strategies

Four pluggable mechanisms (CompactionStrategies), configurable and reorderable per job — see Architecture §4 for what each one does: tool_result_mask, error_line_extraction, and tool_output_recall are on by default; structured_eviction is opt-in.

Granularity and parallel planning

  • Task granularity (fine/medium/coarse) controls how finely the orchestrated plan splits work into tasks for a given rung.
  • Parallel planning lets independent tasks run as concurrent waves feeding into a shared INTEGRATION task, with merge-conflict handling in RunBench and a dedicated parallelization weight knob controlling how aggressively the planner parallelizes.

Groups — the pooled leaderboard

GroupsView pools comparable runs by (task, model, key_hash) — the "key" is a hash of the run's configuration (see StatsService.KEY_FIELDS), not its mode, which is read off the group for display only, not used to decide what pools together — and reports ranked groups (k ≥ 5 comparable valid runs) separately from indicative ones (k < 5, never ranked). Each group card shows functional/composite/agent-result means with 90% confidence intervals, a per-task pass/fail chip matrix, and the excluded-run breakdown (invalid / partial due to Docker-skip-or-infra).

Compare

Two distinct tools on one page (CompareView):

  1. Statistical A/B comparison — pick two groups of run ids, a metric, and comparability flags (allow partial runs, include invalid runs, compare two different models, allow budget mismatch), and get StatsService's verdict with a bootstrap confidence interval and permutation-test p-value. By default it refuses unless the two groups' budgets actually match numerically — the "censor both arms the same way" rule.
  2. Task matching — runner/TaskMatcher.java is an LLM-based matcher that semantically aligns two independently-decomposed --plan-source=agent plans (same goal, different task ids/granularity) before any task-by-task comparison is possible at all.

Experiments

A templated batch of jobs run k times per arm, with an automatic comparison verdict once every job finishes (ExperimentsService, UI: ExperimentsView/ExperimentNewView/ExperimentDetailView). Three templates:

  • harness_effect — one model across multiple harness arms (orchestrated / monolithic / monolithic+rules / parallel), same budgets.
  • model_ab — two models, same harness and budgets.
  • agent_ab — the reference agent vs. Pi, same model and budgets.

Runs — import and browse

RunsView/RunDetailView browse already-produced results/<run-id>/ directories — either freshly produced by a job, or imported via Rescan results/. A run detail page shows the checks table, plan tasks, per-step scores, a speed-vs-context-size chart, a compaction-overhead-over-time chart, full provenance (model, quantization, harness/oracle SHAs, measured context window), and an embedded file browser over the run's own results directory, including a standalone File Viewer for any individual file (Markdown rendered, JSON/JSONL pretty-printed) and a colorized unified-diff Diff View for any task's before/after changes.

See also

Clone this wiki locally