Repository navigation
Task Ladder
tasks/ladder.json is the single source of truth for every task "rung" ACE can run — eight of them
(L1_migration_entity … L7_full_platform, plus the plan-first variant L3p_point_in_time), each
fixing four things at once: the task prompt text the agent actually reads, a default wall-clock
budget, the exact subset of oracle checks it's scored on (every id below links to its own entry in the
Check Registry), and (so) a fixed scoring denominator. Choosing a rung in the New
Job form shows all four immediately — see User Manual: Jobs; this page is the
detailed reference. oracle/RunOracle.java looks a task's rung up by exact name and throws on
anything unrecognized — there is no silent fallback to the full 77-point denominator (see
Architecture §6).
A rung's budget is a default, not a hard limit. JobCfgFactory only writes task_wall_sec (and
every other budget field — implementation, plan, handoff, wrap-up, parallel-plan, or a global
override) into a job's own config when the matching --task-wall/etc. flag is actually present on
argv; left blank, RunBench.rungBudgets() falls back to the rung's own budget_sec from
tasks/ladder.json as the default split between plan and implementation time. Every budget below is
therefore a per-job parameter an operator can override from the New Job form (or per already-run
task, via that task's own Restart tab) — never a cap baked into the rung itself. Only the check
subset and denominator are genuinely fixed per rung.
Every rung is a slice of the same frozen scenario: a small trading platform (accounts, deposits,
orders, point-in-time holdings) built in Java 21 / Spring Boot 3 / Gradle, PostgreSQL via Flyway,
decimal values as JSON strings (never numbers), timestamps as ISO-8601 UTC with millisecond
precision. L7_full_platform is the one rung with no trimmed prompt of its own — it falls back to
tasks/PROMPT.md, the full, untrimmed spec (microservices, a React frontend, OpenAPI 3.1, the
complete frozen HTTP contract) that every other rung's own tasks/<rung>/PROMPT.md is a deliberately
narrower rewrite of. Keeping one domain across every rung, rather than a different toy problem per
difficulty level, is what makes a model's score on L2 and its score on L5 comparable at all.
Every rung's prompt — L7's own tasks/PROMPT.md included — carries the same non-negotiable rules:
money is BigDecimal or integer minor units, never float/double; gradle test must run in a clean
JDK 21 container against a fresh PostgreSQL reachable through SPRING_DATASOURCE_* env vars, needing
neither Docker nor the network. L1–L6 and L3p additionally repeat one line verbatim, byte for
byte, across every one of their own prompt files: "Do not edit anything under task/. This rung
starts from an empty workspace: everything it names must be built here." — L7's own prompt doesn't
carry that exact sentence, but starts from an equally empty workspace in practice; it's simply the one
rung with no smaller starting point to be explicit about.
| Rung | Default budget | Denominator | Checks | What it asks for |
|---|---|---|---|---|
L1_migration_entity |
45 min | 11 | 7 | Scaffold account-service from an empty workspace: a Flyway migration and a money-correct Account entity with non-negative balance constraints, proven by a repository test. |
L2_one_endpoint |
60 min | 27 | 11 | Expose a full account/deposit HTTP contract behind docker compose, backed by an OpenAPI spec and HALF_EVEN-rounded deposits. |
L3_point_in_time |
75 min | 34 | 11 | Add orders with immediate fills and point-in-time holdings queries with an exclusive asOf boundary, replaying sells correctly. |
L3p_point_in_time |
90 min | 36 | 12 | Same feature as L3, but run plan-first: write docs/IMPLEMENTATION_PLAN.md as 3–6 verifiable subtasks before implementation begins. |
L4_state_machine |
75 min | 24 | 7 | Implement the order state machine (NEW → FILLED/CANCELLED) with idempotent order creation via an Idempotency-Key header. |
L5_second_service |
90 min | 15 | 8 | Split the platform into three independently-built services (account, order, market) that talk to each other only over HTTP, each with its own schema. |
L6_compose_health |
60 min | 10 | 5 | Bring up three services under docker compose with real Docker healthchecks and dependency ordering, surviving a slow-starting Postgres. |
L7_full_platform |
4 h | 77 | all 30 | The full production trading platform end to end: microservices, a React frontend, an OpenAPI contract, and every correctness/money/state-machine rule, graded entirely from a clean checkout. |
("Checks" above counts oracle check ids, not weight — see each section below for the exact id list and Check Registry for what every id verifies and how much it's worth. "Default budget" is exactly that — overridable per job; see the note above.)
Checks: S6 S9 M1
M2 M3 B1 B2
— structure, lint, and build only; no C1/C2/F* ids at all. B1/B2 still run inside Docker (Docker-outside-of-Docker, same as every other rung), but
nothing is ever brought up as a running service — there's no compose file to check for, because this
rung never produces one.
Build account-service with a Flyway migration (V1__accounts.sql) and a JPA Account entity
(accountId UUID, currency CHAR(3), availableBalance/reservedBalance NUMERIC(19,4)), with
non-negative-balance constraints enforced at both the DB (a CHECK constraint) and the entity
level. A repository test must prove a negative balance is actually rejected, run against the real
provided PostgreSQL. This is the floor rung — it exists to isolate "can the agent stand up a correct
schema and entity at all" from everything that follows.
Checks: S4 S6 S7
M1 M2 B1 B2
C2 F5 F6 F7
— adds a compose file (S4), a parseable OpenAPI spec (S7), a live readiness check (C2), and the
first functional/bespoke checks.
L1's entity now sits behind three real HTTP endpoints, served via docker compose up on
localhost:8080: POST /accounts → 201, GET /accounts/{id} → 200/404, POST /accounts/{id}/deposits → 200/400, plus GET /health. An openapi.yaml must declare every path,
method, and status code above with no default responses (F6 grades the implementation against it).
Bespoke rule (F7): deposits round HALF_EVEN to 2 dp — "0.005" must read back as "0.00",
"0.015" as "0.02" (a tie-breaking rule most naive implementations get wrong by defaulting to
HALF_UP).
Checks: M1 M2 M4
B1 B2 B3 C2
F1 F2 F3 F8
— keeps L2's account/deposit base and adds orders, plus the first mutation-kill check (B3) and the
first real functional-correctness checks (F1–F3).
On top of L2's endpoints: POST /orders (immediate fill at limitPrice, debiting/crediting the
account) → 201 or 422 on insufficient funds/holdings, and GET /accounts/{id}/holdings?asOf=<instant>
→ 200. The replay must subtract sells (M4's own lint check, and the specific arithmetic B3's
seeded mutation targets) — a buy then an equal sell must restore balance and holdings exactly (F1),
and a point-in-time query between a buy and a later sell must show the position before the sell
(F2). Bespoke rule (F8): the asOf boundary is exclusive (an order executed at exactly
T is NOT included in asOf=T), and the response always echoes the instant actually applied as
asOfApplied, normalised to UTC milliseconds.
Checks: L3's own 11 ids, plus P2 (plan subtasks carry id/goal/deps/criterion)
— the one check L3p adds over L3, accounting for the entire +2 denominator difference.
Identical feature scope and bespoke rules to L3_point_in_time — same orders-and-holdings scenario,
same exclusive-asOf rule — but run through a plan-first protocol: the job's phases are
p1_plan then p2_implementation, and the prompt explicitly requires docs/IMPLEMENTATION_PLAN.md
(3–6 ordered subtasks, each with a goal, dependencies, and a verifiable acceptance criterion) to exist
before any implementation. Because L3 and L3p share every other check and the same underlying
feature, comparing a model's L3 run against its L3p run on the same model/budget is the cleanest
way to isolate the effect of an explicit planning step in this benchmark — everything else about the
task is held constant.
Checks: M1 B1 B2
C2 F3 F4 F9
— notably not a superset of L3: M4/F1/F2/F8 (the point-in-time-holdings checks) drop out
entirely, replaced by F4 (illegal state transitions) and F9 (idempotency). L4 is a sideways move
to a different feature of the same domain, not a deepening of L3.
Keeps accounts, deposits, and orders, but swaps point-in-time holdings out for a formal order state
machine: NEW → FILLED or NEW → CANCELLED, both terminal. POST /orders/{id}/cancel → 200 on a
NEW order, 409 on an already-terminal one — and because orders fill immediately in this
benchmark, cancelling a just-created order is always 409 (F4). Bespoke rule (F9): POST /orders accepts an optional Idempotency-Key header; a repeat with the same key within 60 s must
return the same orderId with 200 (not 201) and must not execute twice — the stored balance
after the duplicate must be identical to after the first call.
Checks: S4 S5 S6
S9 B1 B2 C1
C2 — structure, build, and runtime only. Zero functional/bespoke checks (no
F* ids at all): this rung scores genuine service decomposition, not feature correctness.
The first multi-service rung: three Spring Boot services — account-service, order-service (calls
account-service's own HTTP API to debit the account on an immediate fill), and market-service
(GET /prices/{symbol} against a fixed table: AAPL 10.00, TSLA 250.00, EURUSD 1.0850, 404
otherwise) — each with its own Gradle build (S5 requires ≥3 real Gradle services), its own
Postgres schema, and its own Flyway history, talking to each other only over HTTP, never sharing a
database. docker compose up brings up all three plus Postgres, each reporting healthy (C1
requires ≥3 non-DB services healthy).
Checks: S4 S5 B1
C1 C2 — the smallest denominator of any multi-service
rung, and the only rung with zero lint/functional checks of any kind. Purely a test of compose-level
orchestration.
Three minimal services this time (gateway, account-service, market-service, each exposing only
GET /health) with real Docker healthchecks and depends_on: condition: service_healthy ordering —
the services must survive PostgreSQL starting slowly (retry, don't crash-loop) rather than merely
start in the right order on a fast machine. The gateway's own /health on localhost:8080 must only
return 200 once its dependencies are actually up, not just once the gateway process itself is alive.
Checks: all — every one of the 30 registered checks, including S1–S3/S8
and all three P1–P3 planning checks, none of which any other rung scores at all (confirmed directly against
tasks/ladder.json: no L1–L6 rung's own checks array lists a single S1/S2/S3/S8/P1/P3
id — only L7 does, via "checks": "all", and only L3p separately opts into P2). docs/TASK_DEFINITION.md,
docs/IMPLEMENTATION_PLAN.md, and docs/PROGRESS.md existing at all (S1–S3) and React appearing
in the frontend's own dependencies (S8) are therefore scored only on this rung.
The untrimmed scenario: the full production trading platform end to end, run through all three
phases (p0_definition → p1_plan → p2_implementation...N) from a completely empty workspace —
every microservice, a React (TypeScript) frontend built and served as its own container, the complete
frozen HTTP contract (accounts, deposits, orders, cancellation, idempotency, point-in-time holdings,
the fixed price feed), and every money/state-machine/structure rule every smaller rung checks
individually. Packaging is graded from source: the grader runs docker compose build on a clean
checkout with no build//target//dist/ and no committed .jar/.class files, so every image must
build the application inside itself.
A rung fixes what gets built; task granularity (a separate job setting, only meaningful with an agent-authored plan) controls how finely that work gets split into individually-attempted tasks for the plan grid. It does not add or remove oracle checks — see Features: Granularity and parallel planning for how the setting works and how it interacts with parallel planning.
- Check Registry — what every check id verifies and how, grounded in each checker's own implementation
-
Features: Scoring — weights and how
CheckStatus(PASS/FAIL/NOT_ATTEMPTED/SKIPPED/INFRA) works - Architecture §2: What is being measured and §6: The oracle
- User Manual: Jobs — picking a rung, and overriding its default budgets, in the New Job form
- Design Decisions — the why behind several of ACE's other non-obvious choices