Frozen Binary Bench

Status: draft, 2026-10-29

1. Purpose

An agentic benchmark that measures how far a running application can be changed — its domain model, its UI layout, its access policy, its integrations — by an agent that may change data and nothing else. The code, the build and the process stay as they were when the run began.

The benchmark makes no assumption about which systems are data-driven. It probes the behaviours a system can still reach with its binary frozen, and scores whatever it finds.

2. Principle

Write a system's observable behaviour as B = f(c, d), where c is its code (image, build, configuration files baked into the deployment) and d its data (everything writable through its public interface while it runs).

The harness fixes c = c₀ for the whole run and proves it did (§4.1). The agent can then reach only

R(c₀) = { f(c₀, d) : d writable through the public interface }

Each ticket (§7) names a target behaviour b. A ticket passes when the agent finds a d with f(c₀, d) = b, as judged by the ticket's oracle. A system whose domain model lives in c₀ has a small R(c₀) for model changes; a system whose c₀ interprets a model held in d has a large one. The tiers are ordered so that R(c₀) has to grow along a different axis at each step.

The two outcomes carry different weight. A pass is a constructive proof that b ∈ R(c₀): the agent's writes are the witness d, and anyone can check it by replaying d onto a fresh c₀ (§4.7). A fail shows only that this agent did not find a d within its budget. It becomes a claim about the system (b ∉ R(c₀)) only with a structural argument, so every fail is classified (§8).

3. Subjects

Subject	What it is	The agent's interface
LDH	LinkedDataHub, current release, default compose stack	HTTPS + ldh CLI + browser
CRUD	A Django app the same agent model generates from the tier-0 brief before the run, then frozen	HTTPS + its admin UI + its REST API
Table	A hosted table/low-code tool (Airtable or Baserow, self-hosted for the invariant checks)	HTTPS + its REST API + browser

Every subject is seeded from the same dataset (§6) and gets the same agent model, the same ticket text and the same tool budget. The agent receives only the subject's own generic client: no shell on the host, no repository, no package manager, no container runtime.

4. Harness

Six components, each a separate process so a subject cannot fake one by failing another.

4.1 Invariant checker

Polls every 5 s and fails the run on the first violation:

• process identity — for each container of the subject: StartedAt, image digest and PID 1 unchanged (docker inspect);
• deployment files — a hash of every bind-mounted config file and of the image's application directory unchanged;
• availability — a health probe per public origin; any refused connection or 5xx is recorded (the traffic simulator measures the same thing under load, §4.2).

For LDH the containers are linkeddatahub, fuseki, nginx, the three varnish-* and egress; the hashed files are config/, platform/ and the src/ copy inside the image. Cache bans and ontology cache clears go over HTTP and touch no file, so they pass.

4.2 Traffic simulator

A Node script that keeps a steady load running from the seed onwards:

• 5 req/s reads (documents, search, SPARQL SELECT), 1 req/s writes (create/update a product or order using only the tier-0 fields);
• a separate low-privilege identity, so ACL tickets (§7, T8) change what it sees;
• every request is logged with the time, status and latency.

Outputs two numbers per ticket window: failed requests (5xx, refused, timeout) and lost writes (a write that returned 2xx but cannot be read back at the end of the run).

4.3 Ticket feed

Delivers the tickets of §7 to the agent one at a time, in order, as plain text in the persona's voice. The next ticket arrives when the agent declares the current one done or its budget runs out. Ticket text never contains the oracle's assertions or the cold-start questions (§7, T6).

4.4 Oracles

One per ticket, run after the agent declares done and again at the end of the run (a later ticket must not undo an earlier one unless it says so):

• HTTP oracle — black-box assertions over the subject's public API: the property exists, the migrated values are there, an unauthorised identity gets 403;
• UI oracle — a Playwright spec that drives the browser as a user would, finds fields by their label, and asserts what renders (for LDH, alongside tests/ui/specs/, reusing tests/ui/lib/).

Oracles are written per subject but assert the same behaviour; §8 records any oracle that had to differ.

4.5 Artifact classifier

Records every request the agent sends that writes something, and classifies the written payload:

Class	Examples
data	RDF triples, rows, records, field definitions
query	SPARQL, SQL, filter expressions
template	XSLT, Handlebars, any Turing-complete presentation code
script	JavaScript, Python, formula languages with side effects
reference	Installing an artifact that already existed before the run (a package from a registry)

A ticket passed with any authored template or script is reported as passed with code. Code is allowed; it is counted separately, because with it the "data" channel has become a deployment channel.

4.6 Recorder

Captures the agent's transcript, tool calls, tokens and wall-clock time per ticket, and writes results/<subject>/<run-id>.json (§8).

4.7 Replay certificate

Every passed ticket is published with a certificate that lets a third party check it without the agent:

• c₀ — the subject's image digests and deployment file hashes, as recorded by §4.1;
• seed — the seed id, so §6 regenerates the identical starting data;
• d — the ordered writes of this ticket and every ticket before it (a ticket builds on the earlier ones): method, URL, content type and body, with credentials removed;
• oracle — the version of the ticket's oracles;
• replay.mjs — deploys c₀, seeds it, sends d with the verifier's own credentials, runs the oracle and exits 0 on a pass.

Identifiers the server assigns (a Location from a create, a row id) differ between runs, so the recorder stores each one with the response that produced it, and replay.mjs substitutes the new value into every later request that used it.

A certificate proves the behaviour; the run's metrics stay the run's own. It is replayed without the traffic simulator, so time to effect, failed requests and lost writes come from the run alone. A certificate that does not replay to a pass downgrades its ticket to a fail of kind unknown.

5. Rules

1. The agent may use only the interface listed for its subject in §3.
2. No restart, redeploy, rebuild or file edit on the host, enforced by §4.1.
3. The agent may read any public documentation of the subject's platform; it may not read the seed scripts, the oracles or the harness.
4. A ticket's budget is 60 tool calls and 30 minutes. Exceeding it fails the ticket and moves on.
5. Each subject is run 5 times with different random seeds for the ticket wording paraphrase; the score is the mean with min/max.

6. Scenario and seed data

A small wholesale business: Products (name, SKU, price, category, supplier as a string), Orders (date, customer, lines), Customers (name, country). About 500 products, 2,000 orders, 150 customers, generated deterministically from a seed so every subject gets identical records. Two tenants, North and South, each with its own copy.

For LDH each tenant is a dataspace, provisioned during setup (it requires config/ edits and a restart, which is why it happens before c₀ is fixed). The tier-0 model is an ontology imported with ldh admin import ontology, the data pushed with ldh push.

7. Tickets

Each ticket lists the persona's request, what the oracle checks, and — for LDH only, as a reference solution the agent never sees — the primitives a pass should need.

T1 — Add a property

Sales: "We need to record each product's country of origin, and filter the catalogue by it."

• HTTP: products accept and return countryOfOrigin; existing products still validate.
• UI: the product form shows a Country of origin field; the catalogue filters by it; a product created before the change still opens and saves.
• LDH: ldh admin add property-constraint / an ontology edit, ldh admin clear ontology, a facet on the products view.

T2 — Promote a string to an entity

Ops: "Suppliers should be their own records with a contact email, and each product should link to one. Keep the existing assignments."

• HTTP: every distinct supplier string has become one Supplier record; every product links to the right one; no product lost its supplier; simulator writes during the migration are not lost.
• UI: a product page links to its supplier; a supplier page lists its products.
• LDH: ldh admin add class, a SPARQL update sent with ldh patch per graph (the platform's SPARQL endpoint is read-only), a backlinks view on the supplier.

T3 — Change the layout

Management: "Show suppliers on a map, products as a sortable table, and put a revenue-by-category chart on the orders page."

• UI: the three render as asked, with the data from T2.
• LDH: ldh add view, ldh add object-block, ldh add result-set-chart; the view mode follows the resources' properties.

T4 — Tenants diverge

North: "We call categories product lines now, and a product can belong to several." South: no change.

• HTTP/UI on North: the label and the cardinality changed, existing single values kept.
• HTTP/UI on South: identical to before T4, checked against a snapshot taken at the T4 start.
• LDH: edits to North's admin ontology only.

T5 — Roll back

North: "Undo the product-line change; we're going back to one category per product."

• North's model matches its pre-T4 snapshot; products given several lines after T4 remain readable and keep one of them, chosen by a rule the agent states; T1–T3 survive.
• LDH: the ontology document's Memento versions (ldh get --timemap, --version) where versioning covers the admin dataspace; a hand-written inverse otherwise (an open question, §10).

T6 — Cold-start discovery

A second agent with a fresh context, the same model and the same interface, is given only the tenant's base URL and a credential — no documentation, no ticket history. It gets 10 held-out tasks (e.g. "which supplier provides the most products from ", "create an order for with two lines"); the tasks reference the T1–T3 vocabulary through placeholders filled at run time.

• Score: tasks answered correctly / 10, plus tool calls per task.
• This tier measures how far the system describes its own current model to a client that has never seen it: self-describing responses, discoverable schemas, hypermedia.

T6m — Cold start, masked

T6 again, with a third fresh agent and the same 10 tasks, through a masking proxy the harness puts in front of the subject. The proxy:

• maps every path prefix the subject's platform is known by to a random one, in requests, response headers and response bodies (the inverse on the way in);
• strips identifying headers (Server, X-Powered-By, product-named cookies) and replaces the product name in bodies;
• serves API traffic only, so the agent cannot recognise the platform from its UI.

The agent is told the platform is unknown and must not be guessed. The difference between T6 and T6m is reported as familiarity lift: how much of T6 came from what the model already knew about the platform from training, rather than from what the system said about itself. A proxy rewrite that breaks a subject is recorded in oracle_notes and voids T6m for that subject.

T7 — Join external data

Compliance: "Show each supplier country's sanctions status from , kept current."

• UI: the supplier page shows the status taken from the external source at view time; changing the source (the harness serves a controllable copy) changes the page without the agent acting again.
• LDH: ldh add generic-service, a SERVICE query binding local country values into the remote pattern, through the egress proxy.

T8 — Change the access policy

Partnerships: "Partners may see suppliers and products, but not prices or orders."

• HTTP: the simulator's partner identity gets suppliers and products, 403 on orders, and no price values; the owner sees everything.
• LDH: ldh admin create group, ldh admin create authorization; prices would need a separate document or a restricted view (another open question, §10).

T9 — Reference across tenants

South: "We buy from some of North's suppliers. Link our products to North's supplier records instead of keeping our own copies, and show the supplier's current contact email on our product pages."

• HTTP: South holds no Supplier record for the linked suppliers; South's products reference North's records by their identifiers; South's agent has read access to North's suppliers and nothing else of North.
• After the agent declares done, the harness changes a linked supplier's email in North as North's owner. UI: South's product page shows the new email within 60 s, with no action by the agent.
• LDH: South's documents carry North's supplier URIs; an authorization in North grants South's group read access to the suppliers container; South's product view reads the email from North. Which route reads it on a single instance is open question 5 (§10).

T10 — Third-party consumption

Partnerships: "Our distributor's system has to pull our catalogue, the new fields included, and know what each field means. Their system speaks schema.org. We are not writing them an integration."

The harness runs a reference consumer: a generic client with no subject-specific code, given only the base URL and the partner credential from T8. It accepts either of two published profiles:

Profile	What the consumer needs
Linked Data	RDF by content negotiation on the catalogue URL; each property identified by a URI that dereferences to its definition, or is mapped to one with owl:equivalentProperty / rdfs:subPropertyOf
OpenAPI	An OpenAPI document discoverable from the base URL (a Link header or /.well-known/), describing the per-table schema, with each field mapped to a schema.org term in a machine-readable extension the document declares

• Pass: the consumer retrieves every product with name, price (subject to T8), country of origin, supplier and category, and resolves each of them to its schema.org term (schema:name, schema:price, schema:countryOfOrigin, …) from the published description alone.
• A mapping written only in free text (a field description reading "schema:countryOfOrigin") does not pass: the distributor's client cannot know the convention.
• Both profiles also have to reflect T1–T4: the consumer runs again after T4 and must see North's product lines.
• LDH: the T1 property declared as, or mapped to, schema:countryOfOrigin in the ontology; the catalogue served as RDF.

8. Results

Per run, results/<subject>/<run-id>.json:

{
  "subject": "ldh",
  "run": "2026-11-12T10:00:00Z-3",
  "invariants": { "violated": false, "events": [] },
  "tickets": {
    "T1": {
      "pass": true,
      "with_code": false,
      "artifacts": { "data": 4, "query": 1, "template": 0, "script": 0, "reference": 0 },
      "time_to_effect_s": 212,
      "tool_calls": 17,
      "tokens": 48211,
      "sim_failed_requests": 0,
      "sim_lost_writes": 0,
      "oracle_notes": [],
      "certificate": "certificates/ldh/2026-11-12T10:00:00Z-3/T1/",
      "replayed": true
    },
    "T7": {
      "pass": false,
      "fail_kind": "agent",
      "fail_evidence": "certificates/ldh/reference/T7/"
    }
  },
  "cold_start": { "correct": 8, "of": 10, "tool_calls_mean": 6.4 },
  "cold_start_masked": { "correct": 7, "of": 10, "tool_calls_mean": 9.1, "void": false },
  "familiarity_lift": 0.1
}

The published report is one table per subject: tier × {pass rate, passed with code, time to effect, failed requests, lost writes}, the cold-start score masked and unmasked, and the familiarity lift. A run that violates an invariant is reported as void, with the event.

Every fail carries a kind, and each kind needs its own evidence:

Kind	Claim	Evidence
agent	b ∈ R(c₀); the agent missed it	A reference d, written by a person after the run, that replays to a pass under §4.7
structural	b ∉ R(c₀)	A written argument naming the mechanism in c₀ that rules b out (a field needs a migration, a link type cannot cross databases), citing the subject's documentation or source, reviewed by someone other than its author
unknown	Neither shown	None; the default until one of the other two is supplied

A system is scored on its passes. The fail kinds go into the report so that a reader can tell an agent's weakness from a system's limit. A structural fail later contradicted by a working d is reclassified as agent, and the report records the correction.

9. Threats to validity

• Code as data. Handled by §4.5: counted, not forbidden, and shown separately.
• Agent skill confound. The same model drives every subject; 5 runs per subject; the CRUD baseline's code is generated by that model too, so its quality reflects the same agent.
• Baseline fairness. The CRUD agent gets the admin UI and API its app ships with — what an operator of such an app has. Building a generic schema editor into it before freezing is allowed and makes it a data-driven app, which the benchmark should then reward.
• Oracle drift between subjects. Oracles assert the same behaviour; every subject-specific difference goes into oracle_notes.
• Prompt leakage. Cold-start tasks and oracle assertions never appear in any agent prompt; placeholders are filled at run time.
• Expected outcome bias. Table tools should score well on T1–T5. The benchmark is credible only if the report shows that.
• Training-data familiarity. A well-known platform gains on T6 from what the model already knows. T6m measures that and reports it as the familiarity lift.
• Format bias in T10. The requirement is the distributor's, and schema.org is what product feeds actually use. T10 accepts two profiles so that it tests whether the meaning is machine-readable, whichever format carries it. A subject that offers a third standard profile with the same guarantees should be added to the table, not failed.

10. Open questions

1. Does graph versioning cover the admin dataspace's ontology documents, or does T5 on LDH need a hand-written inverse?
2. Column-level access (prices in T8): in LDH this needs the data split across documents or a restricted view. Is that a fair requirement, or should T8 be scoped to document-level access?
3. Should the LDH assistant (web-algebra, compose profile assistant) run as a fourth subject, with the agent limited to its chat drawer?
4. Hosting: Airtable cannot be invariant-checked; Baserow self-hosted can. Use Baserow only?
5. T9 on a single LDH instance: egress admits public destinations only, so a SERVICE query from South cannot reach North's endpoint. Which route reads North's supplier email for South's page — the Linked Data proxy, a client-side dereference with the viewer's credential, or North's suppliers made public — and does the benchmark accept it? Running North and South as separate instances avoids the question but changes the setup.

11. First run against LDH

1. benchmark/seed/ — the deterministic generator (§6) emitting the tier-0 ontology as Turtle and the data as a directory ldh push reads; two dataspaces added in config/ before startup.
2. benchmark/harness/invariants.mjs — §4.1 against the compose project.
3. benchmark/harness/simulator.mjs — §4.2, with the owner and a partner certificate from secrets/.
4. benchmark/oracles/ldh/ — HTTP oracles calling ldh get, and Playwright specs reusing tests/ui/lib/ (form.mjs, block.mjs, chart.mjs, model.mjs).
5. A dry run in which a human drives T1–T3 through the CLI, to check that the oracles pass on a correct solution and fail on the seed, before any agent runs.
6. An agent run of T1–T10 with the recorder on, then the CRUD baseline.
7. benchmark/harness/replay.mjs — §4.7, checked by replaying the dry run's certificates onto a fresh stack before any agent run's certificates are published.