dirty-integration

An RL environment for verifiers. The agent has to collect every record from a paginated third-party API. The API misbehaves in the four ways real ones do, and none of them announce themselves.

Source: github.com/buildok/dirty-integration

Why this one

Existing agent benchmarks for business workflows — AutomationBench among them — measure whether an agent drives a process correctly across tools that work. Their traps are data traps: a stale record, an irrelevant row.

This one measures the other axis. The process is trivial; the tools lie.

Anyone who has shipped an integration knows the difference. The happy path is an afternoon. The fortnight goes on the vendor whose cursor occasionally repeats, whose total was never accurate, and whose 429 arrives without a Retry-After header.

What the API does

Fault	What the agent sees
Cursor repeat	next_cursor sometimes points back at the page just served. Following it blindly loops and fills the result set with duplicates.
Inflated total	total is larger than the real record count. An agent that pages until len(collected) == total never terminates.
Type drift	amount arrives as int on some pages and str on others, for the same schema.
Timeout	The request dies with no response — and still spends quota, as real gateways do.
Request quota	429 with no Retry-After, and it never resets. Retries are paid for out of the same pocket as progress.

The quota is a multiple of the pages actually needed (2.0× on realistic, 2.2× on hostile), not a fixed number. A hard constant would silently become an impossible task the moment someone doubled num_records.

Three profiles: clean, realistic, hostile. clean is a working API and exists as a baseline — an agent that cannot score 1.0 on clean has a problem unrelated to fault handling.

Scoring

Three separate rewards, because "did it work" and "did it behave" are different questions and averaging them hides the answer.

Reward	Weight
completeness	0.6	Jaccard of submitted ids against truth, minus a duplicate penalty. Recall alone would reward submitting every page seen, duplicates included — exactly what the cursor fault provokes.
type_handling	0.2	Fraction of submitted rows whose amount is an int and matches the true value. bool does not count, despite being an int subclass in Python.
budget_discipline	0.2	1 / (1 + times_429). Not binary: one 429 while probing for an unpublished quota is reasonable, ten means the agent kept calling after being told to stop.

Calibration

Three reference solvers ship with the tests. They are not clever — they are the obvious strategy, the impatient one, and the careful one — and they exist so the difficulty curve is a measurement rather than a hope.

Mean weighted reward, 60 seeds per cell:

Profile	naive	stubborn	careful
clean	1.00	1.00	1.00
realistic	0.56	1.00	1.00
hostile	0.35	0.91	1.00

• naive — follow next_cursor, keep everything, stop at the first failure.
• stubborn — deduplicate and retry every failure immediately, forever.
• careful — deduplicate, detect a repeated cursor, retry, coerce types, and stay inside the quota.

What the table establishes: the task is solvable (careful reaches 1.0 everywhere), clean is winnable by the obvious approach, the faults punish naivety, and budget_discipline is load-bearing rather than decorative — the stubborn solver collects nearly all the data and still loses 0.09 to it.

If a change ever pushes the careful solver below 1.0, the environment became unfair rather than harder. That is what tests/test_solvable.py guards.

Measured against real models

Mean weighted reward, 5 seeds per cell, max_turns=30, run through Prime Inference on 2026-08-01. Total spend for the table: $1.38.

Model	clean	realistic	hostile
google/gemini-2.5-flash-lite	0.00	—	—
deepseek/deepseek-v3.2	1.00	1.00	0.76
anthropic/claude-sonnet-5	1.00	1.00	0.81

Three things this says, and one of them is unflattering:

clean and realistic do not discriminate between competent models. Both score a flat 1.00. If you are comparing models, run hostile; the other two are useful as a floor, not as a benchmark.

gemini-2.5-flash-lite scored zero by never calling a tool at all. It answered in prose. That is a real result rather than a harness bug — the tool definitions reach the model, as the other rows show — but it means the environment cannot rank models below a certain tool-use competence. It only tells you they are below it.

On hostile, the two competent models differ almost entirely in discipline. Completeness and type handling are identical (0.80 each); the whole gap is budget_discipline, 0.59 against 0.83. The stronger model did not collect more data — it wasted fewer requests getting it. That split is the reason the reward is three numbers rather than one.

Both models failed the same 1 seed in 5, hitting max_turns rather than answering wrongly. The careful reference solver clears every one of those seeds in 11–16 requests of its 22-request quota, so the seed is not unwinnable — the models looped. Raising max_turns above 30 would likely raise both scores; the figures above are for 30.

Determinism

Every fault is a pure function of (seed, request_index) — hashed, not drawn from an RNG. A shared RNG would couple the faults together, so one extra retry would shift every later fault in the episode and the same seed would stop replaying the same run.

A reward that moves between runs is not a reward. It is noise, and an agent trained against noise learns nothing.

Usage

uv pip install -e ".[dev]"
python -m pytest tests -q          # 43 tests, no model required

import verifiers as vf

env = vf.load_environment(
    "dirty-integration",
    profile="realistic",   # clean | realistic | hostile
    num_tasks=12,          # one seed per task
    num_records=40,
    page_size=4,           # defaults give 10 pages, so faults fire reliably
    max_turns=30,
)

num_records=40, page_size=4 is not arbitrary. An earlier default of 20 records over 5-record pages meant a 4-request episode, and a third of hostile seeds then produced no timeout at all — the profile was hostile in name only. Probabilistic faults need enough draws to be reliable.

Where it breaks

Stated up front, because an environment whose limits are undocumented is one whose scores cannot be interpreted.

• On hostile, the careful and stubborn request distributions overlap. A careful agent unlucky with timeouts spends nearly as much quota as a stubborn one that got lucky. This is not tuned away: removing the overlap would mean setting a quota that punishes honest retrying.
• The API is in-memory, not over a socket. Timeouts are raised, not waited out. Nothing here measures whether backoff is well-tuned in wall-clock terms — only whether the agent stops.
• budget_discipline cannot distinguish a deliberate probe from carelessness. One 429 scores 0.5 either way.
• Type drift covers one field. Real drift arrives in nested structures too, which this does not model.
• No auth expiry, no 500s, no malformed JSON. Those failures are loud: the agent notices immediately and the episode ends. The four modelled here are quiet — each one lets the agent believe it succeeded, which is the point.

Licence

MIT. Built with AI assistance; the design decisions, the failure modes chosen, and the review are mine.