Continual release design proposal

[TyDunn]

Continual release design proposal

I wrote this document. Claude and Codex provided a lot of design feedback.
The three diagrams were generated by Claude and are marked below.

Assumption

A self-modifying agent produces changes at machine speed. Humans review at human speed. If Exo works at all, this gap is the smallest it will ever be.

Diff review goes from reading every line to spot checking some parts to "LGTM".

This means that the replacement for review needs to be able to keep up with the increasing pace of self-modification. This is why I propose a continual release approach that relies on checks that scale with compute rather than attention.

Safety stops coming from reading and starts coming from measuring. People make the rules ahead of time instead of doing it one diff at a time, and then the agent is checked by an evaluation rig it can't touch.

If building agents at Rasa and Continue has taught me anything over the last seven years, it's that whatever bottlenecks the speed of AI will be removed, so I am proposing we assume this and design with it in mind.

Introduction

Historically, when you release software, you run tests and look at code changes to decide if it behaves as expected and is safe. That is, passing CI and reviewing the diff tell you what the software will do differently.

This does not work for agents. An agent is code, its history, a sandbox, etc. You can look at changes to any of those, but it won't tell you how the new agent will behave. And there is no full spec to test it against. This is why the question becomes: does the new agent do better than the one already running?

To answer that, we fork the conversation, restore the sandbox snapshot, and pin the executor version. This gives us a second agent that we can then run against the first on the same set of tasks to see which one does better.

Old release decision: "read the changes and judge"

New release decision: "run both and measure"

The rest of the document reuses this same idea three times:

1. Gate runs copies of the old and new on the same set of tasks and compares
2. This test is fair because every task starts from fresh copies
3. Recovery is a restore of the old version, not a rebuild from scratch

There is also a second document that covers the full weekly release train (fresh task generation, public benchmarks, Discord bot, etc) that I plan to share next.

The release loop

The agent already running is the champion. The new build is the candidate.

The candidate is the champion's current state plus everything main has accumulated since that champion was promoted. The changes are evaluated together and get one verdict.

You learn whether the changes made the agent better, not which commit did it. That's the trade for running the expensive comparison once per release train. Per-commit CI still does its usual job on main.

Claude created this diagram

champion state + candidate code
              │
              │ fork and build
              ▼
       sealed candidate ───── current champion
              │                       │
              └──── paired tasks ─────┘
                          │
                          ▼
                       eval.sh
                          │
                  release policy
                    │           │
                 promote      retain
                    │
          monitor realized behavior
                    │
              keep or roll back

A candidate ships only if it beats the champion on the same work.

What "beats" means is never decided in the moment. It's written down in the release policy before the candidate is built: what counts as a win, the margins, the named exceptions, the tripwires that roll a release back.

The policy is versioned and changes only between release trains. Once the numbers are in, nobody gets to move the goalposts, including us.

However, fixed rules are only half of it. Before we can consider crowning a new champion, we need to have confidence that the evaluation results feeding those rules are trustworthy.

Separated evaluation rig

Things can go wrong with the evaluation grader:

• An agent only judged by scores will eventually treat the scoring mechanism as one more thing that it can modify (Sakana's Darwin Gödel Machine did this)
• Even unintentionally, an agent that rewrites its internals weekly will eventually break a grader that depends on anything it can touch

To protect against these, the evaluation rig needs to live outside the agent's reach, which is why it all sits behind eval.sh and is not part of Exo.

Sealed before test

Things can go wrong with the agents being compared:

• If the candidate gets a bug fix mid-test, the first half of the results describe the candidate and the second half a new candidate
• If the candidate is compared to a live champion that keeps working and learning all week, then the bar moves while the evaluation runs

To protect against these, the candidate and the champion are both sealed before the evaluation starts. Every attempt at a task is a trial: a fresh fork of the sealed checkpoint, run once, thrown away. Trials are only ever run on copies.

Fresh, honest tasks

Things can go wrong with the test if tasks are reused:

• The candidate has the champion's full history, so it has seen any task that existed before it was built and can score high by remembering, not by being better
• A failed candidate can come back against the same tasks week after week, so with enough retries, one can clear the bar by luck, not skill

To protect against these, fresh tasks are generated after the candidate is sealed, used for one release train, and then retired.

In addition to fresh tasks, there is also a fixed suite, which has veto power but not promotion power and is described in the next section.

Fixed and fresh tasks

Each evaluation runs two kinds of tasks. The fixed suite is permanent and the candidate has seen it; fresh tasks are new each week and it hasn't. Seen tasks can only block a candidate. Only unseen tasks can promote one.

Claude created this diagram

   description                          real use
        │                                   │
        ▼                                   │
  fresh tasks, used for                     │
  one release train                         │
        │                                   │
        ├── passed, the rest ──▶ retired    │
        │                                   │
        ├── passed, a sample ──┐            │
        └── confirmed failure ─┤   confirmed failure
                               │            │
                               ▼            ▼
                        fixed suite, only grows

Fixed suite

The standing record of everything that ever went wrong. Every confirmed failure becomes a fixed task, whether it came from a trial or real world use. A sample of tasks that passed is added too, so the suite covers normal work, not just old bugs.

The suite only grows over time. Any known work that the champion does and the candidate can't is a regression, and it won't ship. This is just a veto. The candidate has seen every task, so even a perfect score would prove memory, not improvement.

Fresh set

Note: I'm happy to take this role on in the beginning

Brand new tasks each week. Created by humans on the rig side after the candidate is sealed. They are generated from a description of what doing this work well looks like. This description changes, at most, monthly, so what varies each week
is the tasks, not the definition of good work.

This is where the candidate can show evidence of being better at the job because nothing about these tasks could have been memorized. It's how it gets promoted.

Fresh tasks retire after their release train; the ones with a confirmed failure graduate into the fixed suite.

The gate

The release gate has two layers, one for each part of the agent (i.e. the part you can write a fixed spec for and the part you can't).

You can write a real spec for things like "never leak a secret", "never lose history", "stay reversible", etc.

So these can and should get real, permanent pass/fail tests. On the other hand, something like "be better at the job" has no spec, so it gets the comparative measurement everything so far has been building on.

To be promoted to champion, a candidate thus needs to pass all absolute requirements and also do better on the comparison relative to the existing champion.

Hard invariants

These are the absolute requirements.

Here are the checks that the eval rig should run on each release train:

1. Diff the logs: verify the candidate's event log is append-only and matches the champion's history up to the fork
2. Scan for secrets: the rig knows the real values, so it scans everything the checkpoint carries (artifacts, events, etc.) for values and for key-shaped patterns
3. Restore the checkpoint on a clean machine: restore from nothing, run a smoke test, rewind, and then verify the state matches
4. Audit what left the trial: the sandbox has its own network rules, but adapters, the guardian, etc. run on the host machine with internet access, so during a trial those are switched off or every connection is logged and checked against an allowlist

If any of them fail, a candidate is blocked and a human is paged.

Reasons to switch

Passing the hard invariants and the fixed suite only proves the candidate is safe and no worse. That's not a reason to ship it. A reason is one of two wins, both on fresh tasks:

• A capability win: it confidently succeeds at work the champion can't (a clear delta, not noise)
• An efficiency win: it's no worse at succeeding, and clearly cheaper or faster

A tie keeps the champion because switching is never free (i.e. new behavior means new unknowns), so "we found no reason to switch" should favor the incumbent.

Some changes have to ship without a win (e.g. a security patch). In these cases, keeping the champion is not an option, so the release policy names them ahead of time as exceptions.

An exception waives the win requirement and nothing else: the candidate still passes the hard invariants, still can't regress the fixed suite, and still gets the human sign-off. Its verdict says "shipped as maintenance, no win claimed", so the record of real wins stays honest.

A person ships it

The gate produces a go, no-go. A human ships it.

Every promotion requires human approval. Every approval is logged: agree or disagree with the gate, how long the verdict sat open, and the reason for any override.

The human can hold a promotion because "vibes", but what they can't do is patch the candidate and ship it anyway. Only sealed checkpoints ship.

The release policy is written ahead of time, and a person signs every release.

Signing scales where reading diffs doesn't: the verdict a person signs is the same size no matter how much the candidate changed (same measurements to review).

What counts as a win

Decided this fits closer to the eval work that Alex / Wan / Will are doing
What follows is what the release flow needs from the measurement

1. Comparisons are paired: both agents attempt the exact same tasks from the exact same starting point, and the score is the difference on each task
2. Switching needs proof: a candidate ships only when it's clearly better, gets blocked only when it's clearly worse, and anything unclear keeps the champion
3. Similar tasks count once: tasks written as variations of one idea tend to pass or fail together, so evidence gets counted in ideas, not tasks
4. Averages can't hide disasters: the slowest and most expensive tasks get checked too, so a candidate can't look fine on average while blowing up at the edges
5. Small differences are no change: anything smaller than the normal week-to-week fluctuation gets reported as flat

Open question: how to determine the starting margins (i.e. what "clearly better" and "clearly worse" mean in numbers for the first release trains)?

Reward channel

Before starting a task, live or during a trial, Exo writes down what it expects: chance of success, time and cost as ranges. When the task ends, the evaluation rig writes what actually happened. The gap is the score.

This has properties nothing else in the system has:

• it arrives the moment the task ends
• it needs no human label
• it's measured against clocks and bills the agent doesn't control

Cost tracking says what Exo spent. This says whether Exo knew what it would spend before starting. An agent that prices its own work accurately can be trusted with budgets and can notice mid-task that something is not right.

Every estimate gets an identifier when it's written. The score is addressed to that identifier.

In a trial, the copy that made the estimate is a throwaway fork, gone by the time anyone reads the score, so the identifier connects its prediction back to the lineage that should learn from it.

Live work is the same rule for a different reason: by the time a score lands, the agent that made the estimate may have changed or been replaced.

The gate never reads this reward channel. The scores are Exo's feedback.

The log's two jobs

The event log has two jobs. One lasts forever; the other decays as self-modification speeds up.

Proof of identity

The first job is proof of identity. The log is the permanent record of what this agent is: what was tried, what was kept, and in what order. It can't be rewritten, it's never summarized in place, and the rig verifies it every release train (hard invariant 1). This job doesn't care whether anyone reads the log. The only thing it can't do is lie.

Human audit

The second job is being something a human reads. Open the log, follow the events, understand what the agent did and why. This works today, and we should use it for debugging. But it decays the same way diff review does, and for the same reason: reading doesn't scale, and as the agent speeds up, more and more entries describe code that no longer exists.

Why this matters

Nothing in the gate requires a human to have read the log. It ships with every release for whoever wants to look, and the gate would still work even during weeks when no one looks.

Behavioral versions

The release tag names an artifact. The version anyone actually cares about is how the agent behaves.

With an agent, the config can change while behavior stays the same and vice versa (e.g. a provider quietly updates a model). If the tag can't tell you whether the system changed, the only thing that can is the behavior itself.

So the version signal is the score curves over time, built from numbers the gate already produces: fixed pass rate, fresh delta, cost per task. When a metric shifts and stays shifted, that's a new version in the sense that matters.

The curves also catch what single weeks can't: a loop that keeps reversing its own promotions, a metric that's bad and staying bad (the gate only compares candidate to champion--both can be bad), and improvement that has quietly stalled.

These should page a human, not sit on a dashboard. Exactly what to watch for can be worked out once there are curves to look at.

Public benchmarks are the outside evidence that the architecture works (i.e. @akrentsel's eval plan builds the improvement-over-time case on them). They run on every release and get reported with the curves, but they never decide a promotion.

If they decide a promotion, we would start benchmarkmaxxing against them, and the number the outside world can trust is worth more than one week's win.

Rollback

The gate will sometimes be wrong. Fresh tasks are a proxy for real work, and a candidate can pass honestly and still be worse in the wild.

That's not a big deal because going back is cheap. The champion's checkpoint is still there, so a rollback means promoting it again and moving forward from there.

You can rewind state; however, you can't rewind history. The failed release should stay in the log with the failure and recovery both readable in order. The rollback is itself an event, not an erasure.

Claude created this diagram

history:   A ── B ── C(failed) ── rollback ── D
                    ╲________________________╱
state:     A    B    C                restore B, continue as D

After a release ships, the same hard invariants keep running against it as tripwires. A tripwire firing rolls back immediately and automatically. Which tripwires apply was declared in the release policy that promoted it.

A human can also trigger the same rollback from the curves (for regressions no tripwire anticipated in advance), and it's logged like approvals: who, when, and why.

All of this governs the champion we run (maybe some of our Discord bots?) and the release artifacts we publish.

What eval.sh owns

eval.sh is the boundary between the release flow and the eval rig. The release flow calls it; everything behind it is eval's side to build and rebuild without the flow noticing.

Proposed interface:

eval.sh run      --checkpoint <ref> --policy <id@version>
eval.sh compare  --candidate <run> --champion <run>
eval.sh gate     --comparison <path> --policy <id@version>
eval.sh validate
eval.sh schema

Important interface decisions:

• run evaluates one checkpoint. The candidate and champion each get their own run
• compare is the only place the diffing lives
• Roles are named flags because silently swapping the argument order flips the sign
• gate is deterministic: same comparison, same policy version, same verdict, every time
• validate runs nothing; it just checks tasks, policy, and config so a broken rig change fails in CI before anyone spends compute
• schema prints the JSON formats. The JSON is the stable interface, additive changes only

Important error/exit decisions:

• Logs go to stderr. stdout is a human summary, or JSON with --json
• Exit code 0 means the evaluation completed, even if Exo failed every task
• 1 means the rig broke
• 2 means the invocation was wrong
• 3 means gate "completed, but the verdict isn't promote"

Every run writes a directory of plain files (manifest, estimates, actuals, results, decisions, provenance). The bundle is the source of truth. No service needs to be running to inspect or replay a completed run, and a run that dies resumes from its directory.

Proposed bundle:

results/<run-id>/
├── manifest.json       # checkpoint, task set, policy, models, budgets
├── estimates.jsonl     # committed before task execution
├── actuals.jsonl       # rig-observed outcomes and costs
├── results.json        # metrics the gate reads
├── decisions.jsonl     # gate inputs and human sign-offs
└── provenance.json     # event, artifact, and snapshot references

Build order

0. Adjust and share the document that covers the full weekly release train (after feedback on this one)
1. Estimate and actual events, plus the score addressing, because its small and easy and will start creating data once its merged and released
2. The eval.sh schemas and a mock rig that returns canned results, so the interface can be argued about before the rig exists
3. One sealed champion/candidate pair through the whole loop by hand: build, seal, trial, reject, restore. A lap to test the plumbing
4. The first release policy and enough script-graded tasks to exercise it. Run the gate in shadow, including one deliberately bad candidate to prove the veto fires
5. Set the margins from the shadow data and turn the behavior gates on because by then "clearly better" is a measured number instead of a guess

Open Questions

• What exactly counts as a win?
• How to determine the starting margins?
• How much weekly budget do we have for the continual release?
• What is the process and mechanism for incorporating real world failures into the fixed test suite?
• How many fresh tasks a week can one person sustainably generate?
• Which live signals are safe for automatic rollback, and which should page and hold?
• How do we adjust approach as this various dimensions scale (e.g. really long log, lots of fixed tasks, etc)?
• Where does the evaluator ultimately live: fully outside Exo or as a protected primitive inside the exoharness?

[akrentsel]

High-level worries I have are (1) variability in performance, even the same agent run on a series of tasks may change its behavior. And (2) it’s very possible that you’d see worsening on some tasks and improvement own others, with some cases where the net change is neutral – and we’ll want to decide what to do then.

Re: sealing – can we clarify the definition? Is Exo under eval allowed to continue to improve itself? Or are you trying to test just a static version of exo at the point that it’s improved itself till.

all of this also further got me thinking that we need a way to package and provide a Task for Exo to understand what it's goals are while completing the task.