Skip to content

AirOpsEnv v3: six models on the board (v0.5.0)

Latest
Latest

Choose a tag to compare

View all tags
@earakely-scale earakely-scale released this 22 Oct 15:44
· 2 commits to main since this release
v0.5.0
b6b71d8

v0.5.0 puts six models on AirOpsEnv v3's board. GPT-6.1 Sol's board from v0.4.0 is joined by five more models, each running JetBlue's control center with three desks on the same 24 storm dates, on the same env, data and grading: GPT-6 Astra, Claude Opus 5.5, Claude Sonnet 5.5, GPT-5.6 Terra and Kimi K3, once a date each, at reasoning effort high. The models, the dates, the caps, the retries and how the models are compared were registered before the first of them ran. Sol isn't run again: its three desks are its board's 24 runs, as recorded. The env airops-occ3, the pack b6-v3, the grading and Sol's runs are unchanged.

Model Mean reward (95% CI) reward_L (95% CI) Below best Spend Cap a desk Median day
1 GPT-6 Astra 0.742 (0.673–0.813) 0.377 (0.356–0.406) 6 $544.24 $23.50 58 min, $21.33
2 GPT-6.1 Sol 0.721 (0.633–0.811) 0.372 (0.351–0.398) 7 $71.38 $3 70 min, $2.79
3 Claude Opus 5.5 0.626 (0.544–0.711) 0.357 (0.340–0.376) 3 $261.46 $9.50 43 min, $10.40
4 Claude Sonnet 5.5 0.556 (0.476–0.644) 0.347 (0.330–0.367) 2 $72.28 $5 27 min, $3.23
5 GPT-5.6 Terra 0.515 (0.443–0.594) 0.337 (0.324–0.353) 1 $84.41 $5 52 min, $3.70
6 Kimi K3 0.426 (0.366–0.491) 0.326 (0.310–0.342) 0 $150.51 $7.50 57 min, $6.01
GPT-6.1 Sol, one agent at all three desks 0.677 (0.592–0.766) 0.366 (0.346–0.391) 5 $35.66 $6 44 min, $1.52
Standard crew desk (B0) 0.386
v2's best plan, blind to passengers 0.602

Rows 1 to 6 are each model's three desks, ranked by their mean; CIs are bootstrap over the 24 dates. Below best: the runs that came in under the best plan found. Median day: a day's wall time and spend at the median.

What the board shows. GPT-6 Astra leads. Claude Sonnet 5.5, GPT-5.6 Terra and Kimi K3 lie below it: their 95% intervals are wholly under Astra's. reward_L, which the best plan doesn't cap, ranks the six in the same order.

What it doesn't.

  • It doesn't separate GPT-6 Astra from GPT-6.1 Sol or Claude Opus 5.5: their intervals overlap Astra's. Astra's lead over Sol, 0.021, is a tenth of Sol's own run-to-run swing on the same day (up to 0.23). Each model played each date once, so its interval is over the dates, not over repeated runs, and nothing is paired across models.
  • The best plan's ceiling compresses the top. A run under the best plan found scores 1.0 whichever model played it and however far under it came: Astra's did on 6 dates and Sol's on 7, Opus's on 3, Sonnet's on 2, Terra's on 1 and Kimi's on none, all on the same 7 dates. reward_L is published beside them.
  • Spend doesn't follow the ranking. Astra's board cost 7.6 times Sol's for 0.021 more; Kimi K3's, twice Sol's, has the lowest mean; Sonnet's and Terra's are near Sol's.

Caps and spend.

  • A desk's cap buys what $3 bought a Sol desk: Sol's median desk's tokens at the model's prices, rounded up to the next $0.50. So GPT-6 Astra $23.50, Claude Opus 5.5 $9.50, Kimi K3 $7.50, and Claude Sonnet 5.5 and GPT-5.6 Terra $5. The caps stopped 5 of the five boards' 360 desks, all crew desks: one of Astra's, three of Opus's and one of Kimi's. Every one of those runs scored.
  • The five boards ran at 6e5ec17 on 22 October 2026, from 05:17 to 07:59 UTC, on five r7i.24xlarge machines, one a model, within a budget of $3,000 for model spend and cloud together. They spent $1,125.30 on the models.
  • Every run scored. GPT-5.6 Terra's 24 took 31 attempts: 7 first attempts lost a desk to Azure's rate limit (provider_error) and scored on a retry. Those 7 cost $12.39, counted in the total but not in Terra's spend on the board, and every attempt is kept in v3_episodes.

Replays. All 168 board runs replay exactly through the env on Linux, as the boards ran, and pass the audit, and every reward and reward_L recomputes from the graded costs and the pack's best plan, bound and B0. Two runs, Terra's on 2023-06-16 and Kimi's on 2023-12-18, mismatched in the crew tools when replayed on a heavily loaded Mac. On an idle Linux machine each replayed exactly three times out of three. The crew tools solve a MIP under a time limit, so a loaded machine can return another answer.

The dataset at tag v0.5.0:

  • v3_board: one row per model and mode, the three-desk rows ranked by their mean, each with its cap, the desks the cap stopped, its spend and tokens; then B0 and Sol's two modes paired by date;
  • v3_episodes and v3_desks: every attempt of every board, Terra's unscored ones included (175 and 477 rows on the eval split);
  • results/eval-v3.json: one models entry per model and mode;
  • runs/occ3-board-<model>/: each board as recorded.

The Space replays all 168 board runs, each card naming its model and mode. A run opens in the control-center room, and the Terminal view is a camera (key 9). Start here is still Sol's three desks at JFK on 8 July 2025.

In the code: scripts/hub_dataset.py publishes each occ3-board-<model> folder as a board of its own and names any run that no attempt scored; scripts/occ_viewer.py builds a board's viewer episodes and list stills without filming them; and the desk agent prices GPT-5.6 Terra and Kimi K3.

uv tool install agentenv-framework \
    --with "agentenv-airops @ git+https://github.com/earakely-scale/agentenv-airops-plugin@v0.5.0"
agent-env airops-occ3 setup
agent-env hf run earakely-scale/AirOpsEnv-AgentEnv@v0.5.0 --bundle airops-occ3 --task wiring-noplay

No new films: v3's films, Sol's three, stay on v0.4.1. v3's release notes and Sol's board are on v0.4.0.

Assets 2
Loading