Skip to content

AirOpsEnv v2: the ops control center (v0.3.0)

Choose a tag to compare

View all tags
@earakely-scale earakely-scale released this 20 Oct 07:07
· 110 commits to main since this release
v0.3.0
a37cc83

Known issue: v0.3.0's install can't load agent-env airops-occ (numpy, pandas, highspy and airportsdata were in extras only), and the Space showed the airports without their aerial photo. Both are fixed in v0.3.1; install that.

AirOpsEnv v2: the ops control center. AI agents run JetBlue's operations control center through a real storm day, split between two desks the way airline control centers split the work. The ops controller runs the flights and aircraft with the FAA: it cancels, holds, and claims and swaps arrival slots in the FAA's programs. The crew scheduler runs the pilots and flight attendants: it calls reserves, swaps crews, deadheads them and rests them in hotels, under FAR 117 and 14 CFR 121.467. Neither can do the other's job, so they talk through typed requests the other desk accepts or declines, and free text. The day is JetBlue's whole network from 04:00, about 700 flights, 220 aircraft and 900 crews and reserves, with the FAA Command Center's ground stops and ground delay programs arriving as they were issued, graded with no judge against the best plan of aircraft and crews found in hindsight, next to its proven bound.

What's in it

  • The env airops-occ and the desk agent airops-desk, in role ops, crew or both. Each desk sees only its own tools on the AgentEnv gateway. The env runs the turns, crew desk then ops then a crew reply each hourly watch, so every run replays exactly. A crew system crews whatever the crew desk leaves alone, by standard practice. Mode one puts one agent at both desks for comparison.

  • The world, rebuilt for whole days.

    • The FAA's ground stops and ground delay programs at JFK and Boston as slots, EDCTs and compression, from the ATCSCC advisories.
    • The FAA's published cost rates.
    • Pilots and flight attendants on trips generated from the published schedule, with reserves, deadheads, hotels, the 2 h pilot extension, all within the crews' legal limits.

  • The pack b6-v2: 24 eval storm dates (12 at JFK, 12 at Boston), 44 training dates and 2 calm controls, from December 2022 to July 2026. Each date has its best plan, its proven bound and the standard crew desk's day (B0).

  • Grading: reward 0.2 + 0.8 exp(−g/0.5) against the best plan, with B0 at 0.308; reward_L against the bound beside it. A run is scored only if its audit passes: news on time, turns, causality, roles, crew legality, the bound, presence.

  • The viewer, now for the control center:

    • a 3D control-center room with the two desks and their requests travelling between them;
    • the crews at the airport: walking to their gates, on the apron bus, taking the hotel van, and aircraft tagged "waiting for crew";
    • the crews on the US map;
    • expert panes with each crew's duty clock, the request log and the desks' pages;
    • films that cut between the room and the airport.

    The room, the crew room and the crews' paths are display-only.

  • FIDELITY.md: what is real, modelled and generated, each stage's checks against the record, and every miss.

The Sol board. GPT-6.1 Sol on the 24 eval dates, once in each mode, pre-registered. All 48 runs scored, replay exactly and pass the audit, and an independent recheck recomputed every reward.

Mode Mean reward (95% CI) reward_L (95% CI) Spend
Two desks, talking 0.598 (0.545–0.658) 0.523 (0.484–0.564) $48.33
One agent at both desks 0.550 (0.497–0.609) 0.487 (0.450–0.526) $24.54
Standard crew desk (B0) 0.308

Two desks beat one agent by +0.049 (CI 0.024–0.075), higher on 18 of 24 dates. That isn't evidence that talking helps:

  • The two desks used about twice the compute.
  • Before any model ran, scripted desks failed the pre-registered gate for coordination.
  • The channel-off run comes next.

On 2023-07-13 the two desks came in under the best plan, so that run scores 1.0 and is flagged; its reward_L is 0.578. On 2026-07-09 a registered retry replaces an attempt whose crew desk lost its connection.

Validated before release, with its misses stated. JetBlue's own decisions replayed in the world fly what JetBlue flew: 1 forced cancellation in 116,986 flights. The FAA delay matches other carriers' recorded delay out of sample (JFK 0.95, Boston 1.10). JetBlue's real day crewed by the generated crews leaves 0.23% of flights without a legal crew. The misses, shipped and documented:

  • the calm-day check passed on 85.6% of dates against a 90% bar;
  • the standard crew desk missed the controls' 3% margin;
  • the crew filter kept somewhat smaller storms.

Films (attached, 1080p, about 70 s each; each ends on the day's score beside the board):

  • airops-occ-hero-bos-20240713.mp4, the hero: two Sol desks through the Boston storm of 13 July 2024, reward 0.806. The room at 04:00 and first light over Boston; the crew desk's HOLD of B6 515 accepted and B6 515 pushing back held until 07:45; the 14:00 ground stop; ops' COVER of B6 1979 accepted in the room; the network map; B6 451's crew walking up to gate C23, its pushback and takeoff; a crew van leaving for the hotel.
  • airops-occ-bos-20231210.mp4: Boston in fog and rain on 10 December 2023, two desks, reward 0.754. The delay program announced for 15:30; ops' COVER of B6 169 accepted; a hotel van; the crew desk putting pilots on B6 1451 and its pushback; B6 265's takeoff; the crew desk's CANCEL of B6 2120 accepted in the room.
  • airops-occ-bos-20250111.mp4: Boston in snow on 11 January 2025, two desks, reward 0.805 (one agent scored 0.819 that day). Five flights cancelled at once on the room's wall; first light over snow; the HOLD of B6 2063 accepted, then the flight cancelled an hour later; the ground stop; a hotel van; a flight waiting for its crew; B6 591's takeoff.

airops-occ-poster.jpg is the poster: the control-center room at 06:17 on 13 July 2024, as the crew desk's hold of B6 515 is accepted.

The dataset at tag v0.3.0 leads with v2:

  • v2_tasks, the default: the 70 dates with every reference, the advisories, the desks' rules and each date's world;
  • v2_runs: every recorded attempt, replayed, with its audit;
  • v2_desks: each desk's transcript as chat messages;
  • v2_board and results/eval-v2.json;
  • bundles/airops-occ/: every date in both modes, the very task files the board ran, so a v2 day plays straight from the dataset with agent-env hf run earakely-scale/AirOpsEnv-AgentEnv@v0.3.0 --bundle airops-occ --task b6-20240713-0400-joint --model <model>.

v1's tables, bundle and runs are unchanged, and airops stays the dataset's default bundle. AirOpsEnv v1 (agent-env airops, the bundle airops, the pack b6-v1) ships as it was.

uv tool install agentenv-framework \
    --with "agentenv-airops @ git+https://github.com/earakely-scale/agentenv-airops-plugin@v0.3.0"
agent-env airops-occ setup
agent-env hf run earakely-scale/AirOpsEnv-AgentEnv@v0.3.0 --bundle airops-occ --task wiring-noplay

Flights BTS; advisories FAA ATCSCC; crew rules eCFR; costs FAA, BLS, GSA; airport © OpenStreetMap contributors (ODbL); imagery USGS The National Map; radar NEXRAD via the Iowa Environmental Mesonet. The 3D scene adapts code from PortSimEnv v1 (FineEnvs, Apache-2.0). Neither JetBlue, the FAA nor any union is affiliated with this project.

Assets 6
Loading