AirOpsEnv v3: passengers and the customer desk (v0.4.0)

AirOpsEnv v3: passengers and the customer desk. AI agents run JetBlue's operations control center through a real storm day with its passengers on board, from the three desks real control centers have. The ops controller runs the flights and aircraft with the FAA and the crew scheduler runs the pilots and flight attendants, as in v2; the new customer desk looks after the passengers. Every flight carries its booked passengers, about 86,000 trips on a storm date, some connecting at JFK, Boston, Fort Lauderdale, Orlando or San Juan, generated from public records. When a flight is cancelled or a connection missed, its passengers are rebooked the way JetBlue's contract says: on JetBlue only, the earliest arrival first, for up to five days, then a refund. The customer desk sees whose connections are at risk and where a cancellation's passengers would go, and asks ops to hold a flight for them, not to cancel one, or to cancel one whose passengers are better off rebooked now; ops decides. The day is graded once, on what really happened, with no judge, under a cost that follows every passenger to their final destination.

What's in it

• The env airops-occ3 and the desk agent airops-desk, now also in role customer. Each desk sees only its own tools on the AgentEnv gateway. Each hourly watch the crew desk takes its turn, then the customer desk, then ops, then the crew desk again if ops asked it something and the customer desk if ops texted it; the env runs the turns, so every run replays exactly.
  • The customer desk's page lists the connections at risk in the next 4 hours and the flights ops might cancel in the next 6, with where their passengers would go if cancelled now. cust_flights and cust_flight look passengers up; cust_request asks ops HOLD, PROTECT, SUGGEST_CANCEL or TEXT.
  • The customer desk proposes and ops decides: an accepted HOLD is ops' hold; PROTECT and SUGGEST_CANCEL are advice ops carries out, if at all, with its own tools. No desk rebooks.
  • Mode one puts one agent at all three desks, with no channel, for comparison.
• Passengers, generated per date from public records: each flight's load from BTS T-100, and who connects where from the DOT's Origin and Destination Survey of the date's own period (DB1B to June 2025, DB1C after). An eval date has a median of 86,148 passenger trips, 2.7% of them connecting. International connections aren't in the public data and aren't modelled.
• Rebooking by JetBlue's published rules: JetBlue only, from the decision, the earliest arrival first (Contract of Carriage §37(c)); up to five days, then a refund (its Customer Service Plan); a flight 3 h or more earlier only when no other seat is left (DOT, 14 CFR 260.2). Crews keep their seats first. No compensation, meals or hotels on a storm day; a night away costs the passenger a lodging night.
• Cost P: each passenger's delay at the final destination, rebooking included; 66 per cancelled passenger, the refund exposure, more under 2 hours' notice; a lodging night per night away; every refunded fare; and v2's aircraft and crew costs.
• The pack b6-v3: v2's 70 dates (24 eval storm dates, 44 training dates, 2 calm controls), each with its passengers and its references under P: the best plan found (a repair of the passenger-aware plans on 16 eval dates, v2's best plan on 7, B0 on 1), every candidate it was chosen from, a capacity-aware lower bound, B0, doing nothing and JetBlue's own day. b6-v2 is unchanged.
• Grading: reward 0.2 + 0.8 exp(−g/0.5) against the best plan under P. B0 scores 0.308 on 20 of the 24 eval dates and more on the 4 where it comes within 10% of the best plan (a mean of 0.386); v2's best plan, blind to passengers, scores a mean of 0.602. reward_L against the bound beside it. A run is scored only if its audit passes, and the audit now plays the passengers' day again.
• The viewer, now with passengers: the customer desk at its pod in the room, its requests travelling to ops and back; the gate lounges behind the terminal's glass filling and boarding; connecting passengers walking gate to gate; the line at the service counter and the hotel shuttle at the curb; a passengers pane with the missed connections, rebookings and nights away; and where passengers are stranded on the US map. The lounges, the counter, the walks and the shuttle are inferred, for display only.
• FIDELITY.md: what is real data and what is generated, the rules' sources, every check and every miss.

The Sol board. GPT-6.1 Sol (reasoning effort high) on the 24 eval dates, once in each mode, registered before the first run, at caps of $3 a desk and $6 for one agent. All 48 runs scored, every one replays exactly through the env and passes the audit, and every reward recomputes from the graded costs. The first attempt was lost when its machine shut itself down before the results were fetched; the board was rerun whole, at these caps.

Mode	Mean reward (95% CI)	reward_L (95% CI)	Spend
Three desks, talking	0.721 (0.633–0.811)	0.372 (0.351–0.398)	$71.38
One agent at all three desks	0.677 (0.592–0.766)	0.366 (0.346–0.391)	$35.66
Standard crew desk (B0)	0.386
v2's best plan, blind to passengers	0.602

Paired by date, three desks score +0.044 over one agent (bootstrap 95% CI 0.007 to 0.083), higher on 14 of 24 dates, lower on 5 and level on 5, and their days cost 1.8% of the best plan less. Both recover most of what standard practice loses: at the median a Sol day saves 71% to 72% of B0's avoidable cost.

• It doesn't show that the third desk or talking helps. The three desks used about twice the compute (2.0× the spend, 2.0× the input and 2.4× the output tokens, more spend on every date), as on v2's board. The margin is small: Sol's score for the same day and mode moves by up to 0.23 from one run to the next, the size of the effect; a sign test gives p = 0.064; and leaving out the 7 dates with a below_best run, the CI crosses zero (−0.002 to 0.090). Nor does the board measure the customer desk by itself: there is no two-desk arm under P. The model customer desk sent 1,303 requests over the 24 days (887 HOLD, 184 PROTECT, 76 SUGGEST_CANCEL, 156 texts), and ops accepted 664.
• 12 of the 48 runs came in under the best plan found, both modes on 5 dates and three desks alone on 2. They score 1.0 and are flagged below_best; reward_L, against the bound, is published beside them. On 4 of those 5 dates B0 itself comes within 10% of the best plan, which is weak there; a longer search for better plans is left to v3.1.
• The caps were $3 a desk and $6 for one agent; 2 crew desks reached 90% or more of their $3 cap (the highest 96%), and none ended by it. The board cost $107.04, the pilot $6.20.

Validated before release, with its misses stated. Each stage was pre-registered and every bar recomputed by an independent agent from the run's files.

• The premise holds. Without crews, v2's aircraft plan costs at least 5% more under P than a passenger-aware plan on 16 of the 24 eval dates (12 needed); with crews, v2's best plan costs at least 5% more than the best plan under P on 17 of 24. It isn't the connections: the same dates pass at half and twice the connecting shares. It is pricing each cancelled passenger by their real rebooking.
• The passengers match their inputs: bookings within 5% of T-100 on all 70 dates, today and on the following days; connection times within 20 minutes of DB1C's at every connecting station.
• The reference replays exactly with a legal crew on every flight, and an independent re-simulation reproduced the 24 eval and 2 control dates exactly.
• The misses, shipped and documented:
  • a scripted customer desk built from published practice did not beat two scripted desks (2 of 24 dates, 12 needed; it cost more on 15): its requests not to cancel mostly protected flights that couldn't fly. The customer desk ships anyway, labelled;
  • the generated connecting shares run 0.1 to 1.9 points under the survey, because connections that can't be placed within 6 hours are dropped;
  • the bound is weak: 54% to 79% under the best plan on the eval dates (median 66%), most of it the solver not finishing, and negative on the calm winter control;
  • the best plan's repair overran its time cap on 49 of 70 dates;
  • JetBlue's own day strands 80% of the passengers off its cancelled flights overnight under JetBlue-only rebooking; no public record checks that.
• The desks, before any model ran: scripted desks played airops-occ3 in process on 4 training dates in both modes; every day graded, audited and replaying byte for byte, with two leak tests (every reference moved, and every delay a desk couldn't yet know changed) giving every page, reply and record byte for byte up to each watch. A Sol pilot on 2 training dates then scored all 4 runs, replaying exactly, for $6.20.

Films (attached, 1080p, 70 s each; each ends on the day's score beside the board's means):

• airops-occ3-hero-jfk-20250708.mp4, the hero: three Sol desks through the storm of 8 July 2025, shown at Kennedy, reward 0.795 (one agent scored 0.655 that day). The FAA's ground stop for JFK; passengers who missed B6 135 to Phoenix walking to the service counter; 17 stranded passengers leaving on the hotel shuttle; the customer desk asking ops not to cancel B6 675 to Hartford (51 passengers booked), ops declining because it can't fly before its deadline, and B6 675 staying at its gate; the HOLD of B6 308 to Portland and its connecting passengers reaching the gate.
• airops-occ3-bos-20231210.mp4: three Sol desks through the storm of 10 December 2023, shown at Boston, reward 0.825 (one agent 0.663). The room at 04:00 and first light over Logan in fog; the FAA's ground delay program for Logan from 15:30; the customer desk's HOLD of B6 333 to San Francisco for a connecting passenger accepted, and B6 333 pushing back held until 21:22; the crew desk's cancellation of B6 2120 accepted; B6 561 to San Juan's takeoff; and after midnight the customer desk's request not to cancel B6 1990, 131 passengers booked, accepted.
• airops-occ3-one-bos-20231210.mp4: the same day at one desk, one Sol agent at all three, reward 0.663, with no customer channel: first light over Logan in fog, the ground delay program, a crew leaving for the hotel, the network map, B6 1940 landing 3 h 14 min late and B6 265 to Fort Myers pushing back 2 h 58 min late.

airops-occ3-poster.jpg is the poster: the control-center room at 19:27 on 8 July 2025, the customer desk's holds for connecting passengers on their way to ops under the video wall, with ground delay programs at JFK and Boston.

The dataset at tag v0.4.0 leads with v3:

• v3_tasks, the default: the 70 dates with their passengers, every reference under P, the desks' rules and each date's world;
• v3_references: every candidate the best plan was chosen from, the bound and the gap, and the best plan itself;
• v3_episodes: every recorded three-desk run, replayed, with its audit; v3_desks: each desk's transcript as chat messages;
• v3_board and results/eval-v3.json;
• bundles/airops-occ3/: every date in both modes, the very task files the board ran ($3 a desk, $6 for one agent), so a v3 day plays straight from the dataset with agent-env hf run earakely-scale/AirOpsEnv-AgentEnv@v0.4.0 --bundle airops-occ3 --task b6-20240713-0400-joint --model <model>.

v2's and v1's tables, bundles and runs are unchanged, and airops stays the dataset's default bundle. AirOpsEnv v2 (agent-env airops-occ, the bundle airops-occ, the pack b6-v2) and v1 (agent-env airops, airops, b6-v1) ship as they were.

v3.1 will add world layers (ramp closures from thunderstorms, hub departure caps, METAR and TAF text for the desks), building on Alec Gutmanstein's work.

uv tool install agentenv-framework \
    --with "agentenv-airops @ git+https://github.com/earakely-scale/agentenv-airops-plugin@v0.4.0"
agent-env airops-occ3 setup
agent-env hf run earakely-scale/AirOpsEnv-AgentEnv@v0.4.0 --bundle airops-occ3 --task wiring-noplay

• Dataset (tag v0.4.0): https://huggingface.co/datasets/earakely-scale/AirOpsEnv-AgentEnv
• Replays in 3D: https://huggingface.co/spaces/earakely-scale/AirOpsEnv-AgentEnv

Flights BTS; passengers BTS T-100 and the DOT's Origin and Destination Survey (DB1B, DB1C); advisories FAA ATCSCC; crew rules eCFR; rebooking rules JetBlue's Contract of Carriage and Customer Service Plan and DOT's 14 CFR 260; costs FAA, BLS, GSA; airport © OpenStreetMap contributors (ODbL); imagery USGS The National Map; radar NEXRAD via the Iowa Environmental Mesonet. The 3D scene adapts code from PortSimEnv v1 (FineEnvs, Apache-2.0). Neither JetBlue, the FAA nor any union is affiliated with this project.