Spike: measure ACA dynamic-sessions transport cap to pick the runner API branch (S vs P)

[jrob5756]

Parent: #284 — Agent-in-Sandbox via a remote aca provider.
Design doc: docs/aca-provider.design.md — Open Question #1 (Blocking / Phase 0), DD3.

Why (blocking)

The aca provider relocates an agent's whole runtime into an ACA custom-container
dynamic session. A single agent turn can run for minutes, so the runner transport
must survive ACA's request/idle-timeout limits. Two ACA limits must not be conflated:

• Per-request forwarded-duration cap — how long one request may stay open.
  Undocumented for sessions; general ACA ingress idle-timeout defaults to 4 min,
  premium max 30 min. This is the real cut-off risk.
• Inter-request Timed cooldown (300–3600 s) — how long an idle session
  survives between requests, reset each time the session API is called.

Two candidate transport branches with identical event/result semantics:

• Branch S (preferred) — single streaming request. One POST /execute streams
  NDJSON event frames until a terminal result frame. True incremental events +
  mid-call partials + real interrupt. Viable only if one request can stay open for
  the whole turn.
• Branch P (fallback) — submit + poll. POST /execute returns a job id; the
  host polls for event batches. Each poll is a fresh request under the cap and
  resets the cooldown. Survives a short cap at the cost of coarser streaming/interrupt.

This spike gates the build and fixes the conditional streaming_events /
interrupt capabilities. No provider code should land before it resolves.

Method

• Stand up a throwaway custom-container session pool (DD6 two-step deploy: push
  image to ACR, then az containerapp sessionpool create --container-type CustomContainer …).
• Bake a trivial heartbeat runner into the image: emits one NDJSON frame/second,
  then a terminal result frame.
• Drive it from a standalone probe script (no Conductor dependency).

What to measure

1. True per-request forwarded-duration cap — hold a single chunked response
   open and record when the sessions endpoint cuts it off. Repeat with premium
   ingress and its request idle-timeout raised toward the 30-min max.
2. Stream durability — does steady chunked streaming survive ≥10 min without
   a drop?
3. Resumability — can a dropped stream be resumed by replaying from a
   Last-Event-ID cursor (AG-UI style)?

Decision rule

• One request reliably survives a full multi-minute turn → Branch S
  (streaming_events=True, real interrupt).
• Request is capped short → Branch P (streaming_events=False,
  hard-abort-only interrupt via stopSession).

Deliverables / acceptance criteria

• The probe script + a short throwaway Dockerfile / pool setup (or scripted az commands).
• Measured per-request cap (default ingress and premium with idle-timeout raised).
• Pass/fail on ≥10-min stream durability.
• Finding on Last-Event-ID resume feasibility.
• A written recommendation — Branch S or Branch P — with the resulting
  streaming_events / interrupt capability values, folded back into
  docs/aca-provider.design.md (DD3 + capabilities table).

References

• Use dynamic sessions
• Session pools (Timed cooldown vs OnContainerExit)
• Premium ingress (request idle timeout: default 4 min, max 30 min)
• AG-UI protocol (resumable event streams)

[jrob5756]

Starting to take a look at this.

[jrob5756]

Phase 0 spike results: transport branch resolved — Branch S (streaming)

Ran the full measurement suite (scaffold in spikes/aca-transport/ on branch
feature/312-aca-transport-spike) against a real ACA custom-container session
pool (West US 2; East US hit a transient AKSCapacityHeavyUsage regional
capacity error, unrelated to this spike).

Environment

Field	Value
Region	West US 2
Pool	CustomContainer, 0.5 CPU / 1.0Gi memory, cooldown 600s
Ingress	default (not premium)
CLI	az 2.82.0 / containerapp 1.3.0b4

Measurements

Test	Result
cap (requested 1800s)	Cut off at 1801.18s
cap disambiguation (requested 2400s)	Cut off at 1801.17s — same mark
durability (900s)	✅ durable, steady ~1s frame gaps, clean terminal
resume (drop @60s, Last-Event-ID)	✅ resume_ok, no gap/dup
poll (submit + poll)	✅ poll_ok, 301 frames / 146 polls, zero gaps

True per-request cap ≈ 1801s (~30 minutes), on default ingress. The first
cap run's cutoff sat suspiciously close to its own requested duration=1800, so
we re-ran with duration=2400 to rule out coincidence. The second run was cut off
at the same ~1801s mark despite requesting 40 minutes — this confirms a real,
reproducible per-request cap of ~30 minutes, and notably it held on default
ingress; premium was not required to reach it.

Recommendation

Branch S (single streaming request) — the measured ~30-minute cap is
comfortably above the expected length of a single agent turn ("minutes", per
NFR4), and Branch S gives true incremental events plus real interrupt semantics
that Branch P cannot match.

Resulting capability values for docs/aca-provider.design.md (DD3 + capabilities
table):

Capability	Value
streaming_events	True
interrupt	True (real, via the in-flight stream)

Caveat: a turn running longer than ~30 minutes wall-clock will hit this cap.
The resume test shows Last-Event-ID resume works cleanly in principle, so the
recommended long-term design is Branch S as the primary transport, with
automatic reconnect-and-resume (via a bounded server-side event log) as the
mitigation for turns that exceed the cap, rather than defaulting to Branch P.

Two scaffold bugs found while running (fixed on the branch)

• READY_SESSIONS=0 is rejected by ACA (SessionPoolInvalidReadySessionInstances
  — requires >=1); corrected the config default to 1.
• The bare python3 on macOS can resolve to an old LibreSSL-linked system Python
  that fails TLS handshakes against Azure endpoints; the measurement script now
  prefers python3.13/3.12/3.11.

Status

• All provisioned Azure resources were torn down (99-teardown.sh) after the
  measurements completed.
• Full write-up + raw JSON: spikes/aca-transport/RESULTS.md and
  spikes/aca-transport/results/*.json (commits 2f1ca20, 4b5574d on
  feature/312-aca-transport-spike; branch not yet pushed/PR'd).
• Next: fold streaming_events=True / interrupt=True into
  docs/aca-provider.design.md DD3 + the capabilities table, then close this
  issue.

[jrob5756]

Resolved by the Phase 0 spike: measurements confirm Branch S (streaming) is viable — a real, reproducible per-request cap of ~30 minutes was measured on default ingress (confirmed via a disambiguation run), well above expected agent-turn length. Durability, resume (Last-Event-ID), and submit+poll were all also verified working cleanly. Full results: #312 (comment). Resulting capability values (streaming_events=True, interrupt=True) are being folded into docs/aca-provider.design.md DD3 + the capabilities table next, tracked under the parent epic #284.