Skip to content

Commit 7522fd2

Browse files
committed
fix(workflows): read the service wire on a detached start
Every durable approval continuation reported "control delivery unreachable" with "Workflow service emitted an unknown record before detached start", while the runner admitted the continuation and ran the turn to completion. Two executions reached terminal/completed and still carried continuation_delivery_failed. The strict start parser required the first NDJSON record to carry kind "event" or a successful kind "result". That is the RUNNER's vocabulary. The API does not call the runner. It calls the deployed workflow service, which re-frames every runner record as an agenta event, {"type", "data"}, with no kind field anywhere. So the strict branch rejected the first record of every continuation, always. _stream_service_started now accepts the first JSON object as the started handshake. A new _detached_start_failure rejects only an explicit failure frame, and reads one in either vocabulary: the service's {"type": "error"}, and the runner's {"kind": "result", "result": {"ok": false}} where a deployment forwards runner records verbatim. The tests replay the record sequences from the browser pass of 2026-09-04. Claude-Session: https://claude.ai/code/session_0164kzT6ttwpBtzvcDC6YzYk
1 parent ef9c028 commit 7522fd2

2 files changed

Lines changed: 134 additions & 21 deletions

File tree

api/oss/src/core/workflows/service.py

Lines changed: 57 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -756,12 +756,17 @@ async def _stream_service_started(
756756
) -> WorkflowServiceDetachedResponse:
757757
"""Stream the service ``/invoke`` and return on the FIRST record (the started handshake).
758758
759-
The runner emits NDJSON ``{"kind": "event"|"result", ...}`` records the moment each is
760-
built; the first one means the run is accepted and owned (the alive-held handshake). We
761-
return then and close the connection — the runner owns the run (alive watchdog) and
762-
persists independently (producer-driven ingest), so draining to completion is unnecessary.
763-
The read timeout is generous (sandbox cold-start can take seconds); we are NOT awaiting
764-
the whole run, so it is not the batch 60s-whole-run budget.
759+
The deployed workflow service emits one NDJSON record the moment each is built; the first
760+
one means the run is accepted and owned (the alive-held handshake). We return then and
761+
close the connection — the runner owns the run (alive watchdog) and persists independently
762+
(producer-driven ingest), so draining to completion is unnecessary. The read timeout is
763+
generous (sandbox cold-start can take seconds); we are NOT awaiting the whole run, so it
764+
is not the batch 60s-whole-run budget.
765+
766+
``strict_first_record`` (a durable control command) surfaces an explicit failure frame
767+
instead of reporting it as a start. It does NOT require a particular record shape: see
768+
``_detached_start_failure`` for why the two producers on this stream disagree about the
769+
vocabulary, and why anything unrecognised is a start.
765770
"""
766771
headers = inject(
767772
{
@@ -813,22 +818,12 @@ async def _stream_service_started(
813818
raise WorkflowDetachedStartFailed(
814819
"Workflow service emitted a non-object record before detached start."
815820
)
816-
kind = record.get("kind") if isinstance(record, dict) else None
817-
if strict_first_record and kind == "result":
818-
result = record.get("result")
819-
if not isinstance(result, dict) or result.get("ok") is not True:
820-
detail = (
821-
result.get("error")
822-
if isinstance(result, dict)
823-
else "malformed result record"
824-
)
821+
if strict_first_record and isinstance(record, dict):
822+
failure = WorkflowsService._detached_start_failure(record)
823+
if failure is not None:
825824
raise WorkflowDetachedStartFailed(
826-
f"Workflow service rejected detached start: {detail}"
825+
f"Workflow service rejected detached start: {failure}"
827826
)
828-
elif strict_first_record and kind != "event":
829-
raise WorkflowDetachedStartFailed(
830-
"Workflow service emitted an unknown record before detached start."
831-
)
832827
record_run_id = (
833828
record.get("run_id") if isinstance(record, dict) else None
834829
)
@@ -844,6 +839,48 @@ async def _stream_service_started(
844839
"Workflow service closed the stream before emitting a started record."
845840
)
846841

842+
@staticmethod
843+
def _detached_start_failure(record: dict) -> Optional[str]:
844+
"""Read an explicit failure out of the FIRST record, in either wire vocabulary.
845+
846+
Two producers can answer this stream and they do not share a vocabulary.
847+
848+
* The deployed workflow SERVICE is the ordinary case. It streams agenta event frames,
849+
``{"type": ..., "data": {...}}``, and its failure frame is ``{"type": "error"}``. There
850+
is no ``kind`` anywhere on that wire.
851+
* The agent RUNNER's own NDJSON, ``{"kind": "event"|"result"}``, reaches this parser only
852+
where a deployment forwards the runner stream verbatim. Its failure is a terminal
853+
``{"kind": "result", "result": {"ok": false}}``.
854+
855+
Everything else is the started handshake. Rejecting an unrecognised record instead is what
856+
made EVERY durable continuation report `unreachable` while the runner was in fact already
857+
running the turn: the service's first frame carries ``type``, never ``kind``.
858+
859+
A runner that refuses a continuation outright never reaches here at all. The SDK turns its
860+
``ok: false`` result into an exception inside the already-committed ASGI response, so the
861+
service closes the stream having written nothing, and the caller raises the
862+
"closed the stream" failure above.
863+
"""
864+
if record.get("kind") == "result":
865+
result = record.get("result")
866+
if isinstance(result, dict) and result.get("ok") is True:
867+
return None
868+
detail = (
869+
result.get("error")
870+
if isinstance(result, dict)
871+
else "malformed result record"
872+
)
873+
return str(detail or "the runner rejected the run")
874+
875+
if record.get("type") == "error":
876+
data = record.get("data")
877+
message = data.get("message") if isinstance(data, dict) else None
878+
code = data.get("code") if isinstance(data, dict) else None
879+
detail = str(message or "the service reported an error")
880+
return f"{detail} ({code})" if code else detail
881+
882+
return None
883+
847884
@staticmethod
848885
def _coerce_invoke_response(
849886
*,

api/oss/tests/pytest/unit/workflows/test_invoke_detached.py

Lines changed: 77 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,8 @@ async def test_stream_service_started_raises_on_http_error():
134134
"[]",
135135
'{"kind": "result", "result": {"ok": false, "error": "rejected"}}',
136136
'{"kind": "result"}',
137-
'{"kind": "unknown"}',
137+
# The service wire's own failure frame (an agenta `error` event).
138+
'{"type": "error", "data": {"type": "error", "message": "no key", "code": "auth"}}',
138139
],
139140
)
140141
async def test_stream_service_started_rejects_failure_or_malformed_first_record(line):
@@ -164,6 +165,81 @@ async def test_stream_service_started_keeps_legacy_best_effort_for_ordinary_trig
164165
assert result.run_id == "run-x"
165166

166167

168+
@pytest.mark.parametrize(
169+
"line",
170+
[
171+
# The record sequence a durable continuation really produced (browser pass,
172+
# 2026-09-04 17:35Z, session d99f32ae / command 01a06d7d): the runner admitted the
173+
# continuation and its first event was a `tool_call`. The DEPLOYED SERVICE re-frames
174+
# every runner record as an agenta event, `{"type", "data"}` — there is no `kind` on
175+
# that wire, and reading the first frame as a runner record called every one of those
176+
# deliveries unreachable while the turn ran to completion underneath the card.
177+
'{"type": "tool_call", "data": {"type": "tool_call", "id": "t1", "name": "Bash"}}',
178+
'{"type": "interaction_response", "data": {"type": "interaction_response"}}',
179+
'{"type": "message", "data": {"type": "message", "text": "ok"}}',
180+
# An unrecognised record is a start, not a failure: only an explicit failure frame is.
181+
'{"kind": "unknown"}',
182+
'{"type": "error_recovered", "data": {}}',
183+
],
184+
)
185+
async def test_stream_service_started_accepts_a_service_event_frame_as_the_start(line):
186+
response = _FakeStreamResponse(lines=[line, '{"type": "done", "data": {}}'])
187+
with patch("httpx.AsyncClient", return_value=_FakeAsyncClient(response)):
188+
result = await _service()._stream_service_started(
189+
url="http://svc/invoke",
190+
credentials="Secret tok",
191+
payload={},
192+
run_id="run-x",
193+
strict_first_record=True,
194+
)
195+
assert result.accepted is True
196+
assert result.run_id == "run-x"
197+
assert response.consumed == 1
198+
199+
200+
async def test_stream_service_started_reports_a_runner_refusal_verbatim():
201+
"""Case (b) of the same browser pass, command 01a06d7a.
202+
203+
The runner refuses a continuation it cannot prove it owns and writes
204+
``{"kind": "result", ok: false}``. Where a deployment forwards that record verbatim the
205+
caller must surface the reason, not report a start.
206+
"""
207+
refusal = (
208+
'{"kind": "result", "result": {"ok": false, "error": '
209+
'"Continuation could not establish alive ownership; retry delivery."}}'
210+
)
211+
response = _FakeStreamResponse(lines=[refusal])
212+
with patch("httpx.AsyncClient", return_value=_FakeAsyncClient(response)):
213+
with pytest.raises(WorkflowDetachedStartFailed) as failure:
214+
await _service()._stream_service_started(
215+
url="http://svc/invoke",
216+
credentials="Secret tok",
217+
payload={},
218+
run_id="run-x",
219+
strict_first_record=True,
220+
)
221+
assert "alive ownership" in str(failure.value)
222+
223+
224+
async def test_stream_service_started_reports_an_empty_stream_as_a_failed_start():
225+
"""The same refusal as it actually reaches the API through the SDK service.
226+
227+
The SDK turns the runner's ``ok: false`` result into an exception inside an ASGI response
228+
whose 200 is already committed, so the service closes the stream having written nothing.
229+
"""
230+
response = _FakeStreamResponse(lines=[])
231+
with patch("httpx.AsyncClient", return_value=_FakeAsyncClient(response)):
232+
with pytest.raises(WorkflowDetachedStartFailed) as failure:
233+
await _service()._stream_service_started(
234+
url="http://svc/invoke",
235+
credentials="Secret tok",
236+
payload={},
237+
run_id="run-x",
238+
strict_first_record=True,
239+
)
240+
assert "closed the stream" in str(failure.value)
241+
242+
167243
async def test_stream_service_started_accepts_success_result_record():
168244
response = _FakeStreamResponse(
169245
lines=['{"kind": "result", "result": {"ok": true}}'],

0 commit comments

Comments
 (0)