Feature Summary
Today a runtime failure in any operator (a Python UDF raising, a filter
referencing a missing attribute, a flaky external call) has no recovery path:
the workflow reports the error and pauses, and the only options are manual
retry or abandoning the run. There is no way to express "attempt this
subgraph; if anything in it fails, run this fallback instead", the
control-flow vocabulary has If for data-driven branching but nothing for
failure-driven branching.
This feature adds block-level try/catch/finally semantics, PL-faithful,
as two new control-block operators:
- TryCatch, guards a subgraph (the try branch). If any operator inside
it fails, the frame's input is replayed from a snapshot through the catch
branch. A third Error Info output port emits one row per caught failure
(errorType, message, operatorId, workerId) for auditing, alerting, or
routing the recovery by error type.
- Finally, the reconvergence point: consumes both branches' tails and
releases exactly one branch's complete output, the try side's on success,
the catch side's on failure, through the output port named for the winner
(Try Result / Catch Result), so downstream logic can also react to
the outcome itself.
Use cases: fallback models when a primary model errors, replaying input
through a cheaper/safer path, quarantining bad batches while keeping the run
alive, catch (SpecificError)-style routing via Error Info + If, and audit
tables of caught failures.
Proposed Solution or Design
Semantics
- The try cone (everything reachable from the Try port, up to the paired
Finally) is one attempt. Any failure inside it aborts the attempt: no
further user code runs on post-failure data, and the same input is replayed
through the catch cone.
- All-or-nothing reconvergence: Finally stages both sides and flushes
only the winner, never a mix, never duplicates. From Catch depends on
From Try, so the release decision is deterministic, never a race.
- The outcome is part of the output: winner rows leave through
Try Result or Catch Result; connecting both ports to one downstream
input (a union) yields "the winner, whichever it was".
- Nesting forms a tree: the innermost frame owns a failure; a caught
failure never leaks past its frame; a double failure (catch branch fails
too) escalates to the enclosing frame, or terminates the run with the
usual console error at top level.
- Failures outside any frame keep today's behavior exactly (console
error + pause, current input retriable). Frames opt their subgraphs into
drain-on-failure semantics; plans without frames are untouched.
- Infrastructure failures (worker crash, node loss, OOM-kill) are
deliberately not catchable, frames catch user-code errors.
Mechanism (minimal engine change)
One idea carries the design: failure becomes a dataflow event. A failing
worker broadcasts an ordinary State carrying a reserved __error__ key
(same convention as If's condition State), then drains. Ports still
complete, so the stream terminates instead of hanging. Everything else is
ordinary operators plus one compiler pass, the coordinator and scheduler are
untouched:
- TryCatch expands to two physical operators: a splitter (live stream
out the Try port + an eager snapshot toward the gate) and a catch gate,
an If generalized to N conditions. The compiler pass synthesizes signal
edges from every try-cone tail to the gate; a dedicated
SignalPartitioning drops tuples at the sender, so signal edges carry only
States and end-of-stream. The gate's snapshot port depends on all signal
ports, which makes resolution timing structural: the snapshot is consumed
only after the try side fully resolved.
- Per-port drain contagion: when an
__error__ State arrives on a port,
it is delivered to the executor first (frame operators react; the default
pass-through forwards it, that is escalation), then the port is poisoned:
subsequent data is discarded without invoking the executor and its finish
hooks are suppressed, while the port still completes normally.
- Attribution follows the failing operator, not the data's destination:
the gate/merger configs carry their frame's cone (baked at compile time)
and act only on own-cone errors; foreign errors pass through to the
enclosing frame.
- Wiring rules validated at compile time with clear messages: try/catch
cones disjoint; Finally inputs must come from the paired frame's cones;
the cones may reach the post-frame region only through the Finally (no
Merger bypass); Error Info may feed the catch cone or the post-frame
region but never the frame's own try cone (structural cycle + temporal
paradox).
(1–8 = execution order. Green = allowed external wiring; red ✕ = rejected at
compile time; dotted = synthesized signal edges; dashed = materialized
snapshot; ↔ = Error Info and catch cone may interconnect.)
Affected Area
- Workflow Engine (Amber)
- Workflow UI
Feature Summary
Today a runtime failure in any operator (a Python UDF raising, a filter
referencing a missing attribute, a flaky external call) has no recovery path:
the workflow reports the error and pauses, and the only options are manual
retry or abandoning the run. There is no way to express "attempt this
subgraph; if anything in it fails, run this fallback instead", the
control-flow vocabulary has
Iffor data-driven branching but nothing forfailure-driven branching.
This feature adds block-level try/catch/finally semantics, PL-faithful,
as two new control-block operators:
it fails, the frame's input is replayed from a snapshot through the catch
branch. A third Error Info output port emits one row per caught failure
(errorType, message, operatorId, workerId) for auditing, alerting, or
routing the recovery by error type.
releases exactly one branch's complete output, the try side's on success,
the catch side's on failure, through the output port named for the winner
(Try Result / Catch Result), so downstream logic can also react to
the outcome itself.
Use cases: fallback models when a primary model errors, replaying input
through a cheaper/safer path, quarantining bad batches while keeping the run
alive,
catch (SpecificError)-style routing via Error Info + If, and audittables of caught failures.
Proposed Solution or Design
Semantics
Finally) is one attempt. Any failure inside it aborts the attempt: no
further user code runs on post-failure data, and the same input is replayed
through the catch cone.
only the winner, never a mix, never duplicates.
From Catchdepends onFrom Try, so the release decision is deterministic, never a race.Try ResultorCatch Result; connecting both ports to one downstreaminput (a union) yields "the winner, whichever it was".
failure never leaks past its frame; a double failure (catch branch fails
too) escalates to the enclosing frame, or terminates the run with the
usual console error at top level.
error + pause, current input retriable). Frames opt their subgraphs into
drain-on-failure semantics; plans without frames are untouched.
deliberately not catchable, frames catch user-code errors.
Mechanism (minimal engine change)
One idea carries the design: failure becomes a dataflow event. A failing
worker broadcasts an ordinary
Statecarrying a reserved__error__key(same convention as
If's condition State), then drains. Ports stillcomplete, so the stream terminates instead of hanging. Everything else is
ordinary operators plus one compiler pass, the coordinator and scheduler are
untouched:
out the Try port + an eager snapshot toward the gate) and a catch gate,
an
Ifgeneralized to N conditions. The compiler pass synthesizes signaledges from every try-cone tail to the gate; a dedicated
SignalPartitioningdrops tuples at the sender, so signal edges carry onlyStates and end-of-stream. The gate's snapshot port depends on all signal
ports, which makes resolution timing structural: the snapshot is consumed
only after the try side fully resolved.
__error__State arrives on a port,it is delivered to the executor first (frame operators react; the default
pass-through forwards it, that is escalation), then the port is poisoned:
subsequent data is discarded without invoking the executor and its finish
hooks are suppressed, while the port still completes normally.
the gate/merger configs carry their frame's cone (baked at compile time)
and act only on own-cone errors; foreign errors pass through to the
enclosing frame.
cones disjoint; Finally inputs must come from the paired frame's cones;
the cones may reach the post-frame region only through the Finally (no
Merger bypass); Error Info may feed the catch cone or the post-frame
region but never the frame's own try cone (structural cycle + temporal
paradox).
(1–8 = execution order. Green = allowed external wiring; red ✕ = rejected at
compile time; dotted = synthesized signal edges; dashed = materialized
snapshot; ↔ = Error Info and catch cone may interconnect.)
Affected Area