Skip to content

Feature Request: Try/Catch/Finally control blocks for workflows #7387

Description

@carloea2

Feature Summary

Image Image Image

Today a runtime failure in any operator (a Python UDF raising, a filter
referencing a missing attribute, a flaky external call) has no recovery path:
the workflow reports the error and pauses, and the only options are manual
retry or abandoning the run. There is no way to express "attempt this
subgraph; if anything in it fails, run this fallback instead", the
control-flow vocabulary has If for data-driven branching but nothing for
failure-driven branching.

This feature adds block-level try/catch/finally semantics, PL-faithful,
as two new control-block operators:

  • TryCatch, guards a subgraph (the try branch). If any operator inside
    it fails, the frame's input is replayed from a snapshot through the catch
    branch. A third Error Info output port emits one row per caught failure
    (errorType, message, operatorId, workerId) for auditing, alerting, or
    routing the recovery by error type.
  • Finally, the reconvergence point: consumes both branches' tails and
    releases exactly one branch's complete output, the try side's on success,
    the catch side's on failure, through the output port named for the winner
    (Try Result / Catch Result), so downstream logic can also react to
    the outcome itself.

Use cases: fallback models when a primary model errors, replaying input
through a cheaper/safer path, quarantining bad batches while keeping the run
alive, catch (SpecificError)-style routing via Error Info + If, and audit
tables of caught failures.

Proposed Solution or Design

Semantics

  • The try cone (everything reachable from the Try port, up to the paired
    Finally) is one attempt. Any failure inside it aborts the attempt: no
    further user code runs on post-failure data, and the same input is replayed
    through the catch cone.
  • All-or-nothing reconvergence: Finally stages both sides and flushes
    only the winner, never a mix, never duplicates. From Catch depends on
    From Try, so the release decision is deterministic, never a race.
  • The outcome is part of the output: winner rows leave through
    Try Result or Catch Result; connecting both ports to one downstream
    input (a union) yields "the winner, whichever it was".
  • Nesting forms a tree: the innermost frame owns a failure; a caught
    failure never leaks past its frame; a double failure (catch branch fails
    too) escalates to the enclosing frame, or terminates the run with the
    usual console error at top level.
  • Failures outside any frame keep today's behavior exactly (console
    error + pause, current input retriable). Frames opt their subgraphs into
    drain-on-failure semantics; plans without frames are untouched.
  • Infrastructure failures (worker crash, node loss, OOM-kill) are
    deliberately not catchable, frames catch user-code errors.

Mechanism (minimal engine change)

One idea carries the design: failure becomes a dataflow event. A failing
worker broadcasts an ordinary State carrying a reserved __error__ key
(same convention as If's condition State), then drains. Ports still
complete, so the stream terminates instead of hanging. Everything else is
ordinary operators plus one compiler pass, the coordinator and scheduler are
untouched:

  • TryCatch expands to two physical operators: a splitter (live stream
    out the Try port + an eager snapshot toward the gate) and a catch gate,
    an If generalized to N conditions. The compiler pass synthesizes signal
    edges
    from every try-cone tail to the gate; a dedicated
    SignalPartitioning drops tuples at the sender, so signal edges carry only
    States and end-of-stream. The gate's snapshot port depends on all signal
    ports, which makes resolution timing structural: the snapshot is consumed
    only after the try side fully resolved.
  • Per-port drain contagion: when an __error__ State arrives on a port,
    it is delivered to the executor first (frame operators react; the default
    pass-through forwards it, that is escalation), then the port is poisoned:
    subsequent data is discarded without invoking the executor and its finish
    hooks are suppressed, while the port still completes normally.
  • Attribution follows the failing operator, not the data's destination:
    the gate/merger configs carry their frame's cone (baked at compile time)
    and act only on own-cone errors; foreign errors pass through to the
    enclosing frame.
  • Wiring rules validated at compile time with clear messages: try/catch
    cones disjoint; Finally inputs must come from the paired frame's cones;
    the cones may reach the post-frame region only through the Finally (no
    Merger bypass); Error Info may feed the catch cone or the post-frame
    region but never the frame's own try cone (structural cycle + temporal
    paradox).
Image

(1–8 = execution order. Green = allowed external wiring; red ✕ = rejected at
compile time; dotted = synthesized signal edges; dashed = materialized
snapshot; ↔ = Error Info and catch cone may interconnect.)

Affected Area

  • Workflow Engine (Amber)
  • Workflow UI

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions