Improve Workflow Runtime Observability with Execution Status and Flow Control Visualization

[yrenat]

Motivation

Texera allows users to design and execute complex workflows, but during execution it is still difficult to understand where data flow is being constrained. When a workflow becomes slow or appears stuck, users may want to know:

• Which part of the workflow is currently under pressure?
• Is an operator slow because it is processing data, waiting for upstream input, or blocked by downstream backpressure?
• Which workflow edge is accumulating pressure?
• Is the bottleneck caused by data flow between operators rather than by the operator state alone?

Texera already exposes multiple operator-level runtime information, such as execution state and operator statistics. However, flow-control/backpressure information is currently kept mostly inside the execution engine and is not visible in the workflow graph to the users.

This proposal focuses on exposing workflow-level flow-control status so users can better understand runtime bottlenecks without changing the underlying execution behavior.

Proposed Solution

This proposal focuses on exposing workflow-level flow-control status by reusing Amber’s existing credit-based flow-control state.

Amber already tracks per-channel flow-control information, such as available credit, queued credit, and whether a channel is overloaded. The backend should periodically summarize this channel-level state into workflow-level link pressure information and publish it through the existing workflow websocket event mechanism.

The first version is designed to expose a compact runtime event containing, for each affected logical workflow link:

• Source operator id.
• Target operator id.
• Target input port id.
• Whether the link is currently overloaded.
• Current/max credit usage, or utilization percentage.
• Optional aggregated queued/inflight byte counts.

Because physical execution may use multiple worker-to-worker channels for one logical workflow edge, the backend should aggregate channel-level signals into one logical-link status before sending it to the frontend.

The frontend can then render this as a runtime overlay on the workflow graph. Edges under pressure can be highlighted by color, with compact labels or tooltips showing utilization details.

Backend Design

The backend will reuse the existing flow-control implementation instead of introducing a separate monitoring system.

The expected backend work includes:

1. Collect flow-control snapshots from existing runtime state.

   Amber already maintains per-channel flow-control state internally. The implementation should expose a read-only snapshot of relevant values, such as:

   • available credit
   • maximum credit
   • queued credit
   • overloaded/backpressured status

2. Aggregate physical channel state into logical workflow-link state.

   Flow control is tracked at the worker-channel level, while the frontend visualizes logical workflow links. Therefore, the backend should group channel-level information by logical workflow connection, such as:

   worker channel metrics
     -> source logical operator + target logical operator + target input port
     -> logical link pressure status
   
   

3. Publish a compact websocket event.

   A possible event shape is:

   FlowControlStatusUpdateEvent {
     links: {
       sourceOperatorId: string;
       targetOperatorId: string;
       targetPortId: string;
       overloaded: boolean;
       queuedCreditBytes?: number;
       availableCreditBytes?: number;
       maxCreditBytes?: number;
       utilization?: number;
     };
   }
   
   

Frontend Design

The frontend will use the new flow-control status event and render it as a workflow graph overlay.

The first version may have a focus on edge-level visualization:

• Use edge color to indicate pressure severity.

• Show compact edge labels only when useful, such as when pressure is non-zero or when a runtime/debug overlay is enabled.

• Show detailed values, such as utilization percentage or queued bytes, in a tooltip or selected-edge details panel.

• Clear the visualization when execution completes, fails, is killed, or the websocket connection closes.

The visualization should avoid overloading existing operator-state colors. Operator status is already used to show execution state, so flow-control pressure should primarily be shown on workflow edges.

Related Issues

• Support flow control (backpressure) visualization in backend #4321: Support flow control/backpressure visualization in backend.
• Support flow control (backpressure) visualization in frontend #4322: Support flow control/backpressure visualization in frontend.

Discussions are welcomed

1. Should the flow-control status be exposed as a new event in the existing
   workflow websocket mechanism?

2. What exact fields should be included in the flow-
   control status event?

3. Should the frontend always show pressure labels, or should labels only
   appear when the runtime/debug overlay is enabled?

4. What update frequency is appropriate to balance responsiveness and
   overhead?

5. Any other comments/suggestions/thoughts

[chenlica]

@yrenat Thanks for sharing these thoughts. Some high-level questions: should we finish the other two issues first? How are new ideas related to those two issues?

[yrenat]

Yes of course. Indeed that's what this discussion is for. The two existing issues already define the high-level goal and divide the work into backend and frontend parts, and my discussion is intended to further clarify the implementation-level design, including control body and impacted code blocks. I want to follow the common practice of "issue->discussion->PR" and I also want to clarify and verify my implementation design before I proceed.