Repository navigation
Replies: 1 comment 1 reply
|
The version check covers a timeout that fires after the approval landed, but the reverse order needs the same care: an approver who clicks approve at minute 31, after the |
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
1. Motivation and Goals
Flink Agents uses Flink's native timers to support delayed execution, covering the following use cases:
This design supports both processing-time and event-time timers. A processing-time timer can continue execution within the current agent run or start a new run. An event-time timer starts a new run. Waiting within the current run retains its context and prevents subsequent inputs with the same key from being processed until the run finishes. Starting a new run allows the original run to finish and subsequent inputs to proceed. Neither mode occupies an Action execution thread while waiting. Delayed tasks use Flink state management and the recovery mechanisms of Flink Agents.
2. Comparison of Two Designs
When a Flink timer fires, it invokes a callback on
ActionExecutionOperator. Flink Agents must then determine which user function to execute. The integration therefore needs to define how users register callbacks and how the runtime dispatches to the corresponding functions.Both designs below use native timers. They differ in whether they introduce a separate mechanism for registering and dispatching user callbacks or reuse the existing Event → Action mechanism.
2.1 Option 1: Expose the Flink Timer API with User Callback Registration and Dispatch
Expose Flink's native Timer API to user Actions and provide an entry point for registering user callbacks. Users register and delete timers within Actions and implement delayed business logic in the registered callbacks.
This design requires two additional mechanisms:
This design is close to Flink's native programming model, but requires new callback registration and dispatch mechanisms. To use Memory, durable execution, and scheduling for the same key in Flink Agents, callbacks must also be integrated with the corresponding Action execution context.
2.2 Option 2: Provide a Flink Agents Timer API That Delivers Events
Flink Agents provides APIs to register and cancel timers. Users implement delayed business logic as ordinary Actions and specify the business Event to deliver when registering a timer. The underlying implementation still uses Flink's native timers.
This design has two stages: defining Actions and registering timers.
Stage 1: Register Actions when defining the Agent and building the Agent plan.
Use the existing Action registration mechanism to establish Event-to-Action bindings in advance. For example:
At this stage, the timeout cleanup Action is registered and bound to the
ApprovalTimeoutEvent type. Different approval requests create different instances of that Event type, all handled by the same Action definition.Stage 2: Register a specific delayed Event while the Agent is running.
The Action that requests approval calls
register_timer()to deliverApprovalTimeout(request_id, version)after 30 minutes. It selectsTimerMode.TRIGGER_AFTER_CURRENT_RUNto start a new run for that Event. The Action definition and Event type binding are already established; the specific Event, trigger time, and execution mode are chosen at runtime.Processing-time timers support both execution modes. Event-time timers always start a new run.
Flink handles time and timer scheduling. Flink Agents provides the API, stores Events, and schedules the corresponding Actions when timers fire.
Delayed business logic uses the same Memory, durable execution, scheduling, and tracing capabilities as ordinary Actions. Business parameters are stored in serializable Events or persistent state that subsequent Actions can read.
2.3 Comparison and Decision
This design adopts Option 2, reusing the existing Event-to-Action binding, dispatch, and execution mechanisms.
3. Initial Scope
The initial version supports:
4. API Design and Usage Examples
4.1 API Definitions
Add the following APIs to
RunnerContext.timer_ididentifies a task for replacement and cancellation, andeventspecifies the business Event to deliver when the timer fires.registerTimer/register_timerandregisterTimerAt/register_timer_atuse processing time.registerEventTimeTimer/register_event_time_timeruse event time.Java:
Python:
The
modeparameter has typeTimerModeand applies only to processing-time timers. It specifies how the delayed task relates to the current run:CURRENT_RUN_WAIT_FOR_TRIGGERTRIGGER_AFTER_CURRENT_RUNThe event-time API has no
modeparameter. Its Events always start new runs. If the current run waited for an event-time timer while watermark advancement waited for that run to finish, the two would wait for each other. This combination is therefore not supported.Java and Python use epoch milliseconds internally. Absolute times in Python must be timezone-aware. Deadlines with sub-millisecond precision are rounded up to the next millisecond. Negative delays, empty timer IDs, timestamp overflow, and non-serializable Events are rejected. For processing-time timers, a zero delay or a deadline in the past makes the task eligible for scheduling after the registration is committed. Event-time timers become due according to the watermark. Business Actions are executed by the scheduler, never synchronously on the API call stack.
4.2 Usage Examples
4.2.1 Approval Timeout
The Agent has registered Actions for requesting approval, handling approval responses, and handling
ApprovalTimeout.The Action that requests approval saves the request state and registers a timeout Event:
TRIGGER_AFTER_CURRENT_RUNallows the current run to finish normally so that the user's approval response can be processed as a subsequent input with the same key.The Action that handles the approval response updates the request state and cancels the timer:
The Action handling
ApprovalTimeoutperforms cleanup only if the request still exists, is awaiting approval, and has a matching version. Cancellation does not retract an Event that has already been enqueued, so the Action must still check the business state.4.2.2 Periodic Status Checks
Consider checking an external job's status every minute. Register the following bindings when building the Agent plan:
Both Event types carry
job_id. The following code shows the two Action bodies, omitting Event class definitions and Agent registration.query_job_status(job_id)is a user-provided function that queries the external job and returns a status string.Each timer triggers one check. After checking the status, the Action decides whether to register another timer. No run is kept waiting between checks, and other inputs with the same key can be processed normally.
job-status:{job_id}identifies the observation task for that job; each subsequent check reuses the same timer ID.For example, if the first check starts at 10:01:00 and the Action completes at 10:01:05, committing the next timer, the next check can start no earlier than 10:02:05. This means waiting one minute after the previous check completes. It is not a fixed-rate schedule and does not replay every missed interval after downtime.
4.2.3 Delayed Execution Within the Same Run
Use
CURRENT_RUN_WAIT_FOR_TRIGGERwhen a business operation needs to continue after ten seconds and still needs the current run's context. BindContinueWorkto the Action that continues the operation when building the Agent plan:The timer takes effect when the Action completes. The execution thread is released, while the current run retains its context and Sensory Memory. After ten seconds,
ContinueWorktriggers the corresponding Action within the original run. The run finishes only after all subsequent Actions complete and no other timers belonging to that run remain pending. The next input with the same key can then start processing.4.2.4 Event-Time Execution
Use an event-time timer when business logic needs to check state after event time reaches a deadline. In this example,
event.occurred_atis a timezone-aware business timestamp carried by the input, andCheckAtDeadlineis already bound to a state-checking Action:If the business timestamp is 10:00, the timer deadline is 10:05. After the original run completes, the runtime fires the timer when processing a watermark at or beyond 10:05, and the Event starts a new run. The wall clock reaching 10:05 does not itself fire the timer; the input stream must produce watermarks.
5. Behavioral Contract
(Flink Agents operator, business key, timer_id). All time domains and execution modes share this ID space.CURRENT_RUN_WAIT_FOR_TRIGGERkeeps the original run open and delivers the Event to that run when the timer fires. Cancellation and replacement update the timers that the run is waiting for.TRIGGER_AFTER_CURRENT_RUNand event-time timers do not require the original run to remain open. Their Events enter the normal run queue for the same key and do not preempt the current run.Multiple timer operations from one Action are applied in call order when the Action completes successfully. For example, registering and then canceling a timer leaves no pending registration. The runtime checks whether the run can finish only after applying the Action's complete result.
Event payloads and attachments are retained until the delayed task finishes. They are not removed when the registering run's Sensory Memory is cleaned up. Timer registration state does not use the TTL of Short-Term Memory.
6. Runtime Integration
6.1 Operator State and Timer Service
Add the following two components to
ActionExecutionOperator.1. Keyed state for timer registrations.
Create a
MapState<String, TimerRegistration>using aMapStateDescriptor:This state is isolated by business key, so different keys can use the same timer ID. Event-time registrations always use
TRIGGER_AFTER_CURRENT_RUNinternally and are not associated with the original run.2. Flink's native
InternalTimerServicefor tracking time and invoking timer callbacks.The operator implements
Triggerableand obtains the timer service during initialization:The string
timer_idserves as the namespace, andthisrefers to the operator. Flink invokes the operator'sonProcessingTime()when a processing-time timer fires andonEventTime()when watermark advancement makes an event-time timer due. Both callbacks look up the business Event in keyed state.The timer service and registration state are linked by business key and timer ID. For example, registering
approval:123foruser-1produces:InternalTimerService(key=user-1, namespace=approval:123, deadline).registrationsuser-1:approval:123 → {time_domain, deadline, mode=TRIGGER_AFTER_CURRENT_RUN, event=ApprovalTimeout(...)}.Events that start new runs reuse the existing
pendingInputEventsqueue. This queue currently buffersInputEventinstances created from external inputs, usingListState<Event>as its underlying type. This design extends it to also accept business Events produced by timers. ForCURRENT_RUN_WAIT_FOR_TRIGGER, the Event enters the original run's Action scheduling flow directly, rather than the input queue for new runs.6.2 Registration, Replacement, and Cancellation
When an Action completes successfully, the runtime calculates deadlines for relative delays and preserves user-specified absolute deadlines. A
CURRENT_RUN_WAIT_FOR_TRIGGERregistration also records the current run's ID. The runtime then saves the registration under the corresponding business key and invokes the native API for its time domain:Replacing a registration follows this sequence: delete the old native timer → replace the registration record → register the new native timer. The runtime uses the old record's time domain and deadline to select the appropriate deletion API. Events that have already entered the execution flow are unaffected.
To cancel a timer, the runtime retrieves its time domain and deadline from
registrations, deletes the corresponding native timer, and removes the registration record. If the record does not exist, no action is required. Replacement and cancellation also update the timers that affected runs are waiting for. Run completion is checked after the Action's complete result has been applied.6.3 Timers Within the Same Run
The underlying timer remains a native Flink timer. This mode changes when the run can finish and where the Event is delivered.
Previously, a run could finish once all its Actions had completed. With
CURRENT_RUN_WAIT_FOR_TRIGGER, the run can finish only when it has no queued or executing Actions and no pending timers belonging to it. The runtime usesowner_run_idin the registration records to track the timers each run is waiting for, and rebuilds these relationships from the records during recovery.The Action that registers a timer can complete normally and release its execution thread. While waiting, the original run retains its context and Sensory Memory. Subsequent external inputs with the same key remain queued.
The Event must enter the original run's Event → Action scheduling flow. If it were placed in
pendingInputEvents, it would wait for the original run to finish while that run waited for the Event, preventing progress.When a timer fires, the runtime first adds its Event to the original run's pending work, then removes the timer registration, and then checks whether the run can finish. Cancellation releases the corresponding wait. Replacement adjusts the wait according to the new registration's execution mode. A
TRIGGER_AFTER_CURRENT_RUNtimer that has not yet fired does not keep the original run open.6.4 Scheduling and Executing Events from Timers
When a timer fires, Flink provides its key, namespace, and deadline, and sets the corresponding current key. The runtime finds the registration using the namespace, checks its time domain and deadline, and dispatches according to the execution mode:
The callback only updates state and schedules work. The Action executor handles the business logic. The Event retains its registered business type, such as
ApprovalTimeout, and the Agent plan's bindings identify the corresponding Action.New runs execute in queue order for each key. For example, if input A is being processed, input B is already queued, and a timer produces Event T for a new run, the execution order is A → B → T. An Event using
CURRENT_RUN_WAIT_FOR_TRIGGERis part of A's subsequent work and executes before A finishes.Actions triggered by timers follow the existing concurrency limits. Work that cannot execute immediately remains in its pending execution state. Waiting within the current run retains run state without occupying an Action execution thread.
All timer state operations execute on the operator thread. Action API calls record operations, which the operator applies when committing the Action result. Timer callbacks access registration records and pending work on the same thread. The existing execution lock protects current-key switching and state access.
Java and Python share the same timer service. The Python API passes registration and cancellation operations to the Java operator. Business Events are stored using the existing cross-language serialization mechanism and dispatched through the existing Java or Python Action execution path.
6.5 Event Time and Watermarks
Event-time timers are driven by upstream watermarks. When Flink Agents receives a watermark, it fires the timers that are due, waits for the business processing triggered by those timers to finish, and then forwards the watermark downstream.
6.5.1 Processing a Watermark
Suppose the operator receives
Watermark = 100and an event-time timer with deadline80has already been registered. Processing follows this sequence:The first step uses arrival order at the operator, not the inputs' event timestamps. If those inputs belong to runs waiting for processing-time timers in
CURRENT_RUN_WAIT_FOR_TRIGGERmode, those runs must also finish first.When the internal Timer Service has processed watermark 100, run A may still be executing, so watermark 100 cannot yet be forwarded. The runtime retains the watermark awaiting emission and tracks the runs it triggered until their work is complete. An event-time timer with deadline 200 is not yet due and does not need to be awaited.
6.5.2 Registering Another Timer That Is Already Due
Continuing the example, suppose an Action in run A registers an event-time timer with deadline
90when it completes. The internal Timer Service has already processed watermark 100, so the new timer is already due.Flink Agents must handle this explicitly. Flink's native
registerEventTimeTimer()only inserts the timer into a queue. Timers fire within the watermark-processing loop. By the time an asynchronous Action completes, the original loop has already returned, so registration alone does not immediately invoke a callback. Flink implementationAfter committing all timer operations from the Action, if a newly registered timer has a deadline no greater than the watermark currently being processed, the runtime runs the internal Timer Service's due-timer check again with that same watermark on the operator thread:
Watermark 100 remains pending throughout this process. If subsequent Actions register more timers with deadlines no greater than 100, the process repeats. Watermark 100 can be forwarded only after the due-timer check has completed and all triggered runs have finished. Both the check and the runs it triggers must be included in the completion condition; finishing run A alone must not cause the watermark to be forwarded early.
6.5.3 Output Timestamps
For a run started by an ordinary input,
sourceTimestampstores the input record's event time. This timestamp propagates through the Actions and Events triggered by that input and ultimately becomes the output record's event time. If the input record has no timestamp,sourceTimestampis null.A
CURRENT_RUN_WAIT_FOR_TRIGGERtimer continues within the original run and therefore preserves itssourceTimestamp. A new run started by a timer has no corresponding new input record, so its timestamp is determined by the timer type:sourceTimestampand the output record's event timeCURRENT_RUN_WAIT_FOR_TRIGGERtimer continuing within the original runsourceTimestampnull and do not assign event time to the output record.The timestamp determines the event time associated with the result. The scheduling sequence in the preceding sections ensures that results are emitted before the watermark. If watermark 100 has already been forwarded and a timer with deadline 80 is registered afterward, its results are treated as late data; the downstream watermark is not moved backward.
7. Recovery and Durable Execution
Flink restores native timers and registration records from checkpoints. Events already handed to runs or pending queues use the existing recovery flow. This section describes how Flink Agents records and replays timer operations.
7.1 Recording and Applying Timer Operations
Registration and cancellation API calls only record operations. When an Action completes successfully, the runtime commits these operations in call order. Intermediate yields or awaits do not register or cancel native timers.
For a relative delay, the runtime calculates
deadline = currentProcessingTime + delaywhen first committing the registration. Absolute-time registration uses the user-specified deadline directly. When ActionState persistence is enabled, timer operations and their resolved time domains, deadlines, execution modes, Events, and Event IDs are saved in the completed Action's result before being applied to managed keyed state andInternalTimerService. Operations usingCURRENT_RUN_WAIT_FOR_TRIGGERalso save the original run's identity.For example, an Action calls
register_timer(delay=timedelta(minutes=30), ...)at 10:00, then completes successfully and commits the registration at 10:10. The deadline is 10:40. Recovery replay continues to use 10:40.7.2 Restoring Timer Operations When Reusing Completed Actions
An Action result may already have been written to the ActionState journal while its timer registrations or cancellations are not yet included in the latest completed checkpoint. During recovery, Flink Agents reuses the completed Action result without executing the Action's code again.
Like
MemoryUpdate, timer operations are saved as part of the Action result and reapplied when that result is reused. Recovery replays Memory updates, emitted Events, and timer operations. Replaying timer operations updates both the registration records in managed keyed state and the native timers inInternalTimerService.For example, if an Action registered an approval timeout timer before completing, recovery restores the saved Event and deadline and registers the native timer even though it skips the Action's code. A cancellation operation removes the corresponding registration record and native timer. Replay uses the saved deadline, execution mode, Event, and Event ID; it does not recalculate the delay or create a new Event. Operations follow the recovery order of Action results and the original call order within each Action.
For
CURRENT_RUN_WAIT_FOR_TRIGGER, recovery restores the original run's context and pending work, then rebuilds the timers it is waiting for from the registration records. Recovery scheduling first applies the results of Actions already present at the recovery point, including their timer operations, and then dispatches Events from due timers. These Events continue within the original run; processing them must not wait for the entire original run to finish. If a registration has already transitioned from a timer to a pending Event, recovery follows the Event's existing execution state.7.3 Retrying Incomplete Actions
If an Action fails before completing successfully, its recorded timer operations have not taken effect. Recovery re-executes the Action and applies its timer operations only after successful completion. Relative delays are calculated from that commit time.
When ActionState persistence is disabled, Actions executed after the checkpoint may also run again, and relative deadlines are calculated from the new commit time. If a business operation requires a fixed deadline, obtain an absolute time from the input or persistent state and use
register_timer_at(). The event-time API already uses absolute time.All reactions