feat(engine): add jump-to-operator support by aglinxinyuan · Pull Request #4444 · apache/texera

aglinxinyuan · 2026-04-22T05:31:32Z

What changes were proposed in this PR?

Add a generic controller-side primitive for jumping execution to the region containing a target operator (JumpToOperatorRegion).

Design:

The schedule produced by WorkflowScheduler remains static; jump behavior does not live in WorkflowScheduler.
The execution-position state is owned by WorkflowExecutionCoordinator, which holds a mutable Schedule reference and rewrites the iteration tail when a jump is requested.
Schedule exposes O(1) getLevelIndexOfOperator lookup plus a rewriteExecutionFrom(levelIndex) data primitive; jump policy stays out of Schedule.
The initial schedule is wired into the coordinator via ControllerProcessor.updateExecutionSchedule(...) after WorkflowScheduler.updateSchedule(...), avoiding inline lambdas in ControllerProcessor.

Any related issues, documentation, discussions?

Closes #4443

Precursor test coverage for related modules (separate PRs against main):

test(amber): add unit test coverage for Schedule iterator semantics #4562 — ScheduleSpec covering Schedule iterator semantics
test(amber): add unit test coverage for WorkflowScheduler #4564 — WorkflowSchedulerSpec covering WorkflowScheduler contract

How was this PR tested?

sbt "WorkflowExecutionService/compile" — passes.
sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.architecture.scheduling.WorkflowExecutionCoordinatorSpec" — 1/1 tests pass.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: ChatGPT (Codex), Claude Code (Claude Opus 4.7)

Yicong-Huang · 2026-04-26T07:27:36Z

I think we should let the coordinator jump back to a stage, rather than "schedule" or "scheduler".

Signed-off-by: Xinyuan Lin <xinyual3@uci.edu>

aglinxinyuan · 2026-04-26T08:47:52Z

I think we should let the coordinator jump back to a stage, rather than "schedule" or "scheduler".

Updated based on the review.

I refactored the implementation so that:

Schedule remains static/immutable
WorkflowScheduler only produces and holds the generated schedule
WorkflowExecutionCoordinator owns the current execution position and handles jumpToOperator(...)

Yicong-Huang · 2026-04-26T09:38:11Z

I think we should let the coordinator jump back to a stage, rather than "schedule" or "scheduler".

Updated based on the review.

I refactored the implementation so that:

Schedule remains static/immutable

WorkflowScheduler only produces and holds the generated schedule

WorkflowExecutionCoordinator owns the current execution position and handles jumpToOperator(...)

Thanks a lot! Let's keep plan/schedule those data classes immutable. I can later add tests to guard this property.

chenlica · 2026-04-27T03:14:32Z

A random comment: the PR number 4444 is very special :-)

Yicong-Huang · 2026-04-27T03:16:45Z

A random comment: the PR number 4444 is very special :-)

one day we will reach 44444 ;)

aglinxinyuan · 2026-04-27T04:34:56Z

@Yicong-Huang, please have another pass.

aglinxinyuan · 2026-04-28T05:18:25Z

Making a major revision based on the offline discussion with @Yicong-Huang. I will let people know when the PR is ready for review.

aglinxinyuan · 2026-04-30T06:43:34Z

@Yicong-Huang, please have a pass. The PR is updated based on the discussion. WorkflowExecutionCoordinator can now accept an updated Schedule, so your use case for dynamic scheduling should also be supported.

Yicong-Huang

LGTM in general, we can differ minor fix later. Please add more tests as this is a very new behavior

Add cases for multi-jump sequences, unknown-target rejection, jump before any pull, jump after schedule exhaustion, and forward jump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Reject any `Schedule` constructed with gaps or non-zero starting level keys. The schedule generator already produces contiguous-from-0 keys, so this only tightens the contract for direct callers and tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

# Conflicts: # amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinator.scala # amber/src/test/scala/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinatorSpec.scala

aglinxinyuan added 2 commits April 21, 2026 01:20

Add scheduler jump-to-operator support

555e55d

Add scheduler jump test

001a409

aglinxinyuan requested a review from Xiao-zhen-Liu April 22, 2026 05:31

aglinxinyuan self-assigned this Apr 22, 2026

github-actions Bot added engine python labels Apr 22, 2026

aglinxinyuan changed the base branch from xinyuan-state-only to main April 22, 2026 05:33

aglinxinyuan added 7 commits April 21, 2026 22:33

Merge branch 'main' into xinyuan-scheduler-jump

eafb7f6

fix fmt

d695dec

Merge branch 'main' into xinyuan-scheduler-jump

abad7f1

Merge branch 'main' into xinyuan-scheduler-jump

3926100

Merge branch 'main' into xinyuan-scheduler-jump

72a3c24

Merge branch 'main' into xinyuan-scheduler-jump

9c00ac5

Merge branch 'main' into xinyuan-scheduler-jump

59e91c9

aglinxinyuan linked an issue Apr 25, 2026 that may be closed by this pull request

Add scheduler support for jumping to a target operator #4443

Open

5 tasks

Yicong-Huang reviewed Apr 26, 2026

View reviewed changes

Comment thread ...la/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinator.scala Outdated

Yicong-Huang reviewed Apr 26, 2026

View reviewed changes

Comment thread ...rc/main/scala/org/apache/texera/amber/engine/architecture/controller/WorkflowScheduler.scala Outdated

Yicong-Huang reviewed Apr 26, 2026

View reviewed changes

Comment thread amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/Schedule.scala Outdated

aglinxinyuan added 2 commits April 26, 2026 00:52

Merge branch 'main' into xinyuan-scheduler-jump

f327f0c

Signed-off-by: Xinyuan Lin <xinyual3@uci.edu>

Refactor jump state into execution coordinator

20e2ced

aglinxinyuan requested a review from Yicong-Huang April 26, 2026 08:47

aglinxinyuan added 2 commits April 26, 2026 01:50

Simplify coordinator jump scheduling

4c8cb5f

fix fmt

b7c2e3b

Yicong-Huang reviewed Apr 26, 2026

View reviewed changes

Comment thread ...la/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinator.scala Outdated

Yicong-Huang reviewed Apr 26, 2026

View reviewed changes

Comment thread amber/src/test/scala/org/apache/texera/amber/engine/architecture/scheduling/ScheduleSpec.scala Outdated

Yicong-Huang reviewed Apr 26, 2026

View reviewed changes

Comment thread ...la/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinator.scala Outdated

Yicong-Huang reviewed Apr 26, 2026

View reviewed changes

Comment thread ...ache/texera/amber/engine/architecture/controller/promisehandlers/JumpToOperatorHandler.scala Outdated

aglinxinyuan changed the title ~~feat(engine): add scheduler jump-to-operator support~~ feat(engine): add jump-to-operator support Apr 27, 2026

aglinxinyuan added 2 commits April 26, 2026 21:27

Merge branch 'main' into xinyuan-scheduler-jump

f726337

fix

baf5dbb

fix fmt

ba6104f

Yicong-Huang requested changes Apr 27, 2026

View reviewed changes

Merge branch 'main' into xinyuan-scheduler-jump

cff7fb1

aglinxinyuan marked this pull request as draft April 28, 2026 05:18

aglinxinyuan added 4 commits April 27, 2026 22:22

update

560b670

update

3579bc1

update

802bdd0

Merge branch 'main' into xinyuan-scheduler-jump

9f5feab

github-actions Bot added the ci changes related to CI label Apr 28, 2026

update

f5ef447

github-actions Bot removed the ci changes related to CI label Apr 28, 2026

aglinxinyuan added 2 commits April 29, 2026 23:27

Merge branch 'main' into xinyuan-scheduler-jump

0f9a5f8

fix fmt

83e6fb5

aglinxinyuan requested a review from Yicong-Huang April 30, 2026 06:40

aglinxinyuan marked this pull request as ready for review April 30, 2026 06:40

Merge branch 'main' into xinyuan-scheduler-jump

178f59b

Yicong-Huang approved these changes Apr 30, 2026

View reviewed changes

Comment thread ...rg/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinatorSpec.scala Outdated

aglinxinyuan and others added 2 commits April 30, 2026 15:48

test(amber): expand jump-to-operator-region coordinator coverage

2ab9e00

Add cases for multi-jump sequences, unknown-target rejection, jump before any pull, jump after schedule exhaustion, and forward jump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

aglinxinyuan mentioned this pull request Apr 30, 2026

test(amber): add unit test coverage for Schedule iterator semantics #4562

Open

aglinxinyuan added 2 commits April 30, 2026 16:22

Merge branch 'main' into xinyuan-scheduler-jump

264ca26

Conversation

aglinxinyuan commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Yicong-Huang commented Apr 26, 2026

Uh oh!

aglinxinyuan commented Apr 26, 2026

Uh oh!

Yicong-Huang commented Apr 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chenlica commented Apr 27, 2026

Uh oh!

Yicong-Huang commented Apr 27, 2026

Uh oh!

aglinxinyuan commented Apr 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aglinxinyuan commented Apr 28, 2026

Uh oh!

aglinxinyuan commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yicong-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aglinxinyuan commented Apr 22, 2026 •

edited

Loading

aglinxinyuan commented Apr 30, 2026 •

edited

Loading