Skip to content

Preserve the original workflow start time on RebuildMutableState - #11668

Merged
simvlad merged 2 commits into
temporalio:mainfrom
simvlad:fix-rebuild-preserve-start-time
Sep 2, 2026
Merged

Preserve the original workflow start time on RebuildMutableState#11668
simvlad merged 2 commits into
temporalio:mainfrom
simvlad:fix-rebuild-preserve-start-time

Conversation

@simvlad

@simvlad simvlad commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

What changed?

AdminRebuildMutableState, invoked by tdbg workflow rebuild, no longer replaces the mutable state
start time with the current time. It reuses the run's recorded start time, falling back to
ExecutionInfo.StartTime. Only if both ExecutionState.StartTime and ExecutionInfo.StartTime are
empty, the current time will be used.

The rebuild now also refreshes the timeout timer tasks so the run and execution deadlines are anchoraed at the time of the call, which is what reset already does. Without it, preserving the start time would make a workflow rebuild after its deadline timeout immediately. It also records ExecutionInfo.MutableStateRebuildTime so that a repaired run could be identify by operator.

Why?

Moving the start time to now, often results in workflow with a negative duration, since the end time is kept as is. Downstream systems may handle this incorrectly.

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)

Potential risks

  1. Some downstream systems may prefer the workflow start time to move.
  2. Admin may intent for the workflow to have a new timeout instead of getting a timeout

@simvlad
simvlad requested a review from a team August 19, 2026 23:08
@simvlad
simvlad requested review from a team as code owners August 19, 2026 23:08
Comment thread service/history/workflow_rebuilder.go Outdated
@simvlad
simvlad force-pushed the fix-rebuild-preserve-start-time branch 2 times, most recently from f0c02b6 to f124cd2 Compare August 20, 2026 20:09
@yycptt yycptt added the reliability-2026 Reliability related changes label Aug 21, 2026
@yycptt

yycptt commented Aug 21, 2026

Copy link
Copy Markdown
Member

a workflow rebuilt after its original run timeout would have expired now times out promptly instead of getting a fresh timeout window.

I think we need to discuss this. The command is not that useful if stuck execution will be immediately timedout after rebuild. And I think we observed case where after rebuild an execution timedout immediately even with current implementation.

@simvlad

simvlad commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Discussed, we will:

  1. Reset both execution and run timeouts
  2. Add the flag on the mutable state to indicate whether the rebuild was ran to show in tdbg, so at least the admin can understand why start time/duration is inconsistent with timeouts

@simvlad
simvlad force-pushed the fix-rebuild-preserve-start-time branch from 56d804a to 8f39ab7 Compare August 31, 2026 23:50
@simvlad

simvlad commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

@yycptt - fixed, updated both execution and run deadlines and recorded the last rebuilt time on the mutable state

@simvlad
simvlad force-pushed the fix-rebuild-preserve-start-time branch from 8f39ab7 to 8177b39 Compare September 1, 2026 05:04
@simvlad
simvlad force-pushed the fix-rebuild-preserve-start-time branch from 1c87f9c to 5585364 Compare September 2, 2026 00:09
@simvlad
simvlad merged commit 45a36d1 into temporalio:main Sep 2, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

oss-foundations reliability-2026 Reliability related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants