Replace orphaned child workflows with no worker progress after force failover - #11815
Conversation
| case enumsspb.WORKFLOW_EXECUTION_STATE_CREATED: | ||
| return mutableState.GetNextEventID() == common.FirstEventID+1 | ||
| case enumsspb.WORKFLOW_EXECUTION_STATE_RUNNING: | ||
| if mutableState.GetNextEventID() != common.FirstEventID+2 || mutableState.HasCompletedAnyWorkflowTask() { |
There was a problem hiding this comment.
does check if there is any workflow task schedule good enough? e.g. use HadOrHasWorkflowTask?
There was a problem hiding this comment.
It's a bit nasty because the goal is to capture WORKFLOW_TASK_SCHEDULED event but not started. Not semantically same to HadOrHasWorkflowTask() check. I have narrowed this check to only WORKFLOW_EXECUTION_STARTED event, where it matched the incident.
|
|
||
| func isOrphanedChildWithoutProgress(mutableState historyi.MutableState) bool { | ||
| executionState := mutableState.GetExecutionState() | ||
| if executionState.GetFirstExecutionRunId() != mutableState.GetWorkflowKey().RunID { |
There was a problem hiding this comment.
if the child continue as new, will it has a different run id to the first execution run id?
There was a problem hiding this comment.
Yes CAN will have a new runID. It on purpose to skip handle the CAN / Cron etc runs. Only when the conflicted run is a first-time run and hasn't progressed, we can safely terminate and replace.
Allow a parent to mark an open child from a losing initiation as zombie and create its replacement atomically when the reuse policy permits. Validate ownership and initiation metadata under the child lock while preserving normal deduplication for accepted children.
9d1d347 to
44cffe5
Compare
| if executionState.GetFirstExecutionRunId() != mutableState.GetWorkflowKey().RunID || | ||
| executionState.GetState() != enumsspb.WORKFLOW_EXECUTION_STATE_CREATED || | ||
| mutableState.GetNextEventID() != common.FirstEventID+1 || | ||
| workflowLease.GetContext().UpdateRegistry(ctx).Len() != 0 { |
There was a problem hiding this comment.
can we just check the pending updates and received signal count?
There was a problem hiding this comment.
I think firstRunID check is needed for not terminating successors of CAN / cron.
The state = CREATED check is also required to make sure no WFT scheduled.
For external writes, such as Pause, Cancel and Signal, they all append history events. I believe checking on FirstEventID + 1 could cover all of them.
3 participants
What changed?
OrphanedChildReplacementInfo, carrying the parent’s current Version History as branch evidence.WorkflowExecutionStarted;history.enableOrphanedChildWorkflowReplacementsetting.Why?
After force failover, a child created from a losing parent branch may conflict with the same child start reissued by the winning branch. Normal workflow-ID conflict handling records
WORKFLOW_ALREADY_EXISTS, leaving the parent unable to make progress and the original child orphaned.Re-linking the existing child is unsafe because its parent coordinates are stored in immutable history. This change instead replaces it only when the new active cluster sees no progress beyond
WorkflowExecutionStarted.Ownership and progress are rechecked while holding the child lock, and termination plus replacement creation are committed atomically.
How did you test it?
Validated with:
go test -tags test_dep ./service/history/api ./service/history -count=1make protomake lint-codePotential risks
VersionHistoryItemslist. An unusually long version history may increase RPC size and could exceed internal gRPC limits; oversized requests fail instead of recovering the child.WORKFLOW_ALREADY_EXISTS.