Activate `Drained`/`Inactive` version if a workflow is "moved" to it by Shivs11 · Pull Request #9147 · temporalio/temporal

Shivs11 · 2026-01-27T15:25:03Z

What changed?

activate drained/inactive versions to draining when they get a workflow started on it
also added a history cache, per history node, so that we don't bombard our version workflows with signals that shall change the drainage status of these workflows.

Why?

versioning correctness, in the sense that if someone were to move a workflow on to a version that is drained, the drainage status should be updated to draining (since it now has one open workflow working on it)

How did you test it?

Potential risks

Sure, this change is lowkey risky. Would appreciate a thorough review.

Note

Medium Risk
Touches history start/reset/signal-with-start/update-options code paths and introduces asynchronous signaling + caching; failures won’t block requests but could cause missed or excessive reactivation signals if misconfigured.

Overview
Ensures worker-versioning drained/inactive versions get reactivated: when a workflow is started, signal-with-started, reset with post-reset option updates, or has execution options updated with a pinned VersioningOverride, the history service now fire-and-forgets a reactivate-version signal to the corresponding version workflow to move it back to DRAINING.

Adds a per-history-node version_reactivation_signal dedup cache (TTL/max-size + metrics) and a global kill-switch (history.enableVersionReactivationSignals) to avoid flooding version workflows; wires the cache + workerdeployment.Client.SignalVersionReactivation through history engine/fx and adds functional tests covering reactivation and cache dedup behavior.

^{Written by Cursor Bugbot for commit 06ad026. This will update automatically on new commits. Configure here.}

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Shivs11 · 2026-02-11T21:27:33Z

 			// The history and mutable state generated above will be deleted by a background process.
 			resp, outcome, conflictErr := s.handleConflict(ctx, creationParams, currentWorkflowConditionFailedError)
-			if conflictErr == nil {
+			if conflictErr == nil && outcome == StartNew {


the main idea here was to only send the reactivation signal iff we have actually started/created a new workflow execution - all other outcomes, except StartNew, do not do that.

ShahabT · 2026-02-12T07:28:12Z

+	shardContext historyi.ShardContext,
+	namespaceID namespace.ID,


I think most places who call this func already have the namespace entry. we should just get that instead of these two args.

hmmm, you are right. Noticed that SignalVersionReactivation on the client already logs all errors via convertAndRecordError - so the Warn log in ReactivateVersionWorkflowIfPinned is redundant.

ShahabT · 2026-02-12T07:29:31Z

+// VersionReactivationSignalerFn is a function type for sending reactivation signals to version workflows.
+// This abstraction allows the history API layer to use the deployment client without importing it directly,
+// avoiding import cycles between history/api and worker/workerdeployment packages.
+type VersionReactivationSignalerFn func(


any reason for this type instead of passing client itself?

circular dependencies; my first stab at this involved me just directly passing the workerDeployment.Client, but that did not go well as:

service/history/api → service/worker/workerdeployment → service/history/api

ShahabT · 2026-02-12T07:33:12Z

+	}
+
+	// Use signaler function to send the signal
+	err = signaler(ctx, nsEntry, pinnedVersion.GetDeploymentName(), pinnedVersion.GetBuildId())


let' make sure this call is async so we don't add latency to user's request.

Shivs11 · 2026-02-12T17:55:36Z

+	go func() {
+		signaler(context.Background(), namespaceEntry, pinnedVersion.GetDeploymentName(), pinnedVersion.GetBuildId())
+	}()


a thing to remember here is that any error, if present, would be caught in the client's recordAndError method and would be logged out. However, I am not returning the error over here since I don't think we should be stopping existing operations based on this error

carlydf

approved with small name change nit

…e-drained-inactive-wfs

…#9147) ## What changed? - activate drained/inactive versions to draining when they get a workflow started on it - also added a history cache, per history node, so that we don't bombard our version workflows with signals that shall change the drainage status of these workflows. ## Why? - versioning correctness, in the sense that if someone were to move a workflow on to a version that is drained, the drainage status should be updated to draining (since it now has one open workflow working on it) ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) ## Potential risks - Sure, this change is lowkey risky. Would appreciate a thorough review.  --- > [!NOTE] > **Medium Risk** > Touches history start/reset/update flows and adds asynchronous signaling into worker-deployment workflows; mis-wiring or cache/config issues could cause missing or excessive reactivation signals and unexpected version state churn under load. > > **Overview** > When workflows are **pinned to a specific deployment version**, history now triggers a fire-and-forget `reactivate-version` signal to the corresponding worker-deployment *version workflow* so versions in `DRAINED/INACTIVE` transition back to `DRAINING`. > > This wiring is applied across start paths (`StartWorkflowExecution` incl. conflict handling, `SignalWithStart`, multi-op start), option changes (`UpdateWorkflowExecutionOptions` after persistence), and `ResetWorkflowExecution` post-reset operations. A new per-history-node `ReactivationSignalCache` (TTL/max-size + metrics tags) deduplicates signals, and a new dynamic config flag (`history.enableVersionReactivationSignals`) plus cache settings control/limit load. > > Worker-deployment adds `Client.SignalVersionReactivation` and the version workflow gains a version-gated handler for `reactivate-version` to update drainage/status and sync summaries; extensive functional tests cover reactivation and cache dedup behavior. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 15a5ac6. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>  --------- Co-authored-by: Carly de Frondeville <cdefrondeville@berkeley.edu>

## What changed? Cherry-pick versioning PRs - #9168 - Cache for system protection - #9262 - Cache for system protection - #9239 - Critical PR to enable sending `TargetVersionChanged` flag for Upgrade-on-CaN feature - #9147 - Tracks version drainage properly when version receives workflows via `VersioningOverride`. Needed for automated worker controllers to correctly scale versioned workers that received workflows via `VersioningOverride`. - #9300 - Needed for `approximate_backlog_count` metric to track Current and Ramping version tasks correctly - #9316 - Needed for `approximate_backlog_count` metric to track Current and Ramping version tasks correctly - #8957 - Contains minor metric improvement. Included because it adds a test harness that is used in the two metrics PRs above - #9250 - Bug fix of task rescheduling edge case during AutoUpgrade Transition ## Why? For OSS v1.30.2 ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s)  --- > [!NOTE] > **Medium Risk** > Touches history/matching worker-versioning paths, adding new caches and changing workflow task/start handling and backlog metric emission; incorrect caching or signaling could affect dispatch/upgrade behavior and observability. > > **Overview** > Adds new worker-versioning protections and upgrade signaling: workflow task started events now persist a `workflow_task_target_worker_deployment_version_changed` flag (and emit a new `workflow_target_version_changed_count` metric) under a new `EnableSendTargetVersionChanged` dynamic config. > > Introduces two new caches with metrics and dynamic config knobs: a `RoutingInfoCache` to avoid repeated `GetTaskQueueUserData` lookups during activity start/transition logic, and a `ReactivationSignalCache` plus `EnableVersionReactivationSignals` to dedupe and asynchronously send “reactivation” signals when workflows are pinned (via start/signal-with-start/reset/update-options) to potentially drained/inactive worker versions. > > Extends matching backlog metrics to support version-attributed reporting by adding `BacklogMetricsEmitInterval` and switching queue DB emission to *physical* backlog gauges (`physical_approximate_backlog_*`) when attribution is enabled, while keeping legacy gauges when disabled. > > Adds frontend scaffolding for a new visibility RPC `CountSchedules` (client plumbing, interception/metadata/quota wiring) but leaves the frontend handler unimplemented, and bumps `go.temporal.io/api` to `v1.62.2`. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit cb8ae14. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>  --------- Co-authored-by: Shahab Tajik <shahab@temporal.io> Co-authored-by: Shivam <57200924+Shivs11@users.noreply.github.com>

Shivs11 added 9 commits January 9, 2026 16:24

current work

4893d34

current tinkering

7136f94

currently working test

7c3e682

Merge origin/main into ss/reactive-drained-inactive-wfs

eae0388

draft 1 with tests

95f110c

added cache alongside tests

9fbe766

Remove CLAUDE.md and revert .gitignore changes

885f033

removed buggy drainage condition

79ae1c1

add version gate for this safe since its not NDE safe

040d04d

Shivs11 commented Jan 27, 2026

View reviewed changes

Comment thread common/worker_versioning/worker_versioning.go Outdated

Shivs11 added 2 commits January 27, 2026 12:01

lint and removal of prints

fa26af0

remove some logs

b0d1138

Shivs11 marked this pull request as ready for review January 29, 2026 21:58

Shivs11 requested review from a team as code owners January 29, 2026 21:58

ShahabT reviewed Jan 29, 2026

View reviewed changes

Comment thread service/history/api/worker_versioning_util.go Outdated

Comment thread service/history/api/worker_versioning_util.go

cursor Bot reviewed Jan 29, 2026

View reviewed changes

Comment thread common/worker_versioning/version_reactivation_signal_cache.go

Comment thread service/worker/workerdeployment/version_workflow.go

Comment thread service/worker/workerdeployment/util.go

Comment thread service/history/api/worker_versioning_util.go Outdated

Shivs11 mentioned this pull request Feb 2, 2026

Add handler-level rate limiting for worker deployment read APIs + move ListWorkerDeployments to Visibility group #9195

Merged

5 tasks

Shivs11 added 2 commits February 3, 2026 12:16

move signal sending to the workerdeployment Client

dfc4094

add dynamic config for thsi

1490d62

cursor Bot reviewed Feb 3, 2026

View reviewed changes

Comment thread service/history/api/updateworkflowoptions/api.go Outdated

Shivs11 and others added 4 commits February 3, 2026 13:47

send signal after update persistence

ef0fa0d

Merge branch 'main' into ss/reactive-drained-inactive-wfs

7139756

fix breaking build

f084d50

Merge branch 'main' into ss/reactive-drained-inactive-wfs

30a5a21

cursor Bot reviewed Feb 11, 2026

View reviewed changes

Comment thread service/history/api/startworkflow/api.go Outdated

address cursor bugbot comments

cc050fe

cursor Bot reviewed Feb 11, 2026

View reviewed changes

Comment thread service/history/api/startworkflow/api.go

only reactivate when outcome is StartNew

c33dd2e

Shivs11 commented Feb 11, 2026

View reviewed changes

Shivs11 added 2 commits February 11, 2026 17:17

add noop worker deployment client to make unit tests pass

87fdbad

fix unit test with an expectation call

4c03435

ShahabT reviewed Feb 12, 2026

View reviewed changes

address comments

38012f0

Shivs11 commented Feb 12, 2026

View reviewed changes

Shivs11 added 6 commits February 12, 2026 12:58

goimports

b9ed17c

lint

39c252a

helpful comment

04be7bb

hoping this resolves my lint issues

3a91e57

revive in lint

adaac7d

finally ran make lint locally (was not working before)

cf74e58

carlydf approved these changes Feb 12, 2026

View reviewed changes

Comment thread common/dynamicconfig/constants.go Outdated

Comment thread common/metrics/metric_defs.go Outdated

Comment thread service/history/api/updateworkflowoptions/api.go

carlydf added 3 commits February 12, 2026 14:14

add version prefix to reactivation cache names

fcecd45

merge

15a5ac6

Merge branch 'main' of github.com:temporalio/temporal into ss/reactiv…

06ad026

…e-drained-inactive-wfs

carlydf enabled auto-merge (squash) February 12, 2026 22:22

carlydf changed the title ~~[Worker-Versioning improvement]: Activate "Drained/Inactive" versions~~ Activate "Drained/Inactive" versions if a workflow is "moved" to that version Feb 12, 2026

carlydf changed the title ~~Activate "Drained/Inactive" versions if a workflow is "moved" to that version~~ Activate Drained/Inactive versions if a workflow is "moved" to that version Feb 12, 2026

carlydf changed the title ~~Activate Drained/Inactive versions if a workflow is "moved" to that version~~ Activate Drained/Inactive version if a workflow is "moved" to it Feb 12, 2026

carlydf merged commit 4668ca5 into main Feb 12, 2026
74 checks passed

carlydf deleted the ss/reactive-drained-inactive-wfs branch February 12, 2026 23:30

carlydf added the release/1.30.2 label Mar 12, 2026

carlydf mentioned this pull request Mar 17, 2026

Versioning cherry-pick for OSS v1.30.2 #9543

Merged

5 tasks

		shardContext historyi.ShardContext,
		namespaceID namespace.ID,

Conversation

Shivs11 commented Jan 27, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

How did you test it?

Potential risks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Shivs11 Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ShahabT Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Shivs11 Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

ShahabT Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Shivs11 Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

ShahabT Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Shivs11 Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlydf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Shivs11 commented Jan 27, 2026 •

edited by cursor Bot

Loading

Shivs11 Feb 12, 2026 •

edited

Loading