Add cache for task queue routing info in History by ShahabT · Pull Request #9168 · temporalio/temporal

ShahabT · 2026-01-29T19:50:22Z

What changed?

Cache the result of GetTaskQueueUserData that history makes to Matching when an activity wants to start a deployment version transition.

Why?

This protects the matching root partition from being hammered by requests when a lot of AutoUpgrade workflows want to start activity-initiated transitions. Activity initiated transitions happen in one of the following cases:

Target version changed while activity was backlogged.
Target version changed while activity was in retry backoff
Target version changed in some edge cases involving parallel activities

How did you test it?

Potential risks

The cache can potentially increase history mem usage but there are knobs to adjust size and ttl.

ShahabT · 2026-01-29T19:51:45Z

service/history/workflow/mutable_state_impl.go

+			// backlog but because of an ongoing transition. See ActivityStartDuringTransition error usage.
+			// TODO (shahab): can we limit this adjustment to apply only for activities who actually faced the
+			// ActivityStartDuringTransition error, and not all others?
+			info.ScheduledTime = timestamppb.New(ms.timeSource.Now())


@yycptt could you please review this line? what to ensure it does not breaks some assumptions in the system.

sorry to be pendantic but I wonder if this should come in a different PR which has some tests that test that there the time delay has been present

It's a fair comment. I did try to spend some time writing the test but this part is soooo internal that I could not come up with a good way to write a non-flaky test. I did though test it manually and verified the delay existed before and my change is canceling it.

Shivs11

approved but I don't think we should have that activity scheduled time change going in without tests

Shivs11 · 2026-01-30T00:18:08Z

common/worker_versioning/routing_info_cache.go

+// RoutingInfoCache is used to cache results of GetTaskQueueUserData
+// calls followed by CalculateTaskQueueVersioningInfo computation.


nit: we can just say that this cache is used to cache results of GetTaskQueueUserData operations since that is the main RPC that is actually intensive.

Shivs11 · 2026-01-30T00:22:17Z

service/history/api/recordactivitytaskstarted/api.go

-	}
-	tqData, ok := resp.GetUserData().GetData().GetPerType()[int32(taskQueueType)]
+	// Check cache first for task queue routing info (independent of workflow ID)
+	current, currentRevisionNumber, ramping, rampingPercentage, rampingRevisionNumber, ok := routingInfoCache.Get(namespaceID, taskQueueName, taskQueueType)


something to remember about this change - I know that the cache TTL is only one second which is quite small, but during that one second, there could in theory be stale task queue user data reads which could make AU workflow/activity tasks bounce back and forth.

I think this is fine, just wanted to paste this here.

Shivs11 · 2026-01-30T00:25:06Z

service/history/workflow/mutable_state_impl.go

+			// backlog but because of an ongoing transition. See ActivityStartDuringTransition error usage.
+			// TODO (shahab): can we limit this adjustment to apply only for activities who actually faced the
+			// ActivityStartDuringTransition error, and not all others?
+			info.ScheduledTime = timestamppb.New(ms.timeSource.Now())


sorry to be pendantic but I wonder if this should come in a different PR which has some tests that test that there the time delay has been present

Shivs11 · 2026-01-30T00:26:30Z

tests/versioning_3_test.go

 		"Error message should include the limit value")
 }

+func (s *Versioning3Suite) TestActivityRetryAutoUpgradeDuringBackoff() {


should we also have some functional tests that test the actual cache implementation that is now present?
I had placed some tests in this file where I increased the TTL and noticed if the calls are going through or not.

sure, I see you have placed unit tests for the cache and since we are in a time crunch, you can ignore my top comment.

Shivs11 · 2026-01-30T00:28:09Z

tests/versioning_3_test.go

+		err := workflow.ExecuteActivity(workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
+			StartToCloseTimeout: 10 * time.Second,
+			RetryPolicy: &temporal.RetryPolicy{
+				InitialInterval:    3 * time.Second, // Give us time to change deployment


could this be flaky? 3 seconds seems less....like what happens if we change the deployment before that?

changed it to 5s

## What changed? Cache the result of `GetTaskQueueUserData` that history makes to Matching when an activity wants to start a deployment version transition. ## Why? This protects the matching root partition from being hammered by requests when a lot of AutoUpgrade workflows want to start activity-initiated transitions. Activity initiated transitions happen in one of the following cases: 1) Target version changed while activity was backlogged. 2) Target version changed while activity was in retry backoff 3) Target version changed in some edge cases involving parallel activities ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks The cache can potentially increase history mem usage but there are knobs to adjust size and ttl.

## What changed? Cherry-pick versioning PRs - #9168 - Cache for system protection - #9262 - Cache for system protection - #9239 - Critical PR to enable sending `TargetVersionChanged` flag for Upgrade-on-CaN feature - #9147 - Tracks version drainage properly when version receives workflows via `VersioningOverride`. Needed for automated worker controllers to correctly scale versioned workers that received workflows via `VersioningOverride`. - #9300 - Needed for `approximate_backlog_count` metric to track Current and Ramping version tasks correctly - #9316 - Needed for `approximate_backlog_count` metric to track Current and Ramping version tasks correctly - #8957 - Contains minor metric improvement. Included because it adds a test harness that is used in the two metrics PRs above - #9250 - Bug fix of task rescheduling edge case during AutoUpgrade Transition ## Why? For OSS v1.30.2 ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s)  --- > [!NOTE] > **Medium Risk** > Touches history/matching worker-versioning paths, adding new caches and changing workflow task/start handling and backlog metric emission; incorrect caching or signaling could affect dispatch/upgrade behavior and observability. > > **Overview** > Adds new worker-versioning protections and upgrade signaling: workflow task started events now persist a `workflow_task_target_worker_deployment_version_changed` flag (and emit a new `workflow_target_version_changed_count` metric) under a new `EnableSendTargetVersionChanged` dynamic config. > > Introduces two new caches with metrics and dynamic config knobs: a `RoutingInfoCache` to avoid repeated `GetTaskQueueUserData` lookups during activity start/transition logic, and a `ReactivationSignalCache` plus `EnableVersionReactivationSignals` to dedupe and asynchronously send “reactivation” signals when workflows are pinned (via start/signal-with-start/reset/update-options) to potentially drained/inactive worker versions. > > Extends matching backlog metrics to support version-attributed reporting by adding `BacklogMetricsEmitInterval` and switching queue DB emission to *physical* backlog gauges (`physical_approximate_backlog_*`) when attribution is enabled, while keeping legacy gauges when disabled. > > Adds frontend scaffolding for a new visibility RPC `CountSchedules` (client plumbing, interception/metadata/quota wiring) but leaves the frontend handler unimplemented, and bumps `go.temporal.io/api` to `v1.62.2`. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit cb8ae14. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>  --------- Co-authored-by: Shahab Tajik <shahab@temporal.io> Co-authored-by: Shivam <57200924+Shivs11@users.noreply.github.com>

Add cache for task queue routing info in History

70a3210

ShahabT requested a review from Shivs11 January 29, 2026 19:50

ShahabT requested review from a team as code owners January 29, 2026 19:50

ShahabT commented Jan 29, 2026

View reviewed changes

add tests

e15a099

Shivs11 approved these changes Jan 30, 2026

View reviewed changes

add wft duration instead of resetting scheduled time

e0d6c27

ShahabT enabled auto-merge (squash) January 30, 2026 02:16

fix lint

8d10e77

ShahabT merged commit b6e5e1e into temporalio:main Jan 30, 2026
108 of 110 checks passed

carlydf added the release/1.30.2 label Mar 12, 2026

carlydf mentioned this pull request Mar 17, 2026

Versioning cherry-pick for OSS v1.30.2 #9543

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cache for task queue routing info in History#9168

Add cache for task queue routing info in History#9168
ShahabT merged 4 commits intotemporalio:mainfrom
ShahabT:cache-routing-info

ShahabT commented Jan 29, 2026

Uh oh!

ShahabT Jan 29, 2026

Uh oh!

Shivs11 Jan 30, 2026

Uh oh!

ShahabT Jan 30, 2026

Uh oh!

Shivs11 left a comment

Uh oh!

Shivs11 Jan 30, 2026

Uh oh!

Shivs11 Jan 30, 2026

Uh oh!

Shivs11 Jan 30, 2026

Uh oh!

Shivs11 Jan 30, 2026

Uh oh!

Shivs11 Jan 30, 2026

Uh oh!

Shivs11 Jan 30, 2026

Uh oh!

ShahabT Jan 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// RoutingInfoCache is used to cache results of GetTaskQueueUserData
		// calls followed by CalculateTaskQueueVersioningInfo computation.

Conversation

ShahabT commented Jan 29, 2026

What changed?

Why?

How did you test it?

Potential risks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shivs11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants