[Experiment] Integrate scheduler readability stack with server main - #64
Draft
chaptersix wants to merge 140 commits into
Draft
[Experiment] Integrate scheduler readability stack with server main#64chaptersix wants to merge 140 commits into
chaptersix wants to merge 140 commits into
Conversation
# What changed? `tdbg` now registers the **archival** task category (`5`) in its task-category registry (`tools/tdbg/app.go`), so `tdbg dlq list/merge/purge --dlq-type 5` works. Previously `merge`/`purge` rejected it with `unknown dlq category 5`. Because the `--dlq-type` help text is generated from the registry, archival now appears there too. Adds a regression test to `TestDLQCommand_V2`. ## Why? `tdbg` built its registry with `tasks.NewDefaultTaskCategoryRegistry()`, which omits `CategoryArchival`. The server registers archival conditionally in `TaskCategoryRegistryProvider` (`temporal/fx.go`) when archival is enabled, but `tdbg` is a client tool with no access to the cluster's archival config, so it rejected `--dlq-type 5` client-side — even though archival tasks are written to the history task DLQ and the server's `AdminService.MergeDLQTasks` accepts the category. That left archival DLQ tasks unrecoverable via the standard tool (operators had to call the RPC directly). Registering the category unconditionally in `tdbg` is safe: with archival disabled there is simply no archival DLQ to operate on. Fixes temporalio#11586. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) --------- Signed-off-by: Tihomir Surdilovic <tihomir@temporal.io> Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
Add nil check for `request.History` before accessing `request.History.Data` in `serializeAppendRawHistoryNodesRequest` to prevent a panic when the caller passes a nil DataBlob. Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
…11177) ## Motivation `TimerSequenceID` (~48 bytes) is heap-allocated and returned as a `*TimerSequenceID` from 5 getter functions: `getUserTimerTimeout`, `getActivityScheduleToStartTimeout`, `getActivityScheduleToCloseTimeout`, `getActivityStartToCloseTimeout`, and `getActivityHeartbeatTimeout`. These are called for every pending timer/activity during `LoadAndSortUserTimers()` and `LoadAndSortActivityTimers()` — a hot path in every workflow task. Callers were already dereferencing the pointer before appending to the value-type `[]TimerSequenceID` slice, so the existing code was allocating on the heap only to immediately copy to the stack. ## Changes - `TimerSequenceID` is now returned by value with a `(TimerSequenceID, bool)` tuple - Idiomatic Go pattern (same as map access) replaces nil-check sentinel - All 5 getter methods updated + call sites and tests adapted ## Impact Eliminates one heap allocation per getter call on the timer-sorting hot path. ## Tests - `service/history/workflow` (1075 tests): ✅ passed --------- Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
## What changed? Instrumented local frontend, CHASM callback, and external Nexus operation HTTP clients with the shared OpenTelemetry transport. Legacy HSM callbacks and cross-cluster forwarding are intentionally out of scope. ## Why? Outbound Nexus HTTP calls need to carry trace context so callbacks and internal frontend calls remain connected to their originating spans. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
- Adds `phase=error` to the existing `replication_lifecycle` wide event;
no new event type or table.
- Covers state-based replication (`SyncVersionedTransition`,
`VerifyVersionedTransition`, and `SyncWorkflowState`) plus standby
transfer, timer, and outbound queue failures.
- Captures sender, passive execution/apply, verification,
recovery/refetch, namespace refresh, Nack, DLQ, and history-branch
cleanup boundaries.
- Uses one shared error builder with small sender, executable-task, NDC,
and standby-queue adapters.
- Records workflow identity, source task identity, target context,
operation, error, attempt/priority, disposition/recovery, and extensible
diagnostics in `details`.
- Identifies apply provenance as `apply_artifact_source=task_payload` or
`sync_state_refetch`.
- Remains gated by `history.emitReplicationLifecycleEvents` (default
off).
## Why?
Replication failures span the sender, passive executor, recovery loop,
and apply layer. Recording these boundaries in the existing lifecycle
event makes the path of a workflow or replication task directly
traceable without adding another event schema.
## Example traces
The examples below are abridged records captured from the two-cluster
XDC test. Events for the same task correlate on `source_cluster`,
`source_shard`, and `source_task_id`; workflow identity is present on
every record.
A state task that fails on the passive cluster and is written to the
DLQ:
```json
{"phase":"sent","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"priority":"High"}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"passive_task_execution","error":"failed to apply replication task","error_type":"serviceerror.InvalidArgument","apply_artifact_source":"task_payload","attempt":1,"priority":"High","target_cluster":"standby"}}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"task_execution","error":"failed to apply replication task","terminal":true,"priority":"High","target_cluster":"standby"}}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"dlq_write","disposition":"dlq","terminal":true,"priority":"High","target_cluster":"standby","target_shard":1}}
```
A verification task that detects missing state, refetches it, and then
verifies successfully:
```json
{"phase":"sent","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"priority":"High"}
{"phase":"executing","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"attempt":1}
{"phase":"applied","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"resend_needed"}
{"phase":"error","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"details":{"operation":"standby_verification","error":"missing mutable state, resend","error_type":"serviceerror.SyncState","recovery_action":"sync_state","priority":"High","target_cluster":"standby"}}
{"phase":"applied","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"applied","details":{"apply_artifact_source":"sync_state_refetch"}}
{"phase":"executing","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"attempt":1}
{"phase":"applied","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"verified"}
```
## How was it tested?
- `go test -tags test_dep ./common/wideevents
./service/history/replication ./service/history/ndc ./service/history
./tests/testcore`
- Changed-lines `golangci-lint`: 0 issues.
- Temporary, uncommitted two-cluster XDC tests forced passive task
failures, DLQ handling, standby verification, and SyncState
refetch/recovery with lifecycle events both enabled and disabled.
- A tiered-processing run confirmed concrete `High` priority on sent and
error records.
## Risks
- Enabling the dynamic config increases event volume; retries and
recovery can produce several error phases for one source task.
- `details.operation`, `details.disposition`, and
`details.recovery_action` distinguish those boundaries.
- Emission is best effort and does not change replication error
propagation, retry, or recovery behavior.
…e status (temporalio#11636) ## What Adds a `--schedule-id` flag to `tdbg schedule migrate status`. When set, it looks up one specific schedule instead of the namespace-wide V1-vs-V2 counts, and reports whether that schedule is currently V1 (workflow-backed), V2 (CHASM), or caught in a migration sentinel state. ## Why During schedule migration triage, an operator often needs to know the status of *one specific* schedule, not aggregate counts. There was no way to answer that without manually poking `execution describe` and reasoning about CHASM node internals by hand. ## How it works It always probes **both** sides — the V1 workflow ID (`temporal-sys-scheduler:<id>`) and the V2/CHASM business ID (`<id>`) — via the same `DescribeMutableState` RPC that `tdbg execution describe` uses, regardless of which ID form was passed in. It deliberately does not short-circuit based on the input's shape (e.g. its prefix) without confirming against the server — an unconfirmed inference is confusing/untrustworthy in a diagnostic tool. This lets it flag both kinds of migration sentinel: - a **CHASM-side sentinel** `Scheduler` component reserving the ID during a V1→V2 migration - a **V1-side `DummyWorkflow`** reserving the workflow ID during a V2→V1 rollback Output leads with a plain-language headline describing the schedule's current, authoritative form (written for someone with no prior knowledge of the V1/V2 migration internals), followed by a note about any sentinel found on the other side, a details table of both sides' raw status, and the exact `execution describe` invocations to inspect each side further. ### Example output (V2→V1 rollback in flight) ``` Schedule "foo" is a V2 (CHASM) schedule. Additionally, a placeholder ("sentinel") V1 workflow exists at workflow ID "temporal-sys-scheduler:foo", reserving that ID while a V2→V1 rollback is in progress. This is expected during rollback and requires no action. Details: V1 (workflow-backed) [workflow ID temporal-sys-scheduler:foo]: sentinel (V2→V1 rollback placeholder) V2 (CHASM) [business ID foo]: genuine Inspect further: V1: tdbg execution describe --workflow-id temporal-sys-scheduler:foo -n <namespace> V2: tdbg execution describe --workflow-id foo --archetype scheduler.scheduler -n <namespace> ``` ## Testing - `go test ./tools/tdbg/... -run TestScheduleStatus -v` — new tests cover: prefixed input, genuine V1, genuine V2, V1 genuine + CHASM sentinel, V1 sentinel + genuine V2, not-found on both sides, and an unexpected/inconsistent-state fallback. Existing aggregate-count tests are unaffected. - `go build ./tools/...`, `go vet ./tools/tdbg/...` - Manually checked `tdbg schedule migrate status --help` output. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
) ## What changed? - Removed blanket timestamp wrapping - cron/retry/start child wf use virtual time directly ## Why? there was a double-shifting for CaN but not for other cases it shall be fixed unifying the clock used by all virtual time propagating cases, and this PR chooses to unify in a way that all cases propagate use virtual time ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s)
## What changed? Wrapped the frontend Nexus dispatch routes with the shared OpenTelemetry HTTP handler. ## Why? Nexus HTTP requests need an inbound server span to connect the caller trace. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s)
## What changed? 1. perf improvement: no time-skipping task regen on full refresh 2. new functional test: claim Nexus HSM timers are part of in-fight nexus operation, and won’t be skipped, add functional test 3. new functional test: add a functional test to verify time skipping won't change retention time 4. trivial bug fix: time-skipping: task regen didn’t read EnableWorkflowExecutionTimeoutTimer ## Why? - correctness related No. 4 though it is an edgy case - perf related No.1 - test coverage No.2,3 ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s)
temporalio#11618) ## What - Don't send scale down signal when task is matched from backlog even if the poll wait time is high. - Do not send scale up signal when task queue is rate limited. ## Why - When a task comes from DB backlog, the poll wait time reflects DB read path latency, not excess pollers — the -1 is not appropriate. Instead we want to apply the normal scale up check. - Similarly, when dispatch is bottlenecked by a task queue rate limit, scaling up pollers won't help. ## How did you test it? Unit tests Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed? Batch operations now target Paused executions in addition to Running ones. The filter auto-appended by adjustQueryBatchTypeEnum changed from ExecutionStatus='Running' to ExecutionStatus='Running' OR ExecutionStatus='Paused', affecting all workflow batch types (terminate/signal/cancel/update-options) and activity batch types (unpause/update-options/reset/terminate/cancel). ## Why? A paused execution is still non-terminal. Users issuing a batch terminate/signal/cancel reasonably expect it to apply to paused targets, but the old Running-only filter silently skipped them. ## How did you test it? - [X] built - [ ] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [X] added new functional test(s) ## Potential risks - The filter now references ExecutionStatus='Paused' on every batch operation. This is ok as it's an additive clause to the visibilty query - This would be changing behavior for batch callers as paused activities are now affected.
…eleasing workflow lock (temporalio#11602) ## What changed? Captured `ExecutionState.Status` before calling `GetReleaseFn()(nil)` in `updateWithStart.Invoke` to avoid a data race with concurrent goroutines that may acquire the lock and modify `ExecutionState` after it is released. Added a regression test that confirms the race under `-race`. Fixes temporalio#11600 ## Why? `Invoke` in `service/history/api/multioperation/api.go` releases the workflow lock at line 196 via `workflowLease.GetReleaseFn()(nil)`, then reads `workflowLease.GetMutableState().GetExecutionState().Status` at line 201 — after the lock is released. Any concurrent goroutine waiting on `Lock()` for the same workflow (e.g. a signal, terminate, or another update) can acquire the lock and modify `ExecutionState` between lines 196 and 201, creating a data race. This matches the pattern noted in the `Updater` struct itself: > WARNING: any references to mutable state data *have to* be copied to avoid data races when used outside the workflow lease. ## How did you test it? - [x] added new unit test(s) The test spawns a concurrent writer that modifies `ExecutionState.Status` after the lock is released. **Before fix:** ``` go test -race -tags test_dep -count=1 \ -run TestUpdateWithStartSuite/TestInvoke_CompletedUpdate_StatusCapturedBeforeRelease \ ./service/history/api/multioperation/ WARNING: DATA RACE Read at ... api.go:201 --- FAIL ``` **After fix:** ``` ok go.temporal.io/server/service/history/api/multioperation ``` ## Potential risks Minimal - single line moved before the release call. No API or persistence behavior change.
## What changed? 1. Binds httpCaller after setting httpClient 2. Uses TransitionStarted.Possible for complete-before-start 3. Records request time for nexus operation cancel ## Why? 1. Binds httpCaller after setting httpClient When clusterID isn't set, a non-nil httpCaller with a nil receiver gets passed to `nexusrpc.NewHTTPClient`. Since the httpCaller is non-nil, it will skip the check that sets the nil httpCaller to the default caller. 2. Uses TransitionStarted.Possible for complete-before-start Matches HSM, for if a start response is lost, and a completion lands while while the operation is in BACKING_OFF. 3. Records request time for nexus operation cancel This just seems like it wasn't being recorded. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s)
…oralio#11617) ## What changed? Validate the retry delay is a valid proto duration. ## Why? Prevent malformed retry delays from poisoning Nexus completions. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks Potentially users could have been sending an invalid proto duration, unclear how exactly, and now we would fail their request.
…tion (temporalio#11373) ## What changed and why? 1. Edge case: If fast-forward completes during a workflow transaction but when time skipping checks at close transaction this fast-forward time is before ms.Now(), time skipping should still get disabled. Now we don't look at time points that are in the past and it is a bug. 2. Time-skipping propagation: Always propagate the time-skipping configuration and fast-forward state to the next run and other executions, regardless of whether time skipping is currently active. This ensures that read APIs (for example, Describe and PollFastForward) continue to return meaningful information instead of nil. - Otherwise, in an edge case where fast-forward completes in the first run and the user polls after the second run has become the current run of the workflow execution, the poll API would return a NotFound error instead of a completed poll result. - Similarly, the Describe API would return a nil configuration instead of the original configuration that should have been propagated. 3. Trivial changes: - simiplify parameter of `propagateTimeSkippingToNextRun` - unify UT names of timeskipping_test.go
…1646) ## The bug `applyUpdatesToSubStateMachine` shadowed the variable it meant to populate: ```go var existing V // outer: never assigned if existing, ok := pendingInfos[key]; ok { // := declares a NEW existing, scoped to the if ... ms.approximateSize -= existing.Size() + getSizeOfKey(key) // inner one — correct here } val := updated if sanitizeFn != nil { val = common.CloneProto(updated) sanitizeFn(existing, val) // outer one — always nil } ``` `:=` only requires *one* new variable on the left (`ok`), and the `if` init statement is its own scope, so `existing` is redeclared there rather than assigned. It compiles because both variables are used, and `go vet` does not check shadowing by default. `sanitizeFn` has therefore always received `current == nil`. ## What this activates Two of the five `sanitizeFn` implementations depend on `current`; the other three (timers, request cancels, signals) ignore it or are `nil`, so they are unaffected. **Activities.** `getActivityTimerTaskStatus` short-circuits to `TimerTaskStatusNone` when `current` is nil, so every applied activity update cleared the *entire* timer task mask rather than only the bits whose deadline moved. The deadline comparison added in temporalio#11565 has never taken effect — that PR is currently a no-op. **Child executions.** `if current != nil { incoming.Clock = current.Clock }` never ran. `Clock` is a local shard vector clock that `sanitizeChildExecutionInfo` strips before replicating, so the incoming copy always arrives nil — the guard exists precisely to restore the local value. Without it, every replicated update to an existing child execution erased it. ## Risk Both are restorations of intended behavior, not new behavior, and both previously failed in the safe direction: - An over-cleared activity mask regenerates a timer task that is already pending. Duplicates are dropped at execution (`processSingleActivityTimeoutTask` re-derives the sequence and fires what expired), so the symptom was extra timer-queue writes, not missed timeouts. - A nil child clock is tolerated by callers — see the comment in `recordchildworkflowcompleted/api.go`: *"it should be fine e.g. that ci.Clock is nil"*. The child execution change is a **no-op unless the local cluster recorded a clock by starting the child itself**, which for a passive cluster means after a failover. On a standby that never started the child, both sides are nil. For activities, preserved bits suppress recreation of a timer task, so it is worth being explicit about why that is safe on the passive side: an expired-but-unresolved standby timer returns `ErrTaskRetry` and stays in the queue rather than being consumed, so there is nothing to recreate. The one path that does drop a task, past `StandbyTaskMissingEventsDiscardDelay`, logs a warning and increments `task_errors_discarded`, so it cannot happen silently. ## Tests Two new tests at the `applyUpdatesToSubStateMachines` level, one per activated `sanitizeFn`. **Both fail with the shadowing restored** (verified: `expected: 13, actual: 0` for the mask; `local child execution clock was not carried over` for the clock). The gap they close: the existing `getActivityTimerTaskStatus` tests call the decision function directly with a non-nil `current`, so no unit test could observe the *caller* passing nil. Nine passing tests, zero coverage of the wiring. Two pre-existing tests needed updating, both informative: - `TestApplyMutation` / `TestApplySnapshot` failed with *"Unexpected call to IsVersionFromSameCluster"* — that mock was never needed because the nil short-circuit made the cluster check unreachable. The gap is itself evidence the path was dead. - `verifyActivityInfos` asserted `s.Equal(int32(TimerTaskStatusNone), actual.TimerTaskStatus)`, encoding the bug and blocking any correct fix. Replaced with the invariant that actually matters: ```go s.Zero(actual.TimerTaskStatus&^originStatus, "TimerTaskStatus gained a bit that was not set locally") ``` Applying an update may carry over or drop locally set bits, but must never introduce one that was not already set — claiming a timer task exists when none does is the direction that loses timers. That holds regardless of which bits a given case preserves, so it will not need rewriting as the mask logic evolves. ## Related, tracked separately While auditing whether the mask can be trusted, one path was found that moves a deadline without invalidating the mask: unpausing an activity. Paused activities are excluded from the timer sequence, so a timeout task firing during a long pause is dropped as an invalid task, yet `unpauseActivityInfo` leaves the bit set. That is pre-existing and active-side, independent of this PR, and is being tracked as its own change. It is worth noting here because this PR removes the passive side's accidental repair for that class of stale bit: today's blanket wipe clears any stale bit on the next replicated activity update. --- `go build ./service/...`, `go vet`, and golangci-lint (repo config, `--new-from-rev=origin/main`) clean; `./service/history/workflow/... ./service/history/ndc/...` pass across 3 consecutive runs. Two pre-existing flakes are skipped in those runs and unrelated to this change (both reproduce on unmodified `main`): `TestTaskRefresherSuite/TestRefreshSubStateMachineTasks` (nanosecond-differing HSM deadlines, map iteration order — fails 7/12 on main) and `TestMutableStateSuite/*/TestApplyWorkflowExecutionOptionsUpdatedEvent_TimeSkippingConfig` (wall-clock resolution, ~1/30). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## Summary
- Generator execution clamps its processed time range to
`LastProcessedTime`, but `computeFutureActionTimes` (used by both
`Describe` and `UpdateFutureActionTimes`) only floored its starting time
at `max(now, UpdateTime)`, ignoring the watermark.
- This let Describe/`FutureActionTimes` advertise occurrences at or
before `LastProcessedTime` that the generator had already processed or
silently skipped.
- Floors the starting time at `LastProcessedTime` as well, so every
advertised future action is guaranteed to be strictly after the
generator's high water mark.
- Reuses `common/util.MaxTime` for this floor and for the existing
`UpdateTime`-vs-`LastProcessedTime` clamp in
`GeneratorTaskHandler.Execute`, instead of hand-rolled `if X.After(Y) {
Y = X }` checks.
## Test plan
- Added
`TestGeneratorTask_FutureActionTimesRespectLastProcessedTimeWatermark`
in `chasm/lib/scheduler/generator_tasks_test.go`, which pushes
`LastProcessedTime` ahead of "now" and asserts all advertised future
times are strictly after it. Verified it fails without the fix and
passes with it.
- Built on a new scheduler-specific `chasmtest` helper
(`newSchedulerTestEngine` + `updateScheduler`/`readScheduler` in
`helper_test.go`), adapted from PR 0 of the `sch-readable` stack, rather
than the `newTestEnv` rapid harness, so the watermark mutation and task
execution cross the same transaction/read boundaries as production. Left
out the parts of that PR not applicable here (frontend-client plumbing,
the generic side-effect-task firing helper), since
`GeneratorTaskHandler` is a pure task handler that doesn't touch the
frontend client.
- `go test -tags test_dep ./chasm/lib/scheduler/...` passes.
## What changed Adds a few Nexus-specific log tags to the handler-side frontend logger. ## Why Mainly for the request ID to debug Nexus calls across namespaces better. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed Adds logs tags for failures on the Nexus frontend path. ## Why Have more details to correlate issues with requests. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed? - Added a separate `namespace_replication_lifecycle` wide event with `created`, `received`, `processed`, and `dlqed` phases. - Included namespace/task identity, source and target clusters, source task ID, retry attempt count, deterministic task fingerprint, and the serialized namespace replication task. - Included the successful `CreateNamespaceRequest` or resolved `UpdateNamespaceRequest` as `persistence_request` on `processed`; duplicate, stale, and skipped tasks omit it. - Passed receiver-side diagnostic metadata through a typed context so the existing `TaskExecutor.Execute` and create/update handler signatures remain unchanged. - Added the dedicated, default-off `system.emitNamespaceReplicationLifecycleEvents` dynamic-config gate, checked explicitly at both the processor and processed-event emitter. - Preserved the namespace replication queue message ID as `source_task_id` when reading tasks. ## Why? Namespace CRUD events describe user-visible namespace mutations, but do not show whether the resulting namespace replication task was queued, received, applied, retried, or sent to the DLQ. These events provide that transport and processing audit trail without additional persistence reads. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) Commands: ```text go test -tags test_dep ./common/wideevents ./common/namespace/nsreplication ./service/worker/replicator ./service/frontend make GOLANGCI_LINT_FIX=false GOLANGCI_LINT_BASE_REV=origin/main lint-code ``` Local two-cluster testing covered create, update, and failover after rebasing onto `origin/main`. Each task produced linked `created -> received -> processed` events, and `processed` contained the expected persistence request. With the dynamic-config flag off, namespace replication still completed and neither cluster emitted a matching lifecycle event.
### What changed Sets gobreaker's OnStateChange hook on the outbound queue circuit breaker pool, logging every transition. ### Why Obtain more details for debugging curcuit breaker in production. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oralio#10878) ## What changed? A blocked `ChasmEngine.pollComponent` now returns a `ShardOwnershipLostError` as soon as its shard moves off this host — by adding a `select` case on the shard's lifecycle context — instead of blocking until the request context deadline. ## Why? Follow-up to temporalio#10860 (requested in review): `pollComponent` had the same gap as the `GetWorkflowExecutionHistory` long poll — nothing in its `select` was tied to shard lifecycle, so a poll in flight when its shard moved stalled until timeout. This lets the caller redirect to the new owner immediately. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s)
Go 1.27 prerequisite that upgrades golangci-lint, goimports, and stringer to versions compatible with the new toolchain.
…ter closure (temporalio#11628) ## What changed? Makes Standalone Activity conflict updates idempotent by recording the `requestID` when attaching callbacks or links and recognizing duplicate request IDs. I needed to add a dedicated CHASM error for ## Why? This prevents a successful attachment whose response was lost from failing on retry or duplicating/replacing callbacks and links. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s)
Go 1.27 prerequisite that applies the `atomictypes` Go fixer to use typed atomic values.
## What changed? Adds `namespace_lifecycle` start and finish events for the namespace handover, force replication, and catchup system workflows. The events carry workflow identity and the core operation inputs. Finished events classify the result as succeeded, canceled, or failed. Force replication reports its cumulative verified workflow count and emits only one start and one finish across a continue-as-new chain. Emission uses one shared activity, the existing `system.emitNamespaceLifecycleEvents` gate, disconnected cleanup for cancellation, and workflow versioning for replay compatibility. Existing shard handover events are unchanged. ## Why? These system workflows currently have no consistent operation-level event pair, which makes it difficult to correlate a namespace migration request with its final outcome. ## How did you test it? - [x] covered by existing tests - [x] added new unit test(s) `go test -tags test_dep ./common/wideevents ./service/worker/migration` `make fmt-imports` `make lint-code` reports no issues introduced by this change; the repository-wide target still reports existing findings on current `main`. ## Potential risks The terminal event is best effort and cannot run after server-side workflow termination or workflow run timeout because those outcomes do not execute workflow cleanup.
## What changed? - Clear an activity’s timer-task status when it is unpaused so timeout tasks are regenerated. - Make ResetActivity with keepPaused=false fully unpause both scheduled and running activities, including clearing pause metadata. - Add and strengthen unit and functional coverage for unpause, reset-unpause, timer regeneration, and keepPaused=true. ## Why? Timeout tasks can fire while an activity is paused and be discarded. Previously, the activity’s timer-task status still indicated that those tasks existed, preventing them from being recreated after unpause and potentially making the timeout ineffective. ResetActivity also bypassed normal unpause handling, and running activities returned early without clearing their paused state. ## How did you test it? - [X] built - [ ] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [X] added new functional test(s) ## Potential risks Unpausing now invalidates existing timeout tasks and recreates the next applicable timer during transaction close. This is correct, but a behavioral change. Stale queued tasks may still be processed and discarded through the existing stamp validation.
## What changed?
- Emit `remote_cluster_lifecycle` wide events for successful and failed
`AddOrUpdateRemoteCluster` and `RemoveRemoteCluster` calls through both
Operator and Admin APIs.
- Capture the request, request fingerprint, caller/auth context when
available, remote response, persistence request, authoritative
pre-mutation persistence state for upserts, cached pre-mutation state
for removals, requested connection/replication transitions, mutation,
outcome, and terminal error details.
- Preserve the `NamespaceLifecyclePayload` envelope and place
remote-cluster-specific fields in `details`.
- Gate emission with the existing `system.emitNamespaceLifecycleEvents`
dynamic config.
## Why?
Remote-cluster connection, replication, and removal changes need an
auditable record that shows what was requested, what state existed
before the mutation, and whether the request succeeded or failed.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] added new unit test(s)
- [ ] added new functional test(s)
Commands:
```text
go test -tags test_dep ./service/frontend ./common/wideevents
make lint-code
```
Local E2E against `development-cluster-a.yaml` covered:
- Operator: create enabled, update disabled, remove.
- Admin: create disabled, update enabled, remove.
- Operator removal rejected while an intact global namespace referenced
both the local and remote clusters.
<details>
<summary>Successful Operator update: connection and replication
disabled</summary>
```json
{
"details": {
"api": "operator",
"call_origin": "AddOrUpdateRemoteCluster",
"caller_type": "api",
"local_cluster": "cluster-a",
"mutation": "updated",
"outcome": "succeeded",
"persisted_before": {
"cluster_address": "127.0.0.1:9433",
"cluster_id": "5234378e-93a3-42e7-9ef0-b0a68ec20a4f",
"cluster_name": "cluster-remote-lifecycle-update-e2e",
"failover_version_increment": 100,
"history_shard_count": 32,
"http_address": "127.0.0.1:9443",
"index_search_attributes": null,
"initial_failover_version": 21,
"is_connection_enabled": true,
"is_global_namespace_enabled": true,
"is_replication_enabled": true,
"tags": {
"environment": "local-e2e",
"purpose": "remote-cluster-lifecycle-update-wide-events"
},
"use_cluster_id_membership": false,
"version": 1,
"version_info": null
},
"persistence_request": {
"cluster_metadata": {
"cluster_address": "127.0.0.1:9433",
"cluster_id": "5234378e-93a3-42e7-9ef0-b0a68ec20a4f",
"cluster_name": "cluster-remote-lifecycle-update-e2e",
"failover_version_increment": 100,
"history_shard_count": 32,
"http_address": "127.0.0.1:9443",
"index_search_attributes": null,
"initial_failover_version": 21,
"is_connection_enabled": false,
"is_global_namespace_enabled": true,
"is_replication_enabled": false,
"tags": {
"environment": "local-e2e",
"purpose": "remote-cluster-lifecycle-update-wide-events"
},
"use_cluster_id_membership": false,
"version_info": null
},
"version": 1
},
"remote_cluster": "cluster-remote-lifecycle-update-e2e",
"remote_cluster_id": "5234378e-93a3-42e7-9ef0-b0a68ec20a4f",
"request": {
"enable_remote_cluster_connection": false,
"enable_replication": false,
"frontend_address": "127.0.0.1:9433",
"frontend_http_address": "127.0.0.1:9443"
},
"request_fingerprint": "e356fbe885b9f750f5e9a35ec01c29dfaf8f82f135bbfefd93304b50dcdbcb89",
"requested_connection_transition": "disabled",
"requested_replication_transition": "disabled"
},
"event_name": "remote_cluster_lifecycle",
"instrumentation_scope": "go.temporal.io/server/common/wideevents",
"namespace": "N/A",
"namespace_id": "N/A",
"phase": "remote_cluster_upsert"
}
```
</details>
<details>
<summary>Failed Operator removal: cluster still referenced by global
namespace</summary>
```json
{
"details": {
"api": "operator",
"cached_before": {
"cluster_id": "992aa482-40a4-46ad-a4ec-c15d853cb37c",
"cluster_name": "cluster-guard-e2e",
"http_address": "127.0.0.1:9543",
"initial_failover_version": 31,
"is_connection_enabled": true,
"is_replication_enabled": true,
"rpc_address": "127.0.0.1:9533",
"shard_count": 32,
"tags": {
"environment": "local-e2e",
"purpose": "cluster-removal-namespace-guard"
}
},
"call_origin": "RemoveRemoteCluster",
"caller_type": "api",
"error": "cannot remove cluster \"cluster-guard-e2e\": still referenced by namespace \"cluster-guard-e2e-ns\"",
"error_code": "FailedPrecondition",
"error_type": "*serviceerror.FailedPrecondition",
"local_cluster": "cluster-a",
"mutation": "unknown",
"outcome": "failed",
"remote_cluster": "cluster-guard-e2e",
"remote_cluster_id": "992aa482-40a4-46ad-a4ec-c15d853cb37c",
"request": {
"cluster_name": "cluster-guard-e2e"
},
"request_fingerprint": "9992050714897282663c1cc8a216ad99eff3aee746c24ac1c2a510a5aa9fd23e"
},
"event_name": "remote_cluster_lifecycle",
"instrumentation_scope": "go.temporal.io/server/common/wideevents",
"namespace": "N/A",
"namespace_id": "N/A",
"phase": "remote_cluster_remove"
}
```
</details>
## What changed? Adds `testcontext.EnsureRemaining` and has `await` use it so long await calls can request additional test-scoped context time while still respecting the test context cap. ## Why? Await calls can need more time than the default test context has left (esp after the environment setup). Extending the test timeout in this way allows for (1) stuck tests to fail earlier than the default test timeout and (2) legitimately longer running tests to pass without manually tweaking the test timeout. --------- Co-authored-by: Sean Kane <sean.kane@temporal.io>
## Summary - capture post-transaction mutable state for fresh `NotFound` snapshots and `IsFirstSync` creation - report successful verify history repairs as `outcome=backfilled` - include the repaired event range and `new_run_id` in the verify applied event - add regression coverage for fresh zombie applies and non-current-branch backfills ## Testing - `go test ./service/history/ndc ./service/history/replication ./common/wideevents -count=1`
…io#11851) ## What Stop retrying worker command dispatch when the failure is a poller timeout (UpstreamTimeout). Transport errors (gRPC unreachable, connection refused) are still retried. ## Why When a worker is gone, each `DispatchNexusTask` attempt blocks a goroutine for the full 10s dispatch timeout waiting for a poller that will never arrive. With 3 max attempts, that's ~32s of blocked resources per command. At scale this adds up: if many workers go away simultaneously (e.g., deployment rollout), each dead worker can accumulate pending cancel commands. With per-destination concurrency of 100, this means up to 100 goroutines × N dead workers × 10s per attempt × 3 attempts — thousands of goroutines blocked on matching RPCs, causing memory pressure and connection buildup. Since worker commands are best-effort, there's no value in retrying after the first timeout. If no poller appeared in 10s, the worker is likely gone. ## How did you test it? - Unit tests: updated `TestExecute_UpstreamTimeout` and `TestHandleError_UpstreamTimeout_ReturnNil` to verify no retry on timeout while transport errors still retry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…o#11853) ## What changed? Workflow start task refresh now uses the workflow start versioned transition instead of the latest execution-state transition. This limits workflow start task regeneration to replication ranges that include the initial workflow transition. ## Why? Later execution-state changes can advance LastUpdateVersionedTransition while the workflow remains running. Partial task refresh then incorrectly treats the workflow as newly started and creates another WorkflowRunTimeoutTask. ## How did you test it? - [ ] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Ran the focused TaskRefresher unit test and the single-cluster passive replication workflow and parallel-activity scenarios with exact active/passive task comparison enabled. ## Potential risks This relies on workflow start being transition count 1, matching transition-history initialization. Full refresh still includes the initial transition and continues regenerating required workflow start tasks.
…#11854) ## What changed? The task refresher no longer creates an UpsertExecutionVisibilityTask for the initial visibility transition, which is already represented by StartExecutionVisibilityTask. Internal workflow state changes now mark visibility as updated only when the externally visible workflow status also changes. ## Why? Scheduling the first workflow task changes the internal execution state from CREATED to RUNNING while its public status remains RUNNING. That advanced VisibilityLastUpdateVersionedTransition without producing an active visibility task, causing passive refresh to create a passive-only visibility upsert. ## How did you test it? - [ ] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Ran focused mutable-state and TaskRefresher unit tests and the single-cluster passive replication workflow and parallel-activity scenarios with exact active/passive task comparison enabled. ## Potential risks Internal execution-state-only changes no longer advance the visibility transition marker. Public workflow status changes still do, and continue to generate the corresponding visibility work.
…11893) I'm trying to reduce the size of a monstrous PR I'm working on, this was one of the changes that could be teased out independently. ## Problem Four places independently unwrapped a `*status.Status` from an error with the same 12-line dance: `common.IsRetryableRPCError`, `commonnexus.ConvertGRPCError`, and a private `isRetryableRPCResponse` in both `chasm/lib/callback` and `service/history/hsm/callbacks`. ## Fix Extract it as `common.GetRPCStatus(err) (*status.Status, bool)` and delete the copies. No behavior change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
) ## What changed? Add a new unit test for `common/callbacks.ValidatorConfig`. ## Why? While working on a PR that extended `ValidatorConfig` and adding new fields, some bugs crept in because the new fields weren't checked in the `Validate` function. This unit test will now catch those automatically, by running `Validate` on an empty `ValidatorConfig`, and then confirming that every field of the type is found in the error message. (Reporting that it is uninitialized.) This PR also allows me to remove this tiny change from an otherwise monstrous PR I am trying to slim down. ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks None --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…poralio#11668) ## What changed? `AdminRebuildMutableState`, invoked by `tdbg workflow rebuild`, no longer replaces the mutable state start time with the current time. It reuses the run's recorded start time, falling back to `ExecutionInfo.StartTime`. Only if both `ExecutionState.StartTime` and `ExecutionInfo.StartTime` are empty, the current time will be used. The rebuild now also refreshes the timeout timer tasks so the run and execution deadlines are anchoraed at the time of the call, which is what reset already does. Without it, preserving the start time would make a workflow rebuild after its deadline timeout immediately. It also records ExecutionInfo.MutableStateRebuildTime so that a repaired run could be identify by operator. ## Why? Moving the start time to now, often results in workflow with a negative duration, since the end time is kept as is. Downstream systems may handle this incorrectly. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks 1) Some downstream systems may prefer the workflow start time to move. 2) Admin may intent for the workflow to have a new timeout instead of getting a timeout
## What changed? - Reuse one capture handler across each dropped-task metric table test. - Install the handler before any task queue partition is created. ## Why? Each table case replaced `matchingEngine.metricsHandler` while a prior case's task queue could still be initializing asynchronously. The queue goroutine reads the handler while emitting physical task queue gauges, producing the race reported in https://github.com/temporalio/temporal/actions/runs/33111032921/job/98654644755. The capture handler already synchronizes capture sessions, so keeping the engine handler stable removes the unsafe write without changing production matching behavior. ## How did you test it? - [x] Priority and fairness variants: all four affected tests passed 50 iterations under the race detector. - [x] Classic variant: no race was reported; a repeated batch hit an unrelated task-queue-close teardown log, and the required isolated rerun passed. - [x] Adversarial review completed with no findings. ## Potential risks Low. This only changes test setup and introduces no imports, dependencies, or production behavior. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Test-only changes to metric capture setup; no production matching behavior. > > **Overview** > Fixes a **data race** in matching engine dropped-task metric tests by keeping `matchingEngine.metricsHandler` stable for the whole table-driven test instead of swapping it in every subtest. > > `captureDroppedOnEngine` is now documented and enforced to run **once** before any task queue partition exists (`getTaskQueuePartitions` must be empty). The four `tasks_dropped` poll-path tests install a single `metricstest.CaptureHandler` at the start and reuse it across subtests; each subtest only starts/stops a capture session (`defer StopCapture`). That avoids concurrent reads of the handler from async partition initialization while another subtest overwrites it. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit a2f545f. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
## What changed?
- Fix parsing ExecutionStatus in tuple (eg: `ExecutionStatus IN
('Running', 'Completed')`)
- Fix parsing negative double values (eg: `CustomDouble > -1.5` was
returning an error)
- Return error when comparing non-bool types with boolean value (eg:
`StartTime = true` was not returning an error).
- Return error when comparing Text type search attribute with an empty
string/no tokens (eg: `CustomText = ' '`). It was already returning an
error with PostgreSQL and SQLite, now also returning an error with MySQL
and Elasticsearch.
- Added unit tests covering all Visibility stores to make sure they all
behave the same. One exception: `ExecutionStatus STARTS_WITH` works with
Elasticsearch, but not with SQL. That's because in Elasticsearch we
store as string, while in SQL we store as int.
## Why?
Bug fixes, and comprehensive unit tests to ensure uniform behavior.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed? Previously you could Terminate a standalone Nexus operation that was already in the Canceled state. This PR closes that gap. ## Why? That behavior was never intended, and is a bug. Canceled is already a terminal state. Not only does terminating an already canceled operation not make sense, it also opens up weird issues where we would mutate a resource that should be in a terminal state. ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks This is a behavior change, and could break a test or something that was relying on this behavior earlier. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…failover (temporalio#11815) ## What changed? - Added request-only `OrphanedChildReplacementInfo`, carrying the parent’s current Version History as branch evidence. - When enabled, a retried child start may atomically terminate an orphaned conflicting run and create a replacement. - Replacement is allowed only when the conflicting child: - belongs to the same parent run; - was initiated on a losing parent branch, while the incoming initiation is on the current branch; - is the first run in its execution chain; - contains only `WorkflowExecutionStarted`; - has no pending in-memory Update. - Existing request-ID deduplication remains ahead of replacement handling. - Added outcome metrics, a post-commit success log, and the disabled-by-default `history.enableOrphanedChildWorkflowReplacement` setting. ## Why? After force failover, a child created from a losing parent branch may conflict with the same child start reissued by the winning branch. Normal workflow-ID conflict handling records `WORKFLOW_ALREADY_EXISTS`, leaving the parent unable to make progress and the original child orphaned. Re-linking the existing child is unsafe because its parent coordinates are stored in immutable history. This change instead replaces it only when the new active cluster sees no progress beyond `WorkflowExecutionStarted`. Ownership and progress are rechecked while holding the child lock, and termination plus replacement creation are committed atomically. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Validated with: - `go test -tags test_dep ./service/history/api ./service/history -count=1` - Focused parent-child XDC integration test - `make proto` - `make lint-code` ## Potential risks - The decision uses the new active cluster’s locally visible state. During a network partition, it cannot know whether the previous active advanced the child beyond the replicated state. - Replacement terminates an existing run. The feature is disabled by default and fails closed unless every ownership, branch, and progress check passes under the child lock. - The request carries the parent’s full current `VersionHistoryItems` list. An unusually long version history may increase RPC size and could exceed internal gRPC limits; oversized requests fail instead of recovering the child. - All history hosts processing child starts in an enabled cluster must understand the request field. An older host may ignore it and record `WORKFLOW_ALREADY_EXISTS`.
…io#11868) ## What changed? When an active Parent `StartChildExecution` task regenerated from late replication finds that the Child is already closed: - Resolve the Child’s current run by Workflow ID using `GetMutableState`. - Verify that the current run belongs to the expected execution chain using `FirstExecutionRunId`. - Refresh only a final current run (`Completed`, `Failed`, `Canceled`, `Terminated`, or `TimedOut`) to regenerate its `CloseExecution` task. - Allow the regenerated task to record Child completion in the Parent. Recovery is restricted to the path where the Parent still has a pending, started `ChildExecutionInfo`. If completion has already been recorded, that entry has been removed and recovery is not triggered. The change also adds: - A namespace-level `history.enableChildWorkflowCompletionRecovery` kill switch. - A `child_workflow_completion_recovery_attempts` metric. - Logging for terminal recovery attempts and Workflow ID reuse mismatches. - Unit coverage for terminal recovery, Workflow ID reuse, disabled recovery, deleted current runs, skipped running/paused successors, and retryable `GetMutableState`/Refresh failures. - XDC coverage for: - A normally completed Child. - A Child that Continued-As-New before its final run completed. - A reset Child whose current reset run completed. - A namespace-scoped, test-only source replication conversion interceptor. The XDC harness uses it to pause Parent raw-task conversion and deterministically reproduce source-side replication backlog without disabling replication, changing cluster metadata, or acknowledging/dropping tasks. ## Why? With cross-shard replication ordering, the Child can arrive and close on the new active cluster before the Parent arrives. The Child’s original `CloseExecution` task cannot notify the missing Parent, and the resulting `NotFound` is acknowledged. When Parent replication arrives later, task refresh regenerates its `StartChildExecution` task. That task finds that the Child has already closed, but previously did not recreate the lost completion notification. Consequently, the Parent could permanently retain the pending Child without recording `ChildWorkflowExecutionCompleted`. The regenerated Parent task now resolves the Child’s current execution chain and refreshes its final current run. This recreates the Child’s `CloseExecution` task, allowing completion recording to converge. `GetMutableState` is used instead of `DescribeWorkflowExecution` because recovery only needs the current execution, workflow status, and first execution run ID. It avoids constructing unrelated Describe data such as pending operations, memo, search attributes, callbacks, and Nexus state. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Commands run: - Targeted `TransferQueueActiveTaskExecutorSuite` recovery unit tests with `-tags test_dep`. - Normal, Continue-As-New, and reset XDC recovery tests with `-tags test_dep`. - Full `TestParentChildXDCTestSuite`. - `make lint-code-fast` with zero issues. - CI-equivalent `govulncheck v1.7.0 -json ./...` with exit code 0. Not run: - Full `saas-temporal` dependency-injection and service-startup compatibility validation. ## Potential risks - `RefreshWorkflowTasks` performs a full task refresh rather than regenerating only `CloseExecution`. It invalidates previously generated tasks and may also regenerate visibility, retention, archival, pending Child, and HSM/CHASM tasks. - In the narrow case where an existing `CloseExecution` task has `DeleteAfterClose=true`, full refresh regenerates it with `DeleteAfterClose=false`. If the original task is invalidated before processing, deletion may not complete. - If the current Child run has already been deleted before Parent replication arrives, there is no execution to refresh and completion still cannot be recovered. A TODO records this remaining gap.
## Summary We held back several fixes in the V1 schedule workflow and bundled them into a single atomic commit with the intention of avoiding having to perform multiple increments of the workflow versioning. The changes are broadly: Migration v1->v2 fixes + CGS version changes. ## Risk This touches V1 schedules workflow, and a mistake risks nondeterminitism. This is relatively high impact and has quite a bit of subtlety. This is therefore a commit we need to merge with care ## Testing and validation 1. Changes have been manually tested on individual PRs already 2. Changes have been manually run through CGS's validation suite (cc @liam-lowe) Our intention is to a) make the 163 release cut and b) continue to manually testing a few operational scenarios manually while this is being merged in. If we see problems we can hotfix this. Manually doing some scenario testing will take time so we intend to do this in parallel. Some of the scenarios we will test manually are: 1. Deploy and rollback to an earlier version (probably with LLM assist, but running locally) 2. Manually lifting the the version of a schedule via dynamic config (manually) ensuring there's no nondeterminitism risk 3. V2->v1 Rollback works as expected Some of the tests we've already run and manually validated(cc @liam-lowe) 1. Version floor works as expected 2. That this change fixes v1->v2 migration 3. That this change avoids the problematic history entries which broke a customer using the coinbase Ruby SDK ## LLM Summary This PR stacks a series of changes to the V1 (legacy) scheduler workflow: two migration-correctness fixes, and dynamic-config levers to safely roll the V1 workflow version forward/backward across a multi-cluster deployment without requiring continue-as-new. Summary of each PR in the stack, in order: ### temporalio#11462 — V1→V2 migration-eligibility fix and migrated-start ID (David Porter) Combines two previously separate fixes under one shared version bump (`v13`), avoiding two separate version-bump deploys for the same version number: - `RefreshBeforeMigrationCheck`: fixes a bug that was preventing V1→V2 migration from ever succeeding under default configuration. - `PreserveMigratedStartIDs`: preserves the request IDs workflows were originally started with, in case of a rollback. - Adds a guard against late migrations caused by a transient error bouncing a migration attempt and the schedule re-attempting it after waking again. - Follows the existing two-phase-rollout pattern: this PR only teaches the scheduler to *understand* v13 (gated behind `hasMinVersion(13)`); `CurrentTweakablePolicies.Version` itself stays at v12 pending a follow-up activation deploy. ### temporalio#11588 — Fix schedule action delay after refresh (Alex Stanfield) - `processWatcherResult` only recorded `DesiredTime` on the long-poll path; when a refresh instead discovered the prior action had completed, `DesiredTime` stayed unset, inflating the reported `ScheduleActionDelay` for back-to-back buffered actions. - Backdates `DesiredTime` on the refresh path too, gated behind a new version, `RefreshCompletionDesiredTime` (v14). - Handles two follow-on correctness gaps surfaced in review: `ALLOW_ALL` starts (never blocked by a running workflow, so shouldn't be backdated to an unrelated close time) and multiple tracked executions in one refresh pass (must only move the recorded close time forward, never backward). - Extracts the decision into a pure, unit-testable `shouldBackdateDesiredTime` function, and a shared `IgnoresRunningWorkflow` helper so `ProcessBuffer` and the new backdate logic can't drift apart on what "waits for a running workflow" means. ### temporalio#11827 — Sort BufferedStarts by due time on CHASM-to-V1 rollback (Alex Stanfield) - `CHASMToLegacyStartScheduleArgs` (the CHASM→V1 rollback conversion) appended trigger-derived `BufferedStarts` after the regular pending ones unconditionally, without sorting by due time. - V1's buffer-processing code assumes `BufferedStarts[0]` is always the earliest-due pending start — an invariant not guaranteed across a rollback, since manual triggers are built by iterating a Go map (randomized order) and appended regardless of their own due time. - Fix: sort the combined list by `ActualTime` after appending, mirroring the sort already applied to `RecentActions` a few lines above. ### temporalio#11831 — Re-evaluate V1 version ceiling per iteration (Alex Stanfield) - The base PR (temporalio#10817, by liam-lowe) added `worker.schedulerV1VersionCeiling` but only applied it on the first `tweakables` evaluation, so raising/removing a ceiling left an in-flight workflow stuck at the capped version until continue-as-new. - Re-reads the ceiling on every evaluation: `next version = max(recorded version, min(binary default, current ceiling))`. The recorded version stays monotonic — a newly-lowered ceiling never downgrades a version already recorded in the current run, but a raised/removed ceiling lets the workflow advance at its next wakeup instead of waiting for continue-as-new. - Recorded via the existing `MutableSideEffect`, so replay consumes history rather than re-evaluating live dynamic config. ### temporalio#11856 — Add worker.schedulerV1VersionOverride (Alex Stanfield) - temporalio#11831 makes the ceiling dynamic, but a ceiling can only restrict a binary default, never promote past it. Adds the namespace-level override needed to activate a newer version already supported by the binary. - `requested version = valid override, otherwise binary default`; `next version = max(recorded version, min(requested version, current ceiling))`. Default `-1` retains the binary default; values below the binary default or above `LatestSchedulerWorkflowVersion` are ignored. - The frontend uses the same override for initial schedule memo/list-info construction, so version-dependent metadata agrees with the first worker task. Base of the stack: temporalio#10817 (liam-lowe) introduced the original (static-per-run) `worker.schedulerV1VersionCeiling` dynamic config that temporalio#11831/temporalio#11856 build on. ## Why? To support safe, gradual rollout/rollback of V1 scheduler workflow version bumps (migration fixes, action-delay-after-refresh fix) in a cross-version multi-cluster topology, without requiring continue-as-new to pick up config changes. ## How did you test it? - [x] built - [x] covered by existing tests - [x] added new unit test(s) --------- Co-authored-by: liam-lowe <56076876+liam-lowe@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: alex.stanfield <13949480+chaptersix@users.noreply.github.com> Co-authored-by: Stephan Behnke <stephanos@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: michaely520 <michaely520@users.noreply.github.com> Co-authored-by: Feiyang Xie <feiyang3cat@outlook.com> Co-authored-by: Kannan <rkannan82@users.noreply.github.com> Co-authored-by: Fred Tzeng <41805201+fretz12@users.noreply.github.com> Co-authored-by: Lakshay <54310363+Lakshaymiddha@users.noreply.github.com> Co-authored-by: samm <sam.mathis@temporal.io> Co-authored-by: Quinn Klassen <klassenq@gmail.com> Co-authored-by: Will Duan <xinw.duan@gmail.com> Co-authored-by: Qian Chen <qyc5937@gmail.com> Co-authored-by: Prathyush PV <prathyush.pv@temporal.io> Co-authored-by: Sean Kane <sean.kane@temporal.io> Co-authored-by: mavemuri <74267563+mavemuri@users.noreply.github.com> Co-authored-by: Rodrigo Zhou <rodrigo.zhou@temporal.io> Co-authored-by: Brian VanLoo <brian.vanloo@gmail.com> Co-authored-by: akbala <akbala@gmail.com> Co-authored-by: Dan Davison <dan.davison@temporal.io> Co-authored-by: Chris Smith <chrsmith@users.noreply.github.com>
## What changed? - Corrected standalone activity dispatch-reason classification for resets and retries. - Added a dispatch hook that runs after CHASM validation and before sending the activity to Matching. - Added coverage for retry, reset, and pause/unpause transitions. ## Why? Consumers need a reliable way to classify and intercept standalone activity dispatches. Reset attempt 1 was previously classified as a retry, and processing before CHASM validation could include stale tasks replaced by reset, update, pause, or unpause operations. These changes ensure dispatch reasons reflect the actual attempt and allow integrations to act only on valid dispatches. ## How did you test it? - [X] built - [ ] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) ## Potential risks - An incorrectly implemented dispatch hook could block, drop, or duplicate an activity dispatch. - Hook errors now propagate through task processing and may cause the dispatch task to retry. - Consumers relying on the previous reset dispatch classification may observe reset attempt 1 changing from RETRY to IMMEDIATE. When no hook is configured, dispatch behavior remains unchanged.
## What changed? Adds a new metric, `task_alertable_attempt`, alongside the existing `task_attempt` histogram in `service/history/queues/executable.go`, to distinguish system-caused retry loops from namespace/customer-caused ones. - New predicate, counting: any `ResourceExhausted` error with `Scope != NAMESPACE` (system-caused throttling, including causes that never set `Scope` at all, e.g. `CIRCUIT_BREAKER_OPEN`), plus a deliberate `BUSY_WORKFLOW` carve-out; and everything else not explicitly excluded. Also excludes `NamespaceNotActive`, `ErrDependencyTaskNotCompleted`, `ErrTaskRetry`, `ErrNamespaceHandover`. - Two emission sites, mirroring `task_attempt`'s: a terminal sample in `Ack()` (unconditional) and an in-flight sample gated by the `task_attempt` 30-attempt limit. - Two tags on both sites: `last_attempt_cause` (the specific `ResourceExhausted` cause, or the Go error type for the catchall case) and `stage` (`terminal` vs. `in_flight`). ## Why? `task_attempt` and its alert can't tell a system-caused retry storm apart from a namespace/customer-caused one — e.g. a task throttled by its own namespace's APS limit climbs into the same 100+ attempt range as a genuinely stuck task, firing the same cluster-wide stuck-task alert and turning it into noise. Full investigation and design rationale in `task-attempt-metric-noise-v2.md`. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s)
>⚠️ I'm a little out of my depth when it comes to the System Nexus Endpoint, and the current requirements around "system payloads". So please review this carefully to keep me honest. > > I'm not sure if this is the right way we want to address this problem, of if instead we should just relax the check in the SDK. (@stuart-wells opened temporalio/api-go#309, which we may also want to take. Independently of this.) ## Context The Temporal server was labeling _everything_ that came through the History Service's `StartNexusOperation` as a system payload. (temporalio#10948) However, when we landed support for visiting nested payloads in the Golang SDK (temporalio/api-go#297) it asserts that all system payloads have BOTH a `"encoding":"binary/protobuf"` AND `"messageType"` metadata key. That's a problem. Now, if the Temporal server were to pick up the latest `api-go` bits, it will introduce test failures. Because we have tests that return payloads encoded with `plain/json` that will fail when ran through the SDK's payload visitor at runtime. ## What changed? ~~This PR makes the requirements surrounding a system payload clearer, and ONLY flags a System Nexus Endpoint payload as a "system payload" IFF if is a properly labeled protobuf message. Otherwise, it doesn't set the system payload tag at all.~~ This PR now adds a check that the payload sent to the System Nexus Endpoint is a protobuf with message type available. It also updates a testcase that was sending `plain/json` responses to the SNE and added a new testcase. ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks We've already shipped code that includes the "label non-Protobuf payloads as system payloads". So it's possible that we've persisted those somewhere, and updating the `api-go` dependency (unless patched there) would cause problems.
## What Change default dynamic config values for worker command dispatch: - `WorkerCommandsDispatchTimeout`: 10s → 5s - `WorkerCommandsMaxAttempts`: 3 → 30 ## Why - **Shorter timeout (5s):** Each dispatch attempt holds an outbound executor thread waiting for a poller. If the worker is gone, that thread is blocked for nothing. Cutting from 10s to 5s means we detect missing workers faster and free up the thread sooner. - **More attempts (30):** The main reason for retrying is to survive matching server restarts — during a rolling restart, dispatch RPCs can fail with transient transport errors until the new pod is ready. With only 3 attempts, commands could be dropped permanently during a routine restart. 30 attempts with the default backoff (initial=1s, coefficient=1.1) spreads retries over ~2 minutes, enough to ride out a restart. Transport errors fail fast (no blocking wait), so more attempts are cheap. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…g shutdown (temporalio#11841) ## What changed? Register worker polls before checking the shutdown cache. ## Why? Prevent a shutdown request from missing a poll that is still starting. The previous implementation handled polls that arrived after shutdown had populated the cache. It did not handle the case where the poll and shutdown calls are interleaved. - The poll checks the cache and sees the worker as active. - Shutdown records the worker and cancels registered polls, but this poll is not registered yet. - The poll registers afterward and remains outstanding. ## How did you test it? Tried adding a unit test to verify this exact race; but this gets into the guts of the impl and makes it hard to read. - [x] existing unit test(s)
## What changed? Removed the extra UpdateCount increment when replicated UpdateInfo entries are inserted into mutable state. Extended mutation and snapshot replication tests to cover a newly replicated workflow update and verify the source count is preserved. ## Why? UpdateCount is already synchronized from the source ExecutionInfo. Incrementing it again while applying UpdatedUpdateInfos caused passive mutable state to report 2 updates when the active mutable state reported 1. ## How did you test it? - [x] run locally and tested manually - [x] added new unit test(s) - [x] added new functional test(s) Ran the focused MutableState ApplyMutation, ApplySnapshot, and UpdateInfos tests. Also ran snapshot-only passive-path replication against the OMES throughput_stress scenario with Workflow Updates, Nexus, child workflows, and continue-as-new; 283 passive applies completed without task or mutable-state differences. ## Potential risks Low. The change only stops passive replication from incrementing a cumulative counter that is already copied from the source artifact.
This changes the weekly CI report to show deltas to help clearly see if sth improved or got worse. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Fix a test data race caused by passing the address of `sql.ErrNoRows` to `assert.ErrorAs`. Since `errors.As` writes the matching error into its target, the assertion could mutate the package-level sentinel while parallel persistence tests read it. Use `require.ErrorIs` to verify sentinel identity without mutating global state.
## What changed? - assign priority key 1 to Worker Deployment and Worker Deployment Version workflow starts - cover the shared start-request builder with a unit test ## Why? Worker versioning workflows run on the per-namespace worker task queue alongside other system workflows. Giving their initial workflow tasks higher-than-default priority prevents versioning operations from being delayed behind default-priority work. ## How did you test it? - `go test -tags test_dep ./service/worker/workerdeployment -run TestMakeStartRequestSetsHighPriority` - `go vet -tags test_dep ./service/worker/workerdeployment`
## What changed? - Fix parsing negative double values. - Consolidated the function to resolve alias (there were two, and not behaving the same way, the legacy was missing some cases). - Added more unit tests to verify the legacy query converter behaves the same as in the new one except for the expected differences. ## Why? Fixes to legacy visibility query converter for SQL and Elasticsearch. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks
This PR replaces the two existing scaling signals for poller autoscaling with improved versions (on a opt-in basis): * Backlog age signal: Instead of checking if backlog age is >200ms (default threshold) to scale pollers, check if overall dispatch latency is greater than the threshold * Ratio signal: Instead of checking if addRate/dispatchRate ratio > 1.2 (default threshold), check if addRate/syncMatchRate > the threshold. This will let us add more pollers if the sync match rate is too low. Both improved signals are only enabled if the `matching.useSignalsV2ForPollerScaling` flag is true - set to `false` by default. ## How we tested it - added unit tests - ran benchmarks in a test cell, with multiple omes workloads (see below) --------- Co-authored-by: Kannan Rajah <kannan.rajah@temporal.io> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
… estimate overall add task rate (temporalio#11699) ## What changed? - use probability of AddTask to root partition to estimate overall add task rate, instead of assuming uniform distribution - to enable this, change the AddTask loadbalancing to ensure that there is at least 1% chance of hitting the root, regardless of how much backlog the root has ## Why? A previous PR changed AddTask load-balancing from uniform random to backlog-aware ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Changes AddTask computation and AddTask load balancing, but improves the estimation of the former, and we are ok with the tradeoff for the latter <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches AddTask write routing and dynamic partition scaling inputs; behavior changes when backlog-aware routing heavily favors child partitions, though a 1% root floor and fallback estimates limit worst-case drift. > > **Overview** > Backlog-aware write routing no longer assumes one root add implies uniform load on every partition. The matching client now returns an **estimated tasks across all partitions** when load-balancing picks the **root** partition, sends it on gRPC metadata (`etap`), and the root partition scaler uses that instead of multiplying by write partition count. > > **Write load balancing** enforces at least a **1% chance** of routing to the root (`writePartitionRootProbabilityFloor`) so sampling stays viable when the root backlog is full. When the root is chosen, the estimate is derived from gap-weighted routing (`randomRound(total/gap0)`); non-root picks carry zero estimate. Fallback paths (uniform random, forced partition via test hooks) still return `partitionCount` on root samples. > > **Scale manager** batches and wakes the partition scaler on **estimated** queue-wide task volume (`AddedTasks(estimatedTasksAllPartitions)`), with threshold `batchSize × currentWrite` once write count is known. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1ee95df. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
Marks three `Predicate` oneof fields, and the messages they hold,
`[deprecated = true]` in `predicates.proto`, with a comment on each
pointing at what replaced them: `TaskTypePredicateAttributes`,
`DestinationPredicateAttributes`, and
`OutboundTaskGroupPredicateAttributes`. Regenerated
`api/persistence/v1/predicates.pb.go`
accordingly.
On top of the annotation, also removes the dead code those three types
left behind:
- The
`*TypePredicate`/`*DestinationPredicate`/`*OutboundTaskGroupPredicate`
case blocks in `AndPredicates`/`OrPredicates`
(`service/history/tasks/predicates.go`). Combining two of these now
falls through to the existing generic `predicates.And`/`predicates.Or`
wrapper instead of a type-specific simplification.
- `// Deprecated: ...` GoDoc comments on `NewTypePredicate`,
`NewDestinationPredicate`, and `NewOutboundTaskGroupPredicate`, so
`staticcheck` flags any new caller.
- Test fixtures that used one of these three types incidentally are
swapped to a still-supported type. The tests that cover these types are
untouched, so that old stored shards work correctly if they ever do
happen to be reloaded.
## Why?
These three types are never constructed anywhere in the server today.
The only place any of them is referenced outside their own message
definitions and unit tests is the read-path switch in
`service/history/queues/convert.go`, which exists to deserialize a
variant if it happens to be stored in a very old shard.
- `TaskTypePredicateAttributes` was added speculatively to be able to
create task-keyed predicates.
- `DestinationPredicateAttributes` and
`OutboundTaskGroupPredicateAttributes` both were introduced during the
outbounb queue's development. Two months later they were unified into
one composite predicate,
`OutboundTaskPredicateAttributes.Group{task_group, namespace_id,
destination}`
Removing their `AndPredicates`/`OrPredicates` case blocks is safe
because both functions already fall back to a generic, correct wrapper
(`predicates.And`/`predicates.Or`) for any pair they don't specifically
simplify. Losing a type's branch only costs simplification precision and
not correctness.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
`make proto` regenerated only `predicates.pb.go`. `go build ./...` and
`go test ./service/history/tasks/... ./service/history/queues/...
./api/persistence/...` all pass.
) ## What changed? Enable the new Visibility unified query converter introduced in v1.30.0 by default. ## Why? The unified query converter is stable, and ready to replace the old one. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks
chaptersix
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The scheduler experiment was validated on a merge of the readability stack (#40) and server main at ad7b229. This PR exposes that existing integration baseline so the readability and experimental scheduler layers form one GitHub stack.
The merge commit is 95d50ed. Its diff includes the intervening main changes and the existing scheduler conflict resolutions. No branch history is rewritten by this stack repair.
Dependency order: #40 → this baseline → #59 → #60 → #61 → #62. All layers remain draft and unmerged.