Merged
Conversation
stephanos
commented
Feb 25, 2026
| if: ${{ !cancelled() }} | ||
| with: | ||
| name: junit-xml--${{github.run_id}}--${{github.run_attempt}}--unit-test | ||
| name: junit-xml--${{ github.run_id }}--${{ steps.get_job_id.outputs.job_id }}--${{ github.run_attempt }}--unit-test |
Contributor
Author
There was a problem hiding this comment.
Aligning artifact name with functional tests to have consistent naming scheme for parsing.
Contributor
There was a problem hiding this comment.
Will this break backwards? so the next week won't have all of the reports but going forward it will?
Contributor
Author
There was a problem hiding this comment.
I think it's worth it since unit tests and integration tests have been flying under the radar anyway so far
stephanos
commented
Feb 25, 2026
| // Group failures by test name | ||
| // Group failures by test name, then remove parent entries whose subtests were observed. | ||
| grouped := groupFailuresByTest(allFailures) | ||
| filterParentTests(grouped, testRunCounts) |
Contributor
Author
There was a problem hiding this comment.
This will remove parent suites from the reports when they have subtests; removes misleading/noisy entries.
stephanos
force-pushed
the
fix-flakereport
branch
from
e5c17ec to
7f5d574
Compare
stephanos
commented
Feb 25, 2026
| // generateSuiteReports creates per-suite flake breakdown from all failures and test runs. | ||
| // Suite flake rate = % of workflow runs where the suite had at least one non-retry failure. | ||
| // Suite flake rate = % of job executions where the suite had at least one non-retry failure. | ||
| func generateSuiteReports(allFailures []TestFailure, allTestRuns []TestRun) []SuiteReport { |
Contributor
Author
There was a problem hiding this comment.
Grouping of suites was too aggressive, now shows all suite runs.
stephanos
force-pushed
the
fix-flakereport
branch
10 times, most recently
from
b499b6e to
9143911
Compare
stephanos
commented
Feb 25, 2026
| ) | ||
|
|
||
| // hoursAgo formats a timestamp as "Xh ago" relative to now. | ||
| func hoursAgo(t time.Time) string { |
Contributor
Author
There was a problem hiding this comment.
I think this is easier as it uses a consistent time unit.
stephanos
force-pushed
the
fix-flakereport
branch
3 times, most recently
from
63c61b7 to
95153bc
Compare
stephanos
commented
Feb 25, 2026
| sr.SuiteName, sr.FlakeRate, sr.FailedRuns, sr.TotalRuns, lastFailure)) | ||
| rate := fmt.Sprintf("%.1f%% (%d/%d)", sr.FlakeRate, sr.FailedRuns, sr.TotalRuns) | ||
| if sr.FlakeRate > 5 { | ||
| rate = "**" + rate + "**" |
Contributor
Author
There was a problem hiding this comment.
Little bit of emphasis.
stephanos
force-pushed
the
fix-flakereport
branch
2 times, most recently
from
c060df7 to
b809a21
Compare
stephanos
force-pushed
the
fix-flakereport
branch
from
b809a21 to
4b01491
Compare
spkane31
approved these changes
Mar 9, 2026
birme
pushed a commit
to eyevinn-osaas/temporal
that referenced
this pull request
Mar 23, 2026
## What changed? Fix issues with flakereport; see comments for details. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) <details><summary>Example</summary> <p> ## Flaky Tests Report - 2026-02-25 11:25:50 ### Overall Statistics * **CI Success Rate**: 4/8 (50.00%) * **Total Test Runs**: 261830 * **Total Failures**: 404 * **Overall Failure Rate**: 1.5 per 1000 tests ### Failure Categories Summary | Category | Unique Tests | |----------|--------------| | CI Breakers | 1 | | Crashes | 1 | | Timeouts | 0 | | Flaky Tests | 129 | ### CI Breakers (Failed All Retries) | Test | CI Runs Broken | Total Failures | Last Failure | Links | |------|---------------|----------------|-------------|-------| | `TestVersionWorkflowSuite/v2/Test_DeleteVersion_QueryAfterDeletion` | 1 | 1 | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/1) | ### Crashes | Test | Flake Rate | Last Failure | Links | |------|------------|-------------|-------| | `unit-test` | **100.0% (2/2)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/1) [2](https://github.com/temporalio/temporal/actions/runs/22296780641/job/1) | ### Flaky Tests | Test | Flake Rate | Last Failure | Links | |------|------------|-------------|-------| | `TestClientDataConverterTestSuite` | **100.0% (1/1)** | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560060) | | `PANIC: runtime error: index out of range [0] with length 0 [recovered, repanicked] — in go.temporal.io/server/tests.TestWorkflowUpdateSuite.func21.2` | **100.0% (1/1)** | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560060) | | `PANIC: Fail in goroutine after TestVersioning3FunctionalSuiteV2/TestWorkflowWithPinnedOverride_NoSticky has completed — in TestVersioning3FunctionalSuiteV0/TestActivityRetryAutoUpgradeDuringBackoff` | **100.0% (1/1)** | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `PANIC: Fail in goroutine after TestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardNoPollForwardForceAsync has completed — in TestVersioning3FunctionalSuiteV2/TestActivityRetryAutoUpgradeDuringBackoff` | **100.0% (1/1)** | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `DATA RACE: Data race detected` | **100.0% (1/1)** | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) | | `PANIC: runtime error: invalid memory address or nil pointer dereference — in go.temporal.io/server/tests.(*DeploymentVersionSuite).TestForceCAN_WithOverrideState.func1` | **100.0% (1/1)** | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestVersioning3FunctionalSuiteV0/TestTransitionDuringTransientTask_WithSignal` | **48.9% (45/92)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456075) [2](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456229) [3](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456270) | | `TestVersioning3FunctionalSuiteV2/TestTransitionDuringTransientTask_WithoutSignal` | **48.9% (45/92)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456093) [2](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456143) [3](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456167) | | `TestVersioning3FunctionalSuiteV2/TestTransitionDuringTransientTask_WithSignal` | **48.9% (45/92)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456136) [2](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456111) [3](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456112) | | `TestVersioning3FunctionalSuiteV0/TestTransitionDuringTransientTask_WithoutSignal` | **43.4% (36/83)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456143) [2](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456236) [3](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456250) | | `TestVersionWorkflowSuite/v2/Test_DeleteVersion_QueryAfterDeletion` | **42.9% (3/7)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/1) [2](https://github.com/temporalio/temporal/actions/runs/22373551837/job/1) [3](https://github.com/temporalio/temporal/actions/runs/22373551837/job/1) | | `TestUserData_FetchesUpTree` | **16.7% (1/6)** | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/1) | | `TestNewServerWithOTEL/with_OTEL_Collector_running` | **16.7% (1/6)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/1) | | `TestInterleavedWeightedRoundRobinSchedulerSuite/TestInactiveChannelDeletionRace` | **16.7% (1/6)** | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/1) | | `TestUserData_FetchesActivityToWorkflow` | **16.7% (1/6)** | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/1) | | `TestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_StartWorkflow` | **16.1% (9/56)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456136) [2](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456111) [3](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068532) | | `TestNewServer` | **15.4% (2/13)** | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/1) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/1) | | `TestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_SignalWithStart` | **11.3% (6/53)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512764) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) [3](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198192) | | `TestDeploymentVersionSuiteV0/TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicy` | **9.6% (5/52)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512848) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198189) [3](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068462) | | `TestDeploymentVersionSuiteV2/TestVersionScavenger_DeleteOnAdd` | **9.6% (5/52)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456111) [2](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512818) [3](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198207) | | `TestDeploymentVersionSuiteV0/TestStartWorkflowExecution_ReactivateVersionOnPinned` | **9.3% (5/54)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512766) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198202) [3](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068483) | | `TestDeploymentVersionSuiteV0/TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinned` | **7.8% (4/51)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512865) [2](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512840) [3](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512870) | | `TestFairnessAutoEnableSuite/Test_Activity_Basic` | **7.7% (4/52)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456112) [2](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512799) [3](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068532) | | `TestDeploymentVersionSuiteV2/TestReactivationSignalCache_Deduplication_UpdateOptions` | **7.7% (4/52)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) [3](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestActivityClientTestSuite/TestActivityScheduleToClose_FiredDuringBackoff` | **7.5% (4/53)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456152) [2](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) [3](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198228) | | `TestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_Reset` | **6.0% (3/50)** | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068487) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560087) [3](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114101) | | `TestDeploymentVersionSuiteV0/TestResetWorkflowExecution_ReactivateVersionOnPinned` | **6.0% (3/50)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512828) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) [3](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068520) | | `TestDeploymentVersionSuiteV2/TestVersionMissingTaskQueues_ValidSetRampingVersion` | **6.0% (3/50)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512818) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198207) [3](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114123) | | `TestTaskQueueStats_Pri_Suite/TestRampingAndCurrentAbsorbUnversionedBacklog/ForceTaskForwardForcePollForwardAllowSync` | **5.9% (3/51)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512818) [2](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068532) [3](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114123) | | `TestWorkerDeploymentSuiteV0/TestSetCurrentVersion_Concurrent_DifferentVersions_NoUnexpectedErrors` | **5.9% (3/51)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198193) [3](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestDeploymentVersionSuiteV0/TestUpdateWorkflowExecutionOptions_ReactivateVersionOnPinned` | **5.9% (3/51)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456393) [2](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512824) [3](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestCronTestClientSuite/TestCronWorkflowCompletionStates` | **5.8% (3/52)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512824) [2](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512861) [3](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/ContinueAsNew` | **5.8% (3/52)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456103) [2](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068523) [3](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestStandaloneActivityTestSuite/TestHeartbeat/HeartbeatKeepsActivityAlive` | **5.8% (3/52)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512824) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198210) [3](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198202) | | `TestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/WorkflowRunTimeout` | **5.8% (3/52)** | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456103) [2](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068523) [3](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_SetCurrentVersion` | **5.8% (3/52)** | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) [2](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) [3](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198228) | | `TestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_UpdateOptions` | 4.1% (2/49) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456093) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560043) | | `TestTaskQueueStats_Classic_Suite/TestAddMultipleTasks_ValidateStats_Cached` | 4.0% (2/50) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestTaskQueueStats_Pri_Suite/TestInactiveVersionDoesNotAbsorbUnversionedBacklog/NoTaskForwardNoPollForwardAllowSync` | 4.0% (2/50) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456111) [2](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198189) | | `TestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardAllowSync` | 4.0% (2/50) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512821) [2](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestWorkerDeploymentSuiteV0/TestSetRampingVersion_AfterDrained` | 4.0% (2/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068420) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestWorkerDeploymentSuiteV2/TestDrainRollbackedVersion` | 4.0% (2/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068525) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestFuncClustersTestSuite/EnableTransitionHistory/TestForceMigration_ResetWorkflow` | 4.0% (2/50) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456146) [2](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068440) | | `TestDeploymentVersionSuiteV0/TestVersionScavenger_DeleteOnAdd` | 4.0% (2/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068551) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestWorkerDeploymentSuiteV2/TestSetCurrentVersion_Concurrent_DifferentVersions_NoUnexpectedErrors` | 4.0% (2/50) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560090) | | `TestPrioritySuite/TestSubqueue_Migration` | 3.9% (2/51) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) [2](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/WorkflowFailureRetry` | 3.9% (2/51) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456152) [2](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560001) | | `TestWorkerDeploymentSuiteV0/TestDrainRollbackedVersion` | 3.9% (2/51) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198228) [2](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068512) | | `TestVersioning3FunctionalSuiteV0/TestPinnedTask_NoProperPoller/NoTaskForwardNoPollForwardForceAsync` | 2.1% (1/48) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068483) | | `TestVersioning3FunctionalSuiteV2/TestPinnedCaN_UpgradeOnCaN_NormalWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsync` | 2.1% (1/48) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114123) | | `TestDeploymentVersionSuiteV0/TestUpdateWorkflowExecutionOptions_SetImpliedPinnedSuccess` | 2.1% (1/48) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512818) | | `TestGetHistoryFunctionalSuite/DisableTransitionHistory/TestGetWorkflowExecutionHistory_All` | 2.1% (1/48) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) | | `TestVersioning3FunctionalSuiteV2/TestUnpinnedCaN` | 2.1% (1/48) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) | | `TestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryNonFirstTask/NoTaskForwardForcePollForwardAllowSync` | 2.1% (1/48) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456112) | | `TestWorkerDeploymentSuiteV0/TestListWorkerDeployments_TwoVersions_SameDeployment_OneCurrent_OneRamping` | 2.1% (1/48) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) | | `TestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardForceAsync` | 2.1% (1/48) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456212) | | `TestVersioning3FunctionalSuiteV2/TestWorkflowWithPinnedOverride_NoSticky/ForceTaskForwardForcePollForwardForceAsync` | 2.1% (1/48) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestAdvancedVisibilitySuiteLegacy/TestListWorkflow_StringQuery` | 2.1% (1/48) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) | | `TestVersioning3FunctionalSuiteV0/TestChildWorkflowInheritance_ParentPinnedByOverride` | 2.1% (1/48) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198284) | | `TestStandaloneActivityTestSuite/TestRequestCancel/MismatchedTokenComponentRef` | 2.1% (1/48) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456111) | | `TestDeploymentVersionSuiteV2/TestVersionMissingTaskQueues_ValidSetCurrentVersion` | 2.1% (1/48) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) | | `TestVersioning3FunctionalSuiteV0/TestPinnedQuery_DrainedVersion_PollersAbsent/ForceTaskForwardNoPollForwardAllowSync` | 2.1% (1/48) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) | | `TestVersioning3FunctionalSuiteV0/TestPinnedWorkflowWithLateActivityPoller/ForceTaskForwardNoPollForwardAllowSync` | 2.1% (1/48) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114104) | | `TestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardAllowSync` | 2.1% (1/48) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512839) | | `TestDeploymentVersionSuiteV2/TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicy` | 2.1% (1/48) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456111) | | `TestDeploymentVersionSuiteV2/TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinned` | 2.1% (1/48) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) | | `TestVersioningFunctionalSuite/TestDispatchQueryOld/ForceTaskForwardNoPollForwardAllowSync` | 2.1% (1/48) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198239) | | `TestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryFirstTask/ForceTaskForwardForcePollForwardForceAsync` | 2.1% (1/48) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardForceAsync` | 2.1% (1/48) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068525) | | `TestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_MultipleVersions_Sorted` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestDeploymentVersionSuiteV2/TestDrainageStatus_SetCurrentVersion_YesOpenWFs` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestVersioning3FunctionalSuiteV0/TestAutoUpgradeCaN_UpgradeOnCaN/ForceTaskForwardNoPollForwardAllowSync` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestWorkerDeploymentSuiteV2/TestSetCurrentVersion_Batching` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068420) | | `TestTaskQueueStats_Classic_Suite/TestCurrentAbsorbsUnversionedBacklog_WhenRampingToUnversioned/NoTaskForwardForcePollForwardForceAsync` | 2.0% (1/49) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560082) | | `TestTaskQueueSuite/TestTaskQueueRateLimit_UpdateFromWorkerConfigAndAPI` | 2.0% (1/49) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456111) | | `TestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardNoPollForwardForceAsync` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestWorkflowUpdateSuite/StickySpeculativeWorkflowTask_AcceptComplete_StickyWorkerUnavailable` | 2.0% (1/49) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560060) | | `TestWorkerDeploymentSuiteV2/TestResourceExhaustedErrors_Converted_To_ReadableMessage` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114116) | | `TestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_TransientWFT_WithSuggest/ForceTaskForwardForcePollForwardForceAsync` | 2.0% (1/49) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456152) | | `TestVersioning3FunctionalSuiteV2/TestPinnedCaN_UpgradeOnCaN_NormalWFT_PinnedOverride_WithSuggest/NoTaskForwardForcePollForwardAllowSync` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardForceAsync` | 2.0% (1/49) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestChildWorkflowSuite/TestCronChildWorkflowExecution` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114114) | | `TestNexusStateReplicationTestSuite/DisableTransitionHistory/TestNexusCallbackReplicated` | 2.0% (1/49) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456192) | | `TestWorkerDeploymentSuiteV0/TestSetWorkerDeploymentRampingVersion_WithCurrent_Unset_Ramp` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestFuncClustersTestSuite/EnableTransitionHistory/TestForceMigration_ClosedWorkflow` | 2.0% (1/49) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198205) | | `TestTaskQueueStats_Classic_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/ForceTaskForwardNoPollForwardForceAsync` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114074) | | `TestWorkflowUpdateSuite/SpeculativeWorkflowTask_ScheduleToStartTimeoutOnNormalTaskQueue` | 2.0% (1/49) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560060) | | `TestStandaloneActivityTestSuite/TestRequestCancel/StaleAttemptToken` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestTaskQueueStats_Classic_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/NoTaskForwardForcePollForwardAllowSync` | 2.0% (1/49) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198284) | | `TestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/ForceTaskForwardForcePollForwardForceAsync` | 2.0% (1/49) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512861) | | `TestTaskQueueStats_Classic_Suite/TestRampingAbsorbsUnversionedBacklog_WhenCurrentIsUnversioned/NoTaskForwardForcePollForwardForceAsync` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestTaskQueueStats_Pri_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/NoTaskForwardNoPollForwardAllowSync` | 2.0% (1/49) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456212) | | `TestTaskQueueStats_Pri_Suite/TestRampingAndCurrentAbsorbUnversionedBacklog/NoTaskForwardNoPollForwardAllowSync` | 2.0% (1/49) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560089) | | `TestFuncClustersTestSuite/DisableTransitionHistory/TestForceMigration_ClosedWorkflow` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114115) | | `TestWorkerDeploymentSuiteV0/TestSetRampingVersion_Concurrent_DifferentVersions_NoUnexpectedErrors` | 2.0% (1/49) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) | | `TestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/NoTaskForwardNoPollForwardAllowSync` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/NoTaskForwardForcePollForwardAllowSync` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/NoTaskForwardForcePollForwardAllowSync` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068523) | | `TestDeploymentVersionSuiteV0/TestForceCAN_WithOverrideState` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestVersioningFunctionalSuite/TestDispatchActivityFailCrossTq/NoTaskForwardForcePollForwardAllowSync` | 2.0% (1/49) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068496) | | `TestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_TransientWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsync` | 2.0% (1/49) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestWorkerDeploymentSuiteV0/TestSetWorkerDeploymentRampingVersion_Batching` | 2.0% (1/49) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestDeploymentVersionSuiteV0/TestDrainageStatus_SetCurrentVersion_NoOpenWFs` | 2.0% (1/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068523) | | `TestScheduleFunctionalSuite/TestNextTimeCache` | 2.0% (1/50) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) | | `TestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/ForceTaskForwardNoPollForwardAllowSync` | 2.0% (1/50) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestPollerScalingFunctionalSuite/TestPollerScalingDecisionsAreSeenProbabilistically` | 2.0% (1/50) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestFairnessSuite/TestMigration_FromFair` | 2.0% (1/50) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114116) | | `TestFairnessAutoEnableSuite/TestMigration_FromClassic` | 2.0% (1/50) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560075) | | `TestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardAllowSync` | 2.0% (1/50) | 23h ago | [1](https://github.com/temporalio/temporal/actions/runs/22367698230/job/64738198193) | | `TestWorkflowTaskTestSuite/TestWorkflowTaskHeartbeatingWithEmptyResult` | 2.0% (1/50) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114142) | | `TestAdvancedVisibilitySuite/TestWorkerTaskReachability_Unversioned_InTaskQueue` | 2.0% (1/50) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestDeploymentVersionSuiteV2/TestDeleteVersion_DeleteRampedVersion` | 2.0% (1/50) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456152) | | `TestWorkerDeploymentSuiteV0/TestDeploymentVersionLimits` | 2.0% (1/50) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114116) | | `TestVersioning3FunctionalSuiteV2/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardForcePollForwardForceAsync` | 2.0% (1/50) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456152) | | `TestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/NoTaskForwardNoPollForwardAllowSync` | 2.0% (1/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068512) | | `TestDeploymentVersionSuiteV2/TestDeleteVersion_Drained_But_Pollers_Exist` | 2.0% (1/50) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456152) | | `TestVersioning3FunctionalSuiteV2/TestQueryWithPinnedOverride_Sticky/ForceTaskForwardForcePollForwardAllowSync` | 2.0% (1/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068512) | | `TestFairnessAutoEnableSuite/TestMigration_FromFair` | 2.0% (1/50) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456152) | | `TestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_SpeculativeWFT_NoSuggest/NoTaskForwardNoPollForwardAllowSync` | 2.0% (1/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068525) | | `TestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_TwoVersions_Sorted` | 2.0% (1/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068512) | | `TestDeploymentVersionSuiteV2/TestDrainageStatus_SetCurrentVersion_NoOpenWFs` | 2.0% (1/50) | 1h ago | [1](https://github.com/temporalio/temporal/actions/runs/22410112796/job/64881456152) | | `TestDeploymentVersionSuiteV2/TestDeleteVersion_ValidDelete` | 2.0% (1/50) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512833) | | `TestScheduleFunctionalSuite/TestListSchedulesReturnsWorkflowStatus` | 2.0% (1/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068523) | | `TestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryFirstTask/ForceTaskForwardForcePollForwardAllowSync` | 2.0% (1/50) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestCronTestSuite/TestCronWorkflow` | 2.0% (1/50) | 44h ago | [1](https://github.com/temporalio/temporal/actions/runs/22329840058/job/64609560092) | | `TestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_NormalWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsync` | 2.0% (1/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068420) | | `TestVersioning3FunctionalSuiteV2/TestAutoUpgradeCaN_UpgradeOnCaN/ForceTaskForwardForcePollForwardAllowSync` | 2.0% (1/50) | 38h ago | [1](https://github.com/temporalio/temporal/actions/runs/22337683305/job/64634068472) | | `TestDeploymentVersionSuiteV2/TestReactivationSignalCache_Deduplication_SignalWithStart` | 2.0% (1/50) | 60h ago | [1](https://github.com/temporalio/temporal/actions/runs/22296780641/job/64495114191) | | `TestVersioning3FunctionalSuiteV2/TestPinnedQuery_DrainedVersion_PollersAbsent/ForceTaskForwardNoPollForwardAllowSync` | 2.0% (1/50) | 21h ago | [1](https://github.com/temporalio/temporal/actions/runs/22373551837/job/64758512837) | ### Flaky Suites | Suite | Flake Rate | Last Failure | |-------|------------|-------------| | `TestAcquireShard_DeadlineExceededErrorSuite` | 2.1% (1/47) | 38h ago | | `TestAcquireShard_OwnershipLostErrorSuite` | 2.1% (1/48) | 44h ago | | `TestActivityClientTestSuite` | **8.3% (4/48)** | 1h ago | | `TestActivityTestSuite` | 2.1% (1/48) | 44h ago | | `TestAdminTestSuite` | 2.1% (1/48) | 44h ago | | `TestAdvancedVisibilitySuite` | 4.2% (2/48) | 44h ago | | `TestAdvancedVisibilitySuiteLegacy` | 2.1% (1/48) | 23h ago | | `TestCallbacksSuiteCHASM` | **8.3% (4/48)** | 1h ago | | `TestChasmTestSuiteLegacy` | 2.1% (1/48) | 44h ago | | `TestChildWorkflowSuite` | 2.1% (1/48) | 60h ago | | `TestClientDataConverterTestSuite` | **100.0% (1/1)** | 44h ago | | `TestClientMiscTestSuite` | 2.1% (1/48) | 44h ago | | `TestCronTestClientSuite` | **6.2% (3/48)** | 21h ago | | `TestCronTestSuite` | 2.1% (1/48) | 44h ago | | `TestDLQSuite` | 4.2% (2/48) | 44h ago | | `TestDeploymentVersionSuiteV0` | **62.5% (30/48)** | 1h ago | | `TestDeploymentVersionSuiteV2` | **22.9% (11/48)** | 1h ago | | `TestFairnessAutoEnableSuite` | **8.3% (4/48)** | 1h ago | | `TestFairnessSuite` | 2.1% (1/48) | 60h ago | | `TestFuncClustersTestSuite` | **8.3% (4/48)** | 1h ago | | `TestGetHistoryFunctionalSuite` | 2.1% (1/48) | 23h ago | | `TestNamespaceInterceptorTestSuite` | **6.2% (3/48)** | 38h ago | | `TestNamespaceSuite` | 1.9% (1/53) | 44h ago | | `TestNexusApiTestSuiteWithTemporalFailures` | 2.1% (1/48) | 44h ago | | `TestNexusStateReplicationTestSuite` | 2.1% (1/48) | 1h ago | | `TestNexusWorkflowTestSuite` | 2.1% (1/48) | 44h ago | | `TestPollerScalingFunctionalSuite` | 2.1% (1/48) | 44h ago | | `TestPrioritySuite` | 4.2% (2/48) | 21h ago | | `TestPurgeDLQTasksSuite` | 2.1% (1/48) | 44h ago | | `TestQueryWorkflowSuite` | 2.1% (1/48) | 38h ago | | `TestRawHistoryClientSuite` | 2.1% (1/48) | 44h ago | | `TestRelayTaskTestSuite` | 2.1% (1/47) | 44h ago | | `TestResetWorkflowTestSuite` | 2.1% (1/48) | 44h ago | | `TestScheduleFunctionalSuite` | **6.2% (3/48)** | 21h ago | | `TestStandaloneActivityTestSuite` | **10.4% (5/48)** | 1h ago | | `TestTLSFunctionalSuite` | 2.1% (1/48) | 44h ago | | `TestTaskQueueStats_Classic_Suite` | **18.8% (9/48)** | 1h ago | | `TestTaskQueueStats_Pri_Suite` | **25.0% (12/48)** | 1h ago | | `TestTaskQueueSuite` | 4.2% (2/48) | 1h ago | | `TestTransientTaskSuite` | 2.1% (1/48) | 44h ago | | `TestUpdateWorkflowSdkSuite` | 2.1% (1/48) | 44h ago | | `TestUserMetadataSuite` | 2.1% (1/47) | 44h ago | | `TestUserTimersTestSuite` | 2.1% (1/47) | 44h ago | | `TestVersionWorkflowSuite` | **20.0% (1/5)** | 21h ago | | `TestVersioning3FunctionalSuiteV0` | **100.0% (48/48)** | 1h ago | | `TestVersioning3FunctionalSuiteV2` | **100.0% (48/48)** | 1h ago | | `TestVersioningFunctionalSuite` | **10.4% (5/48)** | 1h ago | | `TestWorkerDeploymentSuiteV0` | **16.7% (8/48)** | 21h ago | | `TestWorkerDeploymentSuiteV2` | **12.5% (6/48)** | 21h ago | | `TestWorkflowMemoTestSuite` | 2.1% (1/48) | 44h ago | | `TestWorkflowTaskTestSuite` | 4.2% (2/48) | 38h ago | | `TestWorkflowUpdateSuite` | 2.1% (1/48) | 44h ago | </p> </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
Fix issues with flakereport; see comments for details.
How did you test it?
Example
Flaky Tests Report - 2026-02-25 11:25:50
Overall Statistics
Failure Categories Summary
CI Breakers (Failed All Retries)
TestVersionWorkflowSuite/v2/Test_DeleteVersion_QueryAfterDeletionCrashes
unit-testFlaky Tests
TestClientDataConverterTestSuitePANIC: runtime error: index out of range [0] with length 0 [recovered, repanicked] — in go.temporal.io/server/tests.TestWorkflowUpdateSuite.func21.2PANIC: Fail in goroutine after TestVersioning3FunctionalSuiteV2/TestWorkflowWithPinnedOverride_NoSticky has completed — in TestVersioning3FunctionalSuiteV0/TestActivityRetryAutoUpgradeDuringBackoffPANIC: Fail in goroutine after TestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardNoPollForwardForceAsync has completed — in TestVersioning3FunctionalSuiteV2/TestActivityRetryAutoUpgradeDuringBackoffDATA RACE: Data race detectedPANIC: runtime error: invalid memory address or nil pointer dereference — in go.temporal.io/server/tests.(*DeploymentVersionSuite).TestForceCAN_WithOverrideState.func1TestVersioning3FunctionalSuiteV0/TestTransitionDuringTransientTask_WithSignalTestVersioning3FunctionalSuiteV2/TestTransitionDuringTransientTask_WithoutSignalTestVersioning3FunctionalSuiteV2/TestTransitionDuringTransientTask_WithSignalTestVersioning3FunctionalSuiteV0/TestTransitionDuringTransientTask_WithoutSignalTestVersionWorkflowSuite/v2/Test_DeleteVersion_QueryAfterDeletionTestUserData_FetchesUpTreeTestNewServerWithOTEL/with_OTEL_Collector_runningTestInterleavedWeightedRoundRobinSchedulerSuite/TestInactiveChannelDeletionRaceTestUserData_FetchesActivityToWorkflowTestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_StartWorkflowTestNewServerTestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_SignalWithStartTestDeploymentVersionSuiteV0/TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicyTestDeploymentVersionSuiteV2/TestVersionScavenger_DeleteOnAddTestDeploymentVersionSuiteV0/TestStartWorkflowExecution_ReactivateVersionOnPinnedTestDeploymentVersionSuiteV0/TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinnedTestFairnessAutoEnableSuite/Test_Activity_BasicTestDeploymentVersionSuiteV2/TestReactivationSignalCache_Deduplication_UpdateOptionsTestActivityClientTestSuite/TestActivityScheduleToClose_FiredDuringBackoffTestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_ResetTestDeploymentVersionSuiteV0/TestResetWorkflowExecution_ReactivateVersionOnPinnedTestDeploymentVersionSuiteV2/TestVersionMissingTaskQueues_ValidSetRampingVersionTestTaskQueueStats_Pri_Suite/TestRampingAndCurrentAbsorbUnversionedBacklog/ForceTaskForwardForcePollForwardAllowSyncTestWorkerDeploymentSuiteV0/TestSetCurrentVersion_Concurrent_DifferentVersions_NoUnexpectedErrorsTestDeploymentVersionSuiteV0/TestUpdateWorkflowExecutionOptions_ReactivateVersionOnPinnedTestCronTestClientSuite/TestCronWorkflowCompletionStatesTestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/ContinueAsNewTestStandaloneActivityTestSuite/TestHeartbeat/HeartbeatKeepsActivityAliveTestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/WorkflowRunTimeoutTestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_SetCurrentVersionTestDeploymentVersionSuiteV0/TestReactivationSignalCache_Deduplication_UpdateOptionsTestTaskQueueStats_Classic_Suite/TestAddMultipleTasks_ValidateStats_CachedTestTaskQueueStats_Pri_Suite/TestInactiveVersionDoesNotAbsorbUnversionedBacklog/NoTaskForwardNoPollForwardAllowSyncTestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardAllowSyncTestWorkerDeploymentSuiteV0/TestSetRampingVersion_AfterDrainedTestWorkerDeploymentSuiteV2/TestDrainRollbackedVersionTestFuncClustersTestSuite/EnableTransitionHistory/TestForceMigration_ResetWorkflowTestDeploymentVersionSuiteV0/TestVersionScavenger_DeleteOnAddTestWorkerDeploymentSuiteV2/TestSetCurrentVersion_Concurrent_DifferentVersions_NoUnexpectedErrorsTestPrioritySuite/TestSubqueue_MigrationTestCallbacksSuiteCHASM/TestWorkflowNexusCallbacks_CarriedOver/WorkflowFailureRetryTestWorkerDeploymentSuiteV0/TestDrainRollbackedVersionTestVersioning3FunctionalSuiteV0/TestPinnedTask_NoProperPoller/NoTaskForwardNoPollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestPinnedCaN_UpgradeOnCaN_NormalWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsyncTestDeploymentVersionSuiteV0/TestUpdateWorkflowExecutionOptions_SetImpliedPinnedSuccessTestGetHistoryFunctionalSuite/DisableTransitionHistory/TestGetWorkflowExecutionHistory_AllTestVersioning3FunctionalSuiteV2/TestUnpinnedCaNTestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryNonFirstTask/NoTaskForwardForcePollForwardAllowSyncTestWorkerDeploymentSuiteV0/TestListWorkerDeployments_TwoVersions_SameDeployment_OneCurrent_OneRampingTestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestWorkflowWithPinnedOverride_NoSticky/ForceTaskForwardForcePollForwardForceAsyncTestAdvancedVisibilitySuiteLegacy/TestListWorkflow_StringQueryTestVersioning3FunctionalSuiteV0/TestChildWorkflowInheritance_ParentPinnedByOverrideTestStandaloneActivityTestSuite/TestRequestCancel/MismatchedTokenComponentRefTestDeploymentVersionSuiteV2/TestVersionMissingTaskQueues_ValidSetCurrentVersionTestVersioning3FunctionalSuiteV0/TestPinnedQuery_DrainedVersion_PollersAbsent/ForceTaskForwardNoPollForwardAllowSyncTestVersioning3FunctionalSuiteV0/TestPinnedWorkflowWithLateActivityPoller/ForceTaskForwardNoPollForwardAllowSyncTestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardAllowSyncTestDeploymentVersionSuiteV2/TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicyTestDeploymentVersionSuiteV2/TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinnedTestVersioningFunctionalSuite/TestDispatchQueryOld/ForceTaskForwardNoPollForwardAllowSyncTestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryFirstTask/ForceTaskForwardForcePollForwardForceAsyncTestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardForcePollForwardForceAsyncTestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_MultipleVersions_SortedTestDeploymentVersionSuiteV2/TestDrainageStatus_SetCurrentVersion_YesOpenWFsTestVersioning3FunctionalSuiteV0/TestAutoUpgradeCaN_UpgradeOnCaN/ForceTaskForwardNoPollForwardAllowSyncTestWorkerDeploymentSuiteV2/TestSetCurrentVersion_BatchingTestTaskQueueStats_Classic_Suite/TestCurrentAbsorbsUnversionedBacklog_WhenRampingToUnversioned/NoTaskForwardForcePollForwardForceAsyncTestTaskQueueSuite/TestTaskQueueRateLimit_UpdateFromWorkerConfigAndAPITestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardNoPollForwardForceAsyncTestWorkflowUpdateSuite/StickySpeculativeWorkflowTask_AcceptComplete_StickyWorkerUnavailableTestWorkerDeploymentSuiteV2/TestResourceExhaustedErrors_Converted_To_ReadableMessageTestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_TransientWFT_WithSuggest/ForceTaskForwardForcePollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestPinnedCaN_UpgradeOnCaN_NormalWFT_PinnedOverride_WithSuggest/NoTaskForwardForcePollForwardAllowSyncTestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardForceAsyncTestChildWorkflowSuite/TestCronChildWorkflowExecutionTestNexusStateReplicationTestSuite/DisableTransitionHistory/TestNexusCallbackReplicatedTestWorkerDeploymentSuiteV0/TestSetWorkerDeploymentRampingVersion_WithCurrent_Unset_RampTestFuncClustersTestSuite/EnableTransitionHistory/TestForceMigration_ClosedWorkflowTestTaskQueueStats_Classic_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/ForceTaskForwardNoPollForwardForceAsyncTestWorkflowUpdateSuite/SpeculativeWorkflowTask_ScheduleToStartTimeoutOnNormalTaskQueueTestStandaloneActivityTestSuite/TestRequestCancel/StaleAttemptTokenTestTaskQueueStats_Classic_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/NoTaskForwardForcePollForwardAllowSyncTestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/ForceTaskForwardForcePollForwardForceAsyncTestTaskQueueStats_Classic_Suite/TestRampingAbsorbsUnversionedBacklog_WhenCurrentIsUnversioned/NoTaskForwardForcePollForwardForceAsyncTestTaskQueueStats_Pri_Suite/TestCurrentVersionAbsorbsUnversionedBacklog_NoRamping/NoTaskForwardNoPollForwardAllowSyncTestTaskQueueStats_Pri_Suite/TestRampingAndCurrentAbsorbUnversionedBacklog/NoTaskForwardNoPollForwardAllowSyncTestFuncClustersTestSuite/DisableTransitionHistory/TestForceMigration_ClosedWorkflowTestWorkerDeploymentSuiteV0/TestSetRampingVersion_Concurrent_DifferentVersions_NoUnexpectedErrorsTestTaskQueueStats_Classic_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/NoTaskForwardNoPollForwardAllowSyncTestVersioning3FunctionalSuiteV0/TestPinnedQuery_RollbackDrainedVersion/NoTaskForwardForcePollForwardAllowSyncTestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/NoTaskForwardForcePollForwardAllowSyncTestDeploymentVersionSuiteV0/TestForceCAN_WithOverrideStateTestVersioningFunctionalSuite/TestDispatchActivityFailCrossTq/NoTaskForwardForcePollForwardAllowSyncTestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_TransientWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsyncTestWorkerDeploymentSuiteV0/TestSetWorkerDeploymentRampingVersion_BatchingTestDeploymentVersionSuiteV0/TestDrainageStatus_SetCurrentVersion_NoOpenWFsTestScheduleFunctionalSuite/TestNextTimeCacheTestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/ForceTaskForwardNoPollForwardAllowSyncTestPollerScalingFunctionalSuite/TestPollerScalingDecisionsAreSeenProbabilisticallyTestFairnessSuite/TestMigration_FromFairTestFairnessAutoEnableSuite/TestMigration_FromClassicTestTaskQueueStats_Pri_Suite/TestMultipleTasks_WithMatchingBehavior_ValidateStats/ForceTaskForwardNoPollForwardAllowSyncTestWorkflowTaskTestSuite/TestWorkflowTaskHeartbeatingWithEmptyResultTestAdvancedVisibilitySuite/TestWorkerTaskReachability_Unversioned_InTaskQueueTestDeploymentVersionSuiteV2/TestDeleteVersion_DeleteRampedVersionTestWorkerDeploymentSuiteV0/TestDeploymentVersionLimitsTestVersioning3FunctionalSuiteV2/TestPinnedQuery_RollbackDrainedVersion/ForceTaskForwardForcePollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestUnpinnedQuery_NoSticky/NoTaskForwardNoPollForwardAllowSyncTestDeploymentVersionSuiteV2/TestDeleteVersion_Drained_But_Pollers_ExistTestVersioning3FunctionalSuiteV2/TestQueryWithPinnedOverride_Sticky/ForceTaskForwardForcePollForwardAllowSyncTestFairnessAutoEnableSuite/TestMigration_FromFairTestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_SpeculativeWFT_NoSuggest/NoTaskForwardNoPollForwardAllowSyncTestWorkerDeploymentSuiteV0/TestDescribeWorkerDeployment_TwoVersions_SortedTestDeploymentVersionSuiteV2/TestDrainageStatus_SetCurrentVersion_NoOpenWFsTestDeploymentVersionSuiteV2/TestDeleteVersion_ValidDeleteTestScheduleFunctionalSuite/TestListSchedulesReturnsWorkflowStatusTestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryFirstTask/ForceTaskForwardForcePollForwardAllowSyncTestCronTestSuite/TestCronWorkflowTestVersioning3FunctionalSuiteV0/TestPinnedCaN_UpgradeOnCaN_NormalWFT_WithSuggest/ForceTaskForwardNoPollForwardForceAsyncTestVersioning3FunctionalSuiteV2/TestAutoUpgradeCaN_UpgradeOnCaN/ForceTaskForwardForcePollForwardAllowSyncTestDeploymentVersionSuiteV2/TestReactivationSignalCache_Deduplication_SignalWithStartTestVersioning3FunctionalSuiteV2/TestPinnedQuery_DrainedVersion_PollersAbsent/ForceTaskForwardNoPollForwardAllowSyncFlaky Suites
TestAcquireShard_DeadlineExceededErrorSuiteTestAcquireShard_OwnershipLostErrorSuiteTestActivityClientTestSuiteTestActivityTestSuiteTestAdminTestSuiteTestAdvancedVisibilitySuiteTestAdvancedVisibilitySuiteLegacyTestCallbacksSuiteCHASMTestChasmTestSuiteLegacyTestChildWorkflowSuiteTestClientDataConverterTestSuiteTestClientMiscTestSuiteTestCronTestClientSuiteTestCronTestSuiteTestDLQSuiteTestDeploymentVersionSuiteV0TestDeploymentVersionSuiteV2TestFairnessAutoEnableSuiteTestFairnessSuiteTestFuncClustersTestSuiteTestGetHistoryFunctionalSuiteTestNamespaceInterceptorTestSuiteTestNamespaceSuiteTestNexusApiTestSuiteWithTemporalFailuresTestNexusStateReplicationTestSuiteTestNexusWorkflowTestSuiteTestPollerScalingFunctionalSuiteTestPrioritySuiteTestPurgeDLQTasksSuiteTestQueryWorkflowSuiteTestRawHistoryClientSuiteTestRelayTaskTestSuiteTestResetWorkflowTestSuiteTestScheduleFunctionalSuiteTestStandaloneActivityTestSuiteTestTLSFunctionalSuiteTestTaskQueueStats_Classic_SuiteTestTaskQueueStats_Pri_SuiteTestTaskQueueSuiteTestTransientTaskSuiteTestUpdateWorkflowSdkSuiteTestUserMetadataSuiteTestUserTimersTestSuiteTestVersionWorkflowSuiteTestVersioning3FunctionalSuiteV0TestVersioning3FunctionalSuiteV2TestVersioningFunctionalSuiteTestWorkerDeploymentSuiteV0TestWorkerDeploymentSuiteV2TestWorkflowMemoTestSuiteTestWorkflowTaskTestSuiteTestWorkflowUpdateSuite