VS Test Retry Has Stopped Working On Azure Pipeline

[bendursley]

We've noticed in the last few days that a couple of our test runs no longer retries on failures, and upon retry, it is unable to find the name spaces for those tests.
The tests are written in SpecFlow 3.9.74. We were using an old Test SDK (16.5) but on upgrade to 17.3.2, we still have this issue. We are running on Windows-Latest (2022).

We run using the below YAML code:
- task: VSTest@2 displayName: 'VsTest - testAssemblies' inputs: testAssemblyVer2: | **\JourneyTests.dll !**\*TestAdapter.dll !**\obj\** rerunFailedTests: true rerunFailedThreshold: 35 rerunMaxAttempts: 2 testRunTitle: '${{ parameters.Environment }} - Journey Tests' searchFolder: $(PIPELINE.WORKSPACE)\${{ parameters.PipelineResource }}\${{ parameters.ArtifactResource }} ${{ if ne(parameters.CategoryFilter, '') }}: testFiltercriteria: ${{ parameters.CategoryFilter }} env: ${{ if eq(parameters.UseMocks, true) }}: UseMocks: true ${{ else }}: UseMocks: false

In our logs, you can see it start the retry attempt, then fail as it is unable to match the name space

No test matches the given testcase filter FullyQualifiedName=JourneyTests.Features.HappyPathFeature.New user sign up journey.

This seems to have changed sometime between Tuesday of this week and Wednesday of this week. We run these tests on a schedule, so there was no code change between the runs on Tuesday and Wednesday.

Tue: Microsoft (R) Test Execution Command Line Tool Version 17.3.0-preview-20220626-01 (x64) [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET Core 3.1.29) DTA Execution Host 19.210.32906.4

Wed: Microsoft (R) Test Execution Command Line Tool Version 17.3.0-preview-20220626-01 (x64) [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET Core 3.1.29) DTA Execution Host 19.204.32418.1

Environment

• Azure Pipelines
• Windows 2022

[bendursley]

Looks like this is due to a change on the Azure side, see: microsoft/azure-pipelines-tasks#16990

[MBGoldberg-BD]

We're having same issue with Classic pipeline. Hope this gets resolved soon.

[Evangelink]

Closing this one as the issue exists on azdo side.