Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: Passing tests using "DynamicData" not recognized as passed in test run #20051

Open
4 of 7 tasks
nabriis opened this issue Jun 21, 2024 · 0 comments
Open
4 of 7 tasks

[BUG]: Passing tests using "DynamicData" not recognized as passed in test run #20051

nabriis opened this issue Jun 21, 2024 · 0 comments
Labels
bug triage

Comments

@nabriis
Copy link

nabriis commented Jun 21, 2024 •
edited
Loading

New issue checklist

Task name

VSTest

Task version

2.*

Issue Description

We have a test method associated with a test case. It looks as follows:

[TestMethod]
[DynamicData(nameof(LoadData), DynamicDataSourceType.Method)]
[TestCategory("cat1"), TestCategory("cat2"), TestCategory("cat3"),TestCategory("cat4")]
[Timeout(60 * 60 * 1000)]
public void TestNameRedacted(string name, Dictionary<string, string> dataRow)
             ....

The test is associated (via visual studio) with a test case and it is recognized in Azure DevOps that this association exists.

A test project is compiled from a .yml pipeline and stored as an artifact. This artifact is comsumed from a Release Pipeline which then runs the Visual Studio Test Platform Installer and subsequently runs the VsTest - testRun task. YML for the tasks looks like this:

steps:
- task: VSTest@2
  displayName: 'VsTest - testRun'
  inputs:
    testSelector: testRun

according to Azure DevOps. This is the UI view:
image

It is possible to run the Release Pipeline from the Test Plan by selecting the automated test and running it. This initiates the Release Pipeline.

The correct tests are found and executed and all pass (prints as passed in log). The test result file is corrected attached to the test run.

However the associated test is left marked as "Unspecified" and the test suite continues to be "in progress". The Release Pipeline has finished.
image

I have attached the log file (with some redactions) here.
PassedTestsNotRecognized.txt

This error shows up specifically. Not sure if that is the reason?

2024-06-21T12:45:11.6970622Z SliceFetch Aborted. Moving to the TestHostEnd phase

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 11

Relevant log output

2024-06-21T12:39:35.9411644Z ##[warning]RunMessage : Could not load file or assembly 'file:///C:\agents\_work\r1\a\[REDACTED] or one of its dependencies. The module was expected to contain an assembly manifest.
2024-06-21T12:44:34.1679215Z Passed    [REDACTED][REDACTED]
2024-06-21T12:44:34.1680424Z Passed    [REDACTED][REDACTED]
2024-06-21T12:44:34.1681232Z Passed    [REDACTED][REDACTED]
2024-06-21T12:44:34.1682421Z Passed    [REDACTED][REDACTED]
2024-06-21T12:44:34.1683371Z Passed    [REDACTED][REDACTED]
2024-06-21T12:44:34.1684312Z Passed    [REDACTED][REDACTED]
2024-06-21T12:44:34.2085646Z Adding trx file C:\agents\_work\_temp\TestResults\[REDACTED]_2024-06-21_14_44_34.trx to run attachments
2024-06-21T12:44:34.2244350Z **************** Completed test execution *********************
2024-06-21T12:44:36.9071051Z Result Attachments will be stored in LogStore
2024-06-21T12:44:36.9656367Z Publishing test results to test run '15994'.
2024-06-21T12:44:36.9758557Z TestResults To Publish 1, Test run id:15994
2024-06-21T12:44:36.9906257Z Test results publishing 1, remaining: 0. Test run id: 15994
2024-06-21T12:44:37.1197919Z Published test case results: 1
2024-06-21T12:44:37.1583725Z Result Attachments will be stored in LogStore
2024-06-21T12:44:37.1584533Z Run Attachments will be stored in LogStore
2024-06-21T12:45:11.6968004Z Received the command : Stop
2024-06-21T12:45:11.6968810Z TestExecutionHost.ProcessCommand. Stop Command handled
2024-06-21T12:45:11.6970622Z SliceFetch Aborted. Moving to the TestHostEnd phase
2024-06-21T12:45:11.7723943Z Please use this link to analyze the test run : https://dev.azure.com/[REDACTED]/_TestManagement/Runs#_a=resultQuery&runId=15994&queryPath=Recent+Run%2FRun15994
2024-06-21T12:45:11.7725315Z Test run '15994' is in 'Completed' state with 'Total Tests' : 1 and 'Passed Tests' : 0.

Full task logs with system.debug enabled

2024-06-21T12:39:14.0951682Z ##[section]Starting: VsTest - testRun
2024-06-21T12:39:14.1067079Z ==============================================================================
2024-06-21T12:39:14.1067243Z Task : Visual Studio Test
2024-06-21T12:39:14.1067307Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2024-06-21T12:39:14.1067640Z Version : 2.240.1
2024-06-21T12:39:14.1067695Z Author : Microsoft Corporation
2024-06-21T12:39:14.1067760Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
2024-06-21T12:39:14.1067857Z ==============================================================================
2024-06-21T12:39:14.7578767Z In distributed testing flow
2024-06-21T12:39:14.7579065Z ======================================================
2024-06-21T12:39:14.7579883Z Test selector : Test run
2024-06-21T12:39:14.7580641Z Test run Id : '15994'
2024-06-21T12:39:14.7581049Z Search folder : C:\agents_work\r1\a
2024-06-21T12:39:14.7697192Z Action when minimum tests threshold not met : donothing
2024-06-21T12:39:14.7697430Z Minimum tests expected to be run: 0
2024-06-21T12:39:14.7698325Z VisualStudio version selected for test execution : latest
2024-06-21T12:39:14.7699857Z Attempting to find vstest.console from a visual studio installation with version [17.0,18.0).
2024-06-21T12:39:14.7704984Z Distributed test execution, number of agents in job : 1
2024-06-21T12:39:14.7708159Z Run in parallel : false
2024-06-21T12:39:14.7708626Z Run in isolation : false
2024-06-21T12:39:14.7709437Z Path to custom adapters : undefined
2024-06-21T12:39:14.7712005Z Other console options : undefined
2024-06-21T12:39:14.7714168Z Code coverage enabled : false
2024-06-21T12:39:14.7719225Z Diagnostics enabled : false
2024-06-21T12:39:14.7789564Z ======================================================
2024-06-21T12:39:14.7790687Z Source filter: *test.dll,!*TestAdapter.dll,!*\obj**
2024-06-21T12:39:14.8148438Z [command]C:\agents_work_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.240.1\Modules\DTAExecutionHost.exe --inputFile C:\agents_work_temp\input_44a14160-2fcb-11ef-8ac3-7f6fe1edae83.json
2024-06-21T12:39:14.8577293Z ##########################################################################
2024-06-21T12:39:14.8579808Z DtaExecutionHost version 19.240.34919.1.
2024-06-21T12:39:15.9585210Z ===========================================
2024-06-21T12:39:15.9586640Z AgentName: [REDACTED]
2024-06-21T12:39:15.9587342Z ServiceUrl: https://dev.azure.com/[REDACTED]
2024-06-21T12:39:15.9588480Z TestPlatformVersion: 17.8.0-release-23468-02
2024-06-21T12:39:15.9589230Z EnvironmentUri: vstest://env/Application/_apis/release/1/15/15/1
2024-06-21T12:39:15.9589982Z QueryForTaskIntervalInMilliseconds: 3000
2024-06-21T12:39:15.9590678Z MaxQueryForTaskIntervalInMilliseconds: 10000
2024-06-21T12:39:15.9591383Z QueueNotFoundDelayTimeInMilliseconds: 3000
2024-06-21T12:39:15.9592101Z MaxQueueNotFoundDelayTimeInMilliseconds: 50000
2024-06-21T12:39:15.9592786Z ===========================================
2024-06-21T12:39:16.5922599Z Result Attachments will be stored in LogStore
2024-06-21T12:39:16.6169572Z Run Attachments will be stored in LogStore
2024-06-21T12:39:16.7318795Z TestExecutionHost.Execute: Registered TestAgent : 12 : [REDACTED]
2024-06-21T12:39:16.7773915Z Updated Run Settings:
2024-06-21T12:39:16.7787107Z
2024-06-21T12:39:16.7787433Z
2024-06-21T12:39:16.7787584Z 1000
2024-06-21T12:39:16.7787745Z C:\agents_work_temp\TestResults
2024-06-21T12:39:16.7787917Z
2024-06-21T12:39:16.7788036Z
2024-06-21T12:39:16.7858446Z IsValidServiceResponse: Received None command..Service Workflow is not active
2024-06-21T12:39:16.8357811Z Creating run for selected test assemblies with following parameters
2024-06-21T12:39:16.8362619Z SourceFilter: *.dll TestCaseFilter:
2024-06-21T12:39:16.8364196Z Run title: TestRun_Release - AutoTest_Release-AutoTest-14
2024-06-21T12:39:16.8364548Z Build location: C:\agents_work\r1\a
2024-06-21T12:39:16.8364828Z Build Id: 8618
2024-06-21T12:39:17.1662822Z Test run with Id 15994 associated
2024-06-21T12:39:27.8515184Z Received the command : Start
2024-06-21T12:39:27.8543063Z TestExecutionHost.ProcessCommand. Start Command handled
2024-06-21T12:39:27.9628642Z Slice with id = 12, of type = 'Execution' received.
2024-06-21T12:39:28.1063576Z Count of test sources found: 1
2024-06-21T12:39:28.1295330Z =================================================================
2024-06-21T12:39:28.1296059Z Discovering tests from sources
2024-06-21T12:39:28.1340120Z Using new test platform for test execution
2024-06-21T12:39:28.1409227Z Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created
2024-06-21T12:39:30.9659979Z DiscoveryMessage : NUnit Adapter 4.5.0.0: Test discovery starting
2024-06-21T12:39:31.5776891Z DiscoveryMessage : NUnit Adapter 4.5.0.0: Test discovery complete
2024-06-21T12:39:31.7203903Z Number of testcases discovered : 101
2024-06-21T12:39:31.7460524Z Discovered tests 101 from sources
2024-06-21T12:39:31.7460960Z =================================================================
2024-06-21T12:39:31.8050487Z [RunStatistics]This execution slice with id '12', received '1' testcases to execute out of which '1' is discovered.
2024-06-21T12:39:31.8098679Z **************** Starting test execution *********************
2024-06-21T12:39:31.8168066Z Using new test platform for test execution
2024-06-21T12:39:31.8168424Z Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created
2024-06-21T12:39:35.9411644Z ##[warning]RunMessage : Could not load file or assembly 'file:///C:\agents_work\r1\a[REDACTED] or one of its dependencies. The module was expected to contain an assembly manifest.
2024-06-21T12:44:34.1679215Z Passed [REDACTED][REDACTED]
2024-06-21T12:44:34.1680424Z Passed [REDACTED][REDACTED]
2024-06-21T12:44:34.1681232Z Passed [REDACTED][REDACTED]
2024-06-21T12:44:34.1682421Z Passed [REDACTED][REDACTED]
2024-06-21T12:44:34.1683371Z Passed [REDACTED][REDACTED]
2024-06-21T12:44:34.1684312Z Passed [REDACTED][REDACTED]
2024-06-21T12:44:34.2085646Z Adding trx file C:\agents_work_temp\TestResults[REDACTED]_2024-06-21_14_44_34.trx to run attachments
2024-06-21T12:44:34.2244350Z **************** Completed test execution *********************
2024-06-21T12:44:36.9071051Z Result Attachments will be stored in LogStore
2024-06-21T12:44:36.9656367Z Publishing test results to test run '15994'.
2024-06-21T12:44:36.9758557Z TestResults To Publish 1, Test run id:15994
2024-06-21T12:44:36.9906257Z Test results publishing 1, remaining: 0. Test run id: 15994
2024-06-21T12:44:37.1197919Z Published test case results: 1
2024-06-21T12:44:37.1583725Z Result Attachments will be stored in LogStore
2024-06-21T12:44:37.1584533Z Run Attachments will be stored in LogStore
2024-06-21T12:45:11.6968004Z Received the command : Stop
2024-06-21T12:45:11.6968810Z TestExecutionHost.ProcessCommand. Stop Command handled
2024-06-21T12:45:11.6970622Z SliceFetch Aborted. Moving to the TestHostEnd phase
2024-06-21T12:45:11.7723943Z Please use this link to analyze the test run : https://dev.azure.com/[REDACTED]/_TestManagement/Runs#_a=resultQuery&runId=15994&queryPath=Recent+Run%2FRun15994
2024-06-21T12:45:11.7725315Z Test run '15994' is in 'Completed' state with 'Total Tests' : 1 and 'Passed Tests' : 0.
2024-06-21T12:45:11.9801376Z ##########################################################################
2024-06-21T12:45:12.0467822Z ##[section]Finishing: VsTest - testRun

Repro steps

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nabriis @v-schhabra