Incorrect test result view in Azure DevOps after upgrading MSTest to 2.2.4 and higher

[mabadev]

Description

After I updated MSTest.Framework and MSTest.Adapter to 2.2.4 and higher, the test results in Azure DevOps are displayed incorrectly.
Each test interation is displayed in a new line. Before that, the interations were shown under a test case.

For example: A Test with 2 iterations

Before:
TestCase12

• Iteration1
• Iteration2

Currently

• TestCase12
• TestCase12

This a screenshot for the test results on Azure DevOps
For example: Test Case 1129

---

TRX-File look like different
After the upgrade, the trx file does not contain an <InnerResults> element

Hier is a screenshot fom trx-file

---

I use VS 2019 16.11.6
I'm not sure if this is a bug or a new configuration. This is my test configuration: Configure unit tests by using a .runsettings file

[Legoldos]

Hey, I think this issued I recently posted is related/same #1024

[mabadev]

Hey, I think this issued I recently posted is related/same #1024

I think the main problem is the same.
thanks for that

[nixdagbibts]

Any progress on this topic?
We cannot upgrade from 2.2.3 to 2.2.8 until this is fixed :(

[nohwnd]

This is not a bug, mstest now reports data driven tests as single tests. This was done to bring parity to how xunit and nunit does it.

@Haplois There are few duplicates of this issue. Could you link them all to a single one, and post an update on when the work on AzDO will be finished, please?

Also correct my statement above if I am wrong :)

[TroyWalshProf]

@nohwnd
The ONLY reason I and many other engineers use MSTest is because of the seamless integration with Azure DevOps Test Hub.
If you cannot maintain this integration you are just undermining the project.

Here are my concerns:

1. The team broke this integration a year ago and it has yet to be addressed
2. The interesting (wrong) assertion that this is "not a bug"
3. The trx report format has fundamentally changed and that doesn't seem to justify a major or minor version bump
4. I know this project and the Azure DevOps Visual Studio Test task are separate projects, but this kind of thing has the stink of the bad old days of Microsoft. - AKA Microsoft cannot seem to even work with itself

As far as I can tell here are the related/duplicate issues:

• Upgrading MSTest.TestAdapter from 2.2.3 to 2.2.4 breaks data-driven tests #860
• Regression - Upgrading MSTest.TestAdapter from 2.2.3 to 2.2.6 skips all tests #955
• DataRow execution is not working after 2.2.3 version in VSTest@2 task for Test plan #992
• Data driven tests are incompatible with Azure Test Plans after moving to MSTest.TestAdapter 2.2.4+ #1023
• Incorrect test result view in Azure DevOps after upgrading MSTest to 2.2.4 and higher #1026
• MSTestAdapter failed to discover tests because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' does not have an implementation #1035

[nohwnd]

Duly noted. Believe me, I am not happy about this either. We are looking into conditionally reverting this change until it is fixed in azdo.

[TroyWalshProf]

@nohwnd - If I may make a suggestion:
I would love to see splitting or consolidating tests being optional, much like parallelization.
Maybe using an attribute such as "DataDrivenToSingleTest".
This way users have the option of how they want to group such tests.
*This would follow the same basic pattern as the "DoNotParallelize" attribute.

Example:

        [DataTestMethod]
        [DataDrivenToSingleTest] // Allow users to set this on the test, test class, and in the run settings file like you can for parallel
        [DataRow("ConfigJsonEnvRunOverride", "OVERRIDE")]
        [DataRow("OverrideOnly", "OVERRIDE")]

[Legoldos]

This is not a bug, mstest now reports data driven tests as single tests. This was done to bring parity to how xunit and nunit does it.

@Haplois There are few duplicates of this issue. Could you link them all to a single one, and post an update on when the work on AzDO will be finished, please?

Also correct my statement above if I am wrong :)

Hello @nohwnd
Can you please provide link for some kind of documentation where this new behavior is explained?
I couldnt find anything in MSTEST release notes. I would like something to grasp on when I explain this new behavior to my team.

[mabadev]

Duly noted. Believe me, I am not happy about this either. We are looking into conditionally reverting this change until it is fixed in azdo.

Hoping to resolve this obstacle or to find an alternative solution for it. Our project has suffered because of that.

[ChristoWolf]

Hello @nohwnd Can you please provide link for some kind of documentation where this new behavior is explained? I couldnt find anything in MSTEST release notes. I would like something to grasp on when I explain this new behavior to my team.

I would also like to know, as I did not find the related changes by glancing over 2.2.4's release notes and commits.

[MonzT]

Are there any updates to this as yet? also getting very frustrated and using work arounds is less than ideal

[nohwnd]

@Haplois @Evangelink did you get any update from AzDo about the progress of the fix? I know we talked with them a while ago.

[Haplois]

@nohwnd We didn't get an update - I pinged them.

[Haplois]

They have a fix, and expect to ship it next sprint. I asked about exact date.

[Nidolai]

@Haplois Thanks for the update.

[charlesdwmorrison]

Getting the next sprint date would be great.
As far as I can see, this change significantly breaks all test harnesses which are heavily Data Driven. I.e., test harness which have many rows in Excel, Access, or a Database. Teams following that methodology may have 100 test cases in one spreadsheet, with differ data rows of the spreadsheet testing different features, all going through one test method. While this is not ideal, a lot of teams/organizations perform testing.
If one of these rows fails, the entire test fails? That does really skew the results.

Related well-reasoned posts on this issue include:
https://developercommunity.visualstudio.com/t/vstest-test-publication-miscounts-test-cases-and-c/909375
https://developercommunity.visualstudio.com/t/tests-tab-on-build-results-page-shows-wrong-test-c/1385811

Looking forward to seeing what the fix is. If is necessary to bring MSTest.exe in line with NUnit and XUnit, there should be a command line flag to allow for backward compatibility with the old approach.

[Haplois]

I spoke with the product team again, the fix in release pipeline and will be deployed within this sprint (which will be ending on 12th of September). I couldn't get an exact date.

/cc @dogirala from the product team.

[charlesdwmorrison]

So when will milestone 2.3.0 be pushed out?

[Evangelink]

Hi @charlesdwmorrison! My mistake, I should not have put any milestone on this ticket, as far as we know, sprint ended on 12th of September and roll out is supposed to have happened (@dogirala could you confirm?).

If you have tested and you are still experiencing the issue, please let us know and we will ping Azure DevOps team.

[dogirala]

@Evangelink , @charlesdwmorrison I see the fix has been deployed but somehow it didn't reflect in microsoft which is strange. For microsoft org I still see old PTR task version which is 2.203.0, ideally it should be 2.210.0. I am checking with my team and concerned deployments team. Will let you know as soon as I find out something on this.

Other ex pipeline:

[dogirala]

Got to know that this is expected as the deployments are still not completely done and they are in progress. Eta is by end of next week.

[dogirala]

The fix is deployed now. All deployments are completed

[Evangelink]

You can try through https://developercommunity.visualstudio.com/. For the GitHub repo, @dogirala please let us know.

[dogirala]

hi @Evangelink, @charlesdwmorrison I am on top of this, please expect an update by tomorrow. From what I checked the issue is not with the testplans rather it is the 'Associate to testcase' functionality of visual studio. I don't have much info regarding the owing team and the public ticket that would reach to them, but I am working on this.

[Evangelink]

If that's Test Explorer side, we have some contacts (cc @AbhitejJohn)

[dogirala]

@AbhitejJohn could you please take a look at the bug and clarify who should be the right team that owns that workflow? I was having a discussion with couple of folks from test plans and there seems to be some confusion around it.

[dogirala]

Had checked with @AbhitejJohn and got the details on the owning team. The bug is assigned Automation Testing team and will be prioritized in the coming sprints. Will follow-up with the team regarding public ticket on the same. Will update on it here. Thanks

[Evangelink]

Closing this ticket as there is nothing left on MSTest side.

[charlesdwmorrison]

Could we get a URL to where the Automation Testing team is tracking this or providing public updates?

[Evangelink]

Hey all,

I have open a ticket on Dev Community https://developercommunity.visualstudio.com/t/Incorrect-test-result-view-in-Azure-DevO/10359804?q=Incorrect+test+result+view for you all to track. Please do upvote the ticket.

[dogirala]

Hello Amaury, Thanks for the email. Adapting Tcm with Test Adapter latest versions and incorporate any improvements is one of the priority features for us in this semester. Right now, I am working on going through the older customer tickets we have received regarding the issue customers are facing due to, parity mismatch between the Test Adapter changes and how TCM as a service is consuming the trx file. Please give me 1-2 days to go over the older tickets and the suggestions you have shared, to come up with a short plan to address the problem and improve the overall Ux for the customers. Below are the key things we are targeting - Fix all current issues due to data driven changes in latest Test Adapter versions and how trx is file parsed and consumed. - Check feasibility and Incorporate the suggestions you have shared to improve the overall UI/UX to the customer. - Come up with a means to have this parity for future as well, so that if there are any changes done in Test Adapter, corresponding adaptive changes are done in TCM as well. We would require your support in delivering this, would reach out to the test platform team in 1-2 days to discuss more on this. Appreciate your support. Thanks and regards, Prabhakar

…

________________________________ From: Amaury Levé ***@***.***> Sent: Wednesday, May 10, 2023 7:21 PM To: microsoft/testfx ***@***.***> Cc: Durga Vijay Prabhakar Ogirala ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/testfx] Incorrect test result view in Azure DevOps after upgrading MSTest to 2.2.4 and higher (Issue #1026) Hey all, I have open a ticket on Dev Community https://developercommunity.visualstudio.com/t/Incorrect-test-result-view-in-Azure-DevO/10359804?q=Incorrect+test+result+view for you all to track. Please do upvote the ticket. — Reply to this email directly, view it on GitHub<#1026 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYBLNNDV6XYIQ25TG6RPYOTXFOMPDANCNFSM5KN64GTQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dogirala]

@charlesdwmorrison Just sharing an update on the issue you mentioned above. We have merged a fix for that issue. It required a two-part fix, one on the visual studio application and the other on the azure devops. The fix on the visual studio would be available from VS 17.8 Preview 3 version. However, the fix would be applicable for all new associations, so the associations need to be done from visual studio.

[charlesdwmorrison]

Thanks. We are familiar with the association of test case process. Not our favorite thing, but we will check it out. Is this the download point for a VS Preview that contains the fix: https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-notes-preview (I see that this says VS 17.8)?

[dogirala]

Hi yes, this is the page. Right now it says the newest is 17.8 preview 2, the next version should have the fix. Also, just to add, the change is under a featureflag TestExecution.ModifiedFqnAssociation, for hosted ado, I am in the process of enabling this shortly. For onprem, please register and enable this FF in config db. Please reach out to us via CSS for any further assistance on the same

[b3go]

Hi yes, this is the page. Right now it says the newest is 17.8 preview 2, the next version should have the fix. Also, just to add, the change is under a featureflag TestExecution.ModifiedFqnAssociation, for hosted ado, I am in the process of enabling this shortly. For onprem, please register and enable this FF in config db. Please reach out to us via CSS for any further assistance on the same

@dogirala using mstest adapter and framework 3.2.0, azure devops server 2022 and self hosted agents with VS 2022 17.8.6. I added this row in tbl_RegistryItems in the tfs_configuration db and restarted the IIS:

PartitionId	ParentPath	ChildItem	RegValue
1	#\FeatureAvailability\Entries\TestExecution.ModifiedFqnAssociation\	AvailabilityState\	1

Running tests with a parallel strategy doesn't work. Only the first DataRow is executed, the others are not mentioned (neither .trx, nor Test Result Window:

However running on one agent works, as all DataRows are properly executed and also displayed in Test Resuts:

Unfortunately running on one agent is not possible in this case. These are "unstable" tests that have to rerun if failed (some static cache issues in big legacy app). Which results in this error microsoft/azure-pipelines-tasks#16187 if executed without batch/slicing

Maybe I also missunderstood for what the fix is for :D

[dogirala]

hi @b3go, yes, we've identified this bug and we've fixed this now. Can you share what is the version of Vstest task you are using?. And can you update the Visual studio test task version and give this a try and let me know if you are facing this issue?

[b3go]

Hi @dogirala, had some troubles building the task but I was finally able to get it working. I am now using VSTestV2 2.229.0 but still getting the same issue. With the parallel strategy the slicing of the test cases with DynamicData/DataRow doesn't seem to work. I still get only the first case.

I've found this in the documentation https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines#i-am-running-into-issues-when-running-data-driven-xunit-and-nunit-tests-with-some-of-the-task-options-are-there-known-limitations
As stated in vstest@2 it is not supported to distribute the tests on multiple different agents in xUnit and NUnit. Could it be that this is now also the same with MsTest because since 2.2.4 the test names are generated in the same way as in xUnit and NUnit?

[dogirala]

@b3go apologies for the delay, I see you are using 2.229.0, from what I checked the change we did went into 2.234.0, can you try with any version vstestv2 2.234.0 or more, the distributed scenario should work as expected, please let me know if you are still seeing this issue, I'll try to connect with you to debug it further

[b3go]

@dogirala thanks a lot for your support. I updated the task to 2.235.2 but no luck, only the first case is found and executed:

[dogirala]

@b3go can you help me with couple of queries?

• Are these mstest unittests that you are running?
• What is the exact scenario in vstest you are running? Is it testassemblies + parallel strategy?
• For DynamicData/DataRow, are you overriding the displayName method and setting a customer displayName? --> If yes, can you try generating it something like below?

```
public string GetDisplayName(MethodInfo methodInfo, object[] data)
  {
      if (data != null)
      {
          var testParamString = new StringBuilder();
          testParamString.Append(methodInfo.Name);
          testParamString.Append(' (');
          var methodParameters = methodInfo.GetParameters();
          for (int i = 0; i < methodParameters.Length; i++)
          {
              testParamString.Append(data[i]);
              testParamString.Append(',');
          }
          testParamString.Length--;
          testParamString.Append(')');

          return testParamString.ToString();
      }
      return methodInfo.Name;
  }

I am trying to understand the exact scenario you are running and the issue you are facing

[b3go]

Hi @dogirala thank you very much for your continued support :)

• Are these mstest unittests that you are running?

Yes we currently use MSTest.TestAdapter and TestFramework v3.2.0

• What is the exact scenario in vstest you are running? Is it testassemblies + parallel strategy?

This is the relevant pipeline yaml:

- job: run_tests
  strategy:
    parallel: 5
  workspace:
    clean: all
  steps:
    - checkout: none
    - task: DownloadBuildArtifacts@0
      inputs:
        buildType: 'current'
        downloadType: 'single'
        artifactName: 'testdrop'
        itemPattern: '**'
        downloadPath: '$(Build.BinariesDirectory)'
    - task: VSTest@2
      inputs:
        testSelector: 'testAssemblies'
        testAssemblyVer2: 'My.*Test.dll'
        searchFolder: '$(Build.BinariesDirectory)\testdrop'
        testFiltercriteria: 'TestCategory!=Dummy'
        runInParallel: true
        codeCoverageEnabled: false
        distributionBatchType: 'basedOnExecutionTime'
        batchingBasedOnExecutionTimeOption: 'customTimeBatchSize'
        customRunTimePerBatchValue: '480'
        rerunFailedTests: true
        rerunMaxAttempts: '5'

• For DynamicData/DataRow, are you overriding the displayName method and setting a customer displayName? --> If yes, can you try generating it something like below?

This is the method signature of that particular test I am testing the behavior with:

[DataTestMethod]
[DataRow(1, BuchungsVorgangArt.Bankbuchung, BewegungsGruppeType.EG, FremdSchadenAbrechnungArtType.Eingang)]
[DataRow(2, BuchungsVorgangArt.Bankbuchung, BewegungsGruppeType.EG, FremdSchadenAbrechnungArtType.Eingang)]
[DataRow(1, BuchungsVorgangArt.Bankbuchung, BewegungsGruppeType.AG, FremdSchadenAbrechnungArtType.Ausgang)]
[DataRow(2, BuchungsVorgangArt.Bankbuchung, BewegungsGruppeType.AG, FremdSchadenAbrechnungArtType.Ausgang)]
[DataRow(3, BuchungsVorgangArt.Umbuchung, default, FremdSchadenAbrechnungArtType.Umbuchung)]
[DataRow(4, BuchungsVorgangArt.Umbuchung, default, FremdSchadenAbrechnungArtType.Umbuchung)]
[DataRow(5, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(6, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(7, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(8, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(9, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
[DataRow(10, BuchungsVorgangArt.Ausbuchung, default, FremdSchadenAbrechnungArtType.Kosten)]
public void Test_if_CreateFremdschadenAbrechnungStrategy_CreateFremdschadenAbrechnung_works(
    long kontoArtBewegungId,
    BuchungsVorgangArt
    buchungsVorgangArt,
    BewegungsGruppeType bewegungsGruppe,
    FremdSchadenAbrechnungArtType expectedFremdSchadenAbrechnungArt)
{
}

I hope this information helps!

Edit: actually I created a demo repo which results in exactly the same error:
https://github.com/b3go/DataRowTests

[dogirala]

@b3go thanks for sharing the details, will check and get back shortly.

[b3go]

@dogirala Hi do you have any updates on this matter?

[dogirala]

Hi @b3go, apologies for the delay, I was out of office, the past couple of weeks, I am back to office this week, I will look into this and create a tracking item for this.

[catalin7mar]

@dogirala Hi! Do you have any updates? At my new company, I discovered that the following issue is still present, which is why we are still using TestAdapter 2.2.3.

Here's the behavior we're observing (Agent Version 2.195.2, TestAdapter 3.4.3):

We run our permission tests on a remote desktop using various test plans from Azure DevOps (Net 6).

When running the tests on my local machine, the result of failed test cases is correct:

However, when I create a release in Azure, the output is not correctly captured:

How it should look (TestAdapter 2.2.3)

[dogirala]

Hi @catalin7mar, are you using ADO onprem server or hosted?, can you share the Organization Name if you are on hosted?

[catalin7mar]

@dogirala Hi! onprem

[dogirala]

@catalin7mar can you please register and turn on this FF --> TestExecution.EnableParentChildForEmptyDataDrivenDynamicTests

For registration, you can follow the below steps -

Execute the below on configuration DB, replace the {featureflagname} with the above highlighted

INSERT INTO tbl_RegistryItems (PartitionId, ParentPath, ChildItem, RegValue)

VALUES
(1,'#\FeatureAvailability\Definitions{FeatureFlagName}', 'Description', 'Flag for controlling enabling of association to testcase workflows with modified Fully Qualified Name'),
(1,'#\FeatureAvailability\Definitions{FeatureFlagName}', 'Name', '{FeatureFlagName}'),
(1,'#\FeatureAvailability\Definitions{FeatureFlagName}', 'Owner', '{Service}')

EXEC prc_SetRegistryValue 1, '#\FeatureAvailability\Entries{FeatureFlagName}\AvailabilityState', 1

Restart the IIS Web App for the Azure DevOps collection.

[b3go]

Hi @dogirala me again :D

how does TestExecution.EnableParentChildForEmptyDataDrivenDynamicTests add to TestExecution.ModifiedFqnAssociation? (btw I think you missed a \ between Defintions/Entries and {FeatureFlagName})

Could be that I actually made a mistake enabling the Feature Flag, I only added the row manually

PartitionId	ParentPath	ChildItem	RegValue
1	#\FeatureAvailability\Entries\TestExecution.ModifiedFqnAssociation\	AvailabilityState\	1

So what I would do now is:

INSERT INTO tbl_RegistryItems (PartitionId, ParentPath, ChildItem, RegValue)

VALUES
(1,'#\FeatureAvailability\Definitions\TestExecution.ModifiedFqnAssociation', 'Description', 'Flag for controlling enabling of association to testcase workflows with modified Fully Qualified Name'),
(1,'#\FeatureAvailability\Definitions\TestExecution.ModifiedFqnAssociation', 'Name', 'TestExecution.ModifiedFqnAssociation'),
(1,'#\FeatureAvailability\Definitions\TestExecution.ModifiedFqnAssociation', 'Owner', '{Service}')

EXEC prc_SetRegistryValue 1, '#\FeatureAvailability\Entries\TestExecution.ModifiedFqnAssociation\AvailabilityState', 1

What would be the value for {service}?

Thanks!