MSTestAdapter failed to discover tests because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' does not have an implementation

[Shanayara]

Description

We've long had a running VSTest pipeline for our projects unit tests, which target Framework 4.8 and run MSTest.

The YAML of the unit test on DevOps is very minimal:

- name: DebugBuildFolder
  default: '$(DebugBuildFolder)'
- name: DebugBuildArtifactName
  default: '$(DebugBuildArtifactName)'

steps:
  - task: DownloadPipelineArtifact@2
    displayName: 'Download ${{ parameters.DebugBuildArtifactName }} artifact'
    inputs:
      buildType: 'current'
      artifactName: ${{ parameters.DebugBuildArtifactName }}
      targetPath: '$(Build.SourcesDirectory)'
  - task: VSTest@2
    displayName: 'UnitTests on debug build'
    inputs:
      testAssemblyVer2: '${{ parameters.DebugBuildFolder }}\FrameworkUnitTests.dll'

We recently added an empty MSTest project to our solution, which targets Core 6 instead:

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>

    <IsPackable>false</IsPackable>

    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.8" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.8" />
    <PackageReference Include="coverlet.collector" Version="3.1.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

</Project>

With that project simply present, an error message appears and not a single unit test is discovered. Despite that new MSTest project not being referenced by any other project, and despite VSTest correctly only being set on the same FrameworkUnitTests.dll as before. For every test class xxxTests, the log contains a message of the form:

[MSTest][Discovery][D:\a\1\s\Build\Debug\UnitTests.dll] MSTestAdapter failed to discover tests in class FrameworkUnitTests.xxxTests' of assembly 'D:\a\1\s\Build\Debug\FrameworkUnitTests.dll' because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' from assembly 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' does not have an implementation..

and at the end there is a message:

No test is available in D:\a\1\s\Build\Debug\FrameworkUnitTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

The problem does not arise in Visual Studio, I guess because it doesn't need the TestAdapter but runs the tests directly. Notably, no project in the solution referenced the new MSTest project or it's single (empty, default) unit test. As soon as we remove the new Core MSTest project from the solution, all tests get discovered again. Weird!

Now, I was able to discover the potential root of the problem:

The Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll in the build output folder has a different size when the Core MSTest project is present - it's just 59kb, compared to109kb before. I then discovered that because of the Core 6 project, the NuGet reference to MSTest.TestAdapter - which both shows as version 2.2.8 - will actually resolve to different dlls than before. Namely, it uses the dll from mstest.testadapter\2.2.8\build\netstandard1.5 instead of mstest.testadapter\2.2.8\build_common, which targets Framework 4.5

The class TestContextImplementation in Framework 4.5 contains a public DataRow attribute, while the TestContextImplementation in Standard 1.5 does not. The moment the Core project is added to the solution, MSBuild will copy the Standard 1.5 dll into the output directory. But the Framework 4.5 UnitTest.dll was linked against the Framework 4.5 version, where the DataRow attribute IS present. When VSTest loads that library it discovers that the implementation for DataRow is missing and quits. (I don't know why Standard 1.5 doesn't need this implementation, since it also should have to implement https://docs.microsoft.com/de-de/dotnet/api/microsoft.visualstudio.testtools.unittesting.testcontext?view=visualstudiosdk-2022 - but it apparently doesn't?)

I don't currently know how to solve this problem - there seems to be no way to force the 4.8 project to also use the Standard 1.5 version for the MSTest package, thereby linking against the same library that the other project uses, see also:
dotnet/sdk#1791
NuGet/Home#7416

I don't know if it is actually well-defined which Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll gets copied to the output folder of a build, is there a way to control this? After all there is a factual conflict between the two libraries - but maybe this isn't discovered since both officially register as MSTest 2.2.8. We consistently see the Standard 1.5 library copied to the output on our Azure DevOps pipeline, so the problem means no tests are discovered for our Framework VSTest above.

One workaround I see right now would be to put the unit tests for Core and Framework into different solutions. It would make sense for us to use the same solution however, since we have projects targetting Framework, newer ones targetting Core and some common projects referenced by both the Framework and Core projects targetting Standard 2.0 for compatibility. In any case, MSTest surely shouldn't just break when both Framework and Core projects use it in a solution.

Another workaround might be be to remove the TestAdapter libraries from the build output folder and run VSTest with a /TestAdapterPath parameter pointing to the correct MSTest version, depending on which project gets tested. But that's rather hacky, especially since I wouldn't know of a nice way to always copy one version of the NuGet package to, say, a folder "FrameworkTestAdapter" and the other to "CoreTestAdapter" so I can reference these in different VSTest tasks.

But maybe there is a way to fix this problem "properly"? For instance, I don't see why it's necessary that the TestContextImplementation exposes public members that simply aren't present in the

Steps to reproduce

Create and build two MSTest projects in the same solution, one targetting Core, the other targetting Framework 4.8

Expected behavior

The Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll that gets copied to the build folder works for running both MSTest projects

Actual behavior

The above dll targeting netstandard1.5 is copied to the build folder. For the Framework MSTest project, running the unit tests from the dll via vstest.console will produce an error:
[MSTest][Discovery][D:\a\1\s\Build\Debug\UnitTests.dll] MSTestAdapter failed to discover tests in class FrameworkUnitTests.xxxTests' of assembly 'D:\a\1\s\Build\Debug\FrameworkUnitTests.dll' because Method 'get_DataRow' in type 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestContextImplementation' from assembly 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' does not have an implementation..

Environment

• Operating system: Windows 11
• Build version of vstest.console: 17.0.0
• Package version of MSTest framework and adapter: 2.2.8
• Other installed packages and their versions on the test project: see above

[AF250329]

I hit this issue too... This is strange that Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll compiled as [assembly: TargetFramework(".NETPortable,Version=v4.5,Profile=Profile259", FrameworkDisplayName = ".NET Portable Subset")] but it always prefer (?) .NET Core version of Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll if one of the projects in solution is .net core based as in description above.... Very strange behavior that depend on how the start-up project was compiled (to full .NET Framework or .NET Core version....) I mean there's vstest.console.exe and testhost.net461.exe in the way before it get to adapter....

[feruilob]

I had a netstandard2.0 project and another multi-target project (net472, netcorapp3.1) to test the netstandard2.0 project and I was hitting this issue.
Adding <AppendTargetFrameworkToOutputPath>true</AppendTargetFrameworkToOutputPath> to all projects helped because now the tests were not getting confused between assembly versions now that each assembly had its own unique path.
Hope that helps!

[Evangelink]

@Shanayara @AF250329 @feruilob Are you still facing the issue with 2.2.10? If so, would you be able to provide a reproducer so we can work on a fix?

[wangyuan8421]

@Evangelink Yesterday I debug this issue a little bit. The issue is still there. Here is the step to reproduce it.

• Windows 11, Visual Studio 2022, create a MSTEST project in Net 6.0
• It will install ["MSTest.TestAdapter", "MSTest.TestFramework"] version 2.2.8 by default
• The unit test will run and test without problem

• Then, I upgrade the ["MSTest.TestAdapter", "MSTest.TestFramework"] >= 2.2.10
• The Visual Studio test explorer will not find any unit test from dll

• Test explorer warning

[Evangelink]

@wangyuan8421 Can you see missing method get_DataRow in the logs? If not, I think this might be a different problem for which I would recommend to open a separate issue.

[wangyuan8421]

Ah, @Evangelink, I checked the log and it missing get_properties. I will create a new issue then

[vinayakmsft]

Hi @wangyuan8421 / @Evangelink is this issue resolved?

[HarmanveerKaur]

Hi All, I am facing an issue in a release pipeline and this issue looks similar to what i am facing.

VStest task in the release pipeline is not able to find a dll although it is present in the build artefact.
Error says- The test source file "D:..........bin\x64\Release\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll " provided was not found.

MSTest Adapter version - 2.1.2
.Net Framework version - 4.7.2

@wangyuan8421 , @Evangelink is this issue still open?

[wangyuan8421]

@vinayakmsft @HarmanveerKaur this issue is not solve yet. but I have a work around.
what I found is another issue. FYI: #1256

[HarmanveerKaur]

@wangyuan8421 are you talking about this workaround?

[Evangelink]

Can you send us a zip or git repo reproducing the issue so we can test if current main of mstest fixed the issue or if we need to work on a fix?

[vinayakmsft]

@Evangelink Is there a DRI- handle name for issues related to Mstest test adpater/test platform?

[HarmanveerKaur]

@Evangelink I have sent the details on your microsoft email. Kindly take a look.

[Evangelink]

@Evangelink Is there a DRI- handle name for issues related to Mstest test adpater/test platform?

Hi @vinayakmsft. What do you mean by DRI handle name?

[vinayakmsft]

Hi @Evangelink its basically the ICM owner team name. So , can you please let me know what your ICM team name?

[Evangelink]

@vinayakmsft @HarmanveerKaur Sorry for the ping but I don't recall if we have sorted out this issue offline or is there still something to do?

[Evangelink]

I will move forward by closing this issue. Please let me know if there was something else to be done.

[bmacombe]

If it helps some future person running across a similar issue, I resolved it by forcing the MSTestAdapter reference in my project.

<Reference Include="Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter">	 
   <HintPath>$(PkgO2S_MSTest_TestAdapter\build\net462\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll</HintPath> 
   <Private>True</Private>
</Reference>

[eXpl0it3r]

This is indeed an issue, if you write all the contents to the same output directory.

Using MSBuild's binary log and MSBuild Structured Log Viewer, you can easily see the double writes, that causes this kind of issue.

If your test project for example runs with .NET Framework 4.8, but references a .NET Standard 2.0 library (e.g. a Source Code Generator project), you end up with having the netfx and netstandard version being written both to the same output directory. Now, if the netstandard version is written last, you end up with the netfx test not being able to find the needed methods, as it can only find the netstandard assembly.