[Bug]: Harmless warning: The referenced component 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter' could not be found

[Evangelink]

Issue Description

Following @YuliiaKovalova advice by creating an issue here for a problem reported on MSTest: microsoft/testfx#2173

Context
On MSTest, we have a dll Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll that is copied to user output (to be picked up by vstest.console.exe) that we don't add as reference to ensure the user doesn't access public types declared in that dll.

With MSTest 3.2.0+, we introduced a new standalone runner and for this runner to work, we do need the dll to be referenced because the generated file we produce uses some types from this dll. We add this dll through a conditional Reference (see https://github.com/microsoft/testfx/blob/main/src/Adapter/Build/Common/MSTest.TestAdapter.targets#L34). There is a problem of declaration (that should be fixed by microsoft/testfx#2220) but even fixing it I can still reproduce the issue.

Related issue
I found the following issue that's similar microsoft/appcenter-sdk-dotnet#543 but their resolution was to remove the reference from the targets. I am wondering if I have any other option.

Steps to Reproduce

See microsoft/testfx#2173

Expected Behavior

Dll should not be shown or shown without warning and no warning should be shown in the error list.

Actual Behavior

An error appears in the error list.
The reference always appear and is annotated with a warning.

Analysis

Looking at the issue with Rainer it seems to be originating from csprojui.dll.

There is no warning emited in CLI nor in binlogs (CLI or VS one). I have also looked at the binlog produced using the specific debug environment variables but cannot find the issue either.

It would seems to be affecting only VS that seems to be ignoring the condition and seems to not be able to resolve the dll despite its existence. I have tried putting the condition on some ItemGroup but it's not working either.

Versions & Configurations

No response

[YuliiaKovalova]

I have done some investigation from my side - it looks like a bug/feature in project system (csproj).
if you flip condition , the reference is still displayed, but without warning. In both cases this reference persists.
Plan to dig more this month.

[YuliiaKovalova]

This may be an expected behavior in VS due to the flag presence, need to doublecheck.

msbuild/src/Build/Definition/ProjectLoadSettings.cs

Lines 52 to 56 in 299e051

	/// <summary>
	/// By default, evaluations performed via <see cref="Project"/> evaluate and collect elements whose conditions were false (e.g. <see cref="Project.ItemsIgnoringCondition"/>).
	/// This flag turns off this behaviour. <see cref="Project"/> members that collect such elements will throw when accessed.
	/// </summary>
	DoNotEvaluateElementsWithFalseCondition = 32,

[Evangelink]

Even if that's ignoring the condition, why is it considering that the dll doesn't exist when it actually does?

[Mek7]

Just found this issue in github. Yes, we are struggling with the warning as well for the last maybe 2 weeks. The reference has a yellow exclamation mark. When I remove it manually, the warning goes away, but not forever. After project is unloaded, reloaded, and rebuild, the reference comes back. What's strange is that there is no change in csproj file whatsoever after I remove the reference manually. I guess there is nothing more to do here from my side than wait for a fix.

[YuliiaKovalova]

Hi @Evangelink and @Mek7,

I have some details for you regarding the condition interpretation in the Project System within VS, particularly the legacy version, which poses compatibility challenges. To address this issue, we can propose a workaround involving the use of , like so:

<Project>
....
    <Choose>
      <When Condition=" '$(EnableMSTestRunner)' == 'true' ">
        <ItemGroup>
          <Reference Include="Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter" HintPath="$(MSBuildThisFileDirectory)Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll" />
        </ItemGroup>
      </When>
    </Choose>
...
</Project>

However, this solution does not account for situations where you open a clean project, perform a restore, and expect the reference to be immediately visible in the tree.

Unfortunately, you need to reopen VS or reload the project to trigger additional evaluation in order to see it, as depicted in the images below:

In theory, we could initiate the evaluation round after each restore. However, since the described case is rare and we can provide a workaround, we prefer to avoid the additional performance hit. Naturally, if the same problem is reported by a larger number of customers, we will reconsider this decision.

[Mek7]

@YuliiaKovalova thanks for your proposal, unfortunately it made no difference, the yellow triangle and build warning was still present. I tried multiple variants of the code to no avail. Our nuget packages are stored in Dependencies folders so I amended the path, but that did not help either - yes, the file exists on disk. I also tried the HintPath in a separate element, as other references use this syntax - no dice.

  <Choose>
    <!-- Avoid false warning about missing reference (msbuild bug) -->
    <!-- https://github.com/dotnet/msbuild/issues/9698#issuecomment-1945763467 -->
    <When Condition=" '$(EnableMSTestRunner)' == 'true' ">
      <ItemGroup>
        <Reference Include="Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter">
          <HintPath>..\..\Dependencies\MSTest.TestAdapter.3.2.1\build\net462\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll</HintPath>
        </Reference>
      </ItemGroup>
    </When>
  </Choose>

Reloading the project and restarting VS did not help.

[YuliiaKovalova]

@YuliiaKovalova thanks for your proposal, unfortunately it made no difference, the yellow triangle and build warning was still present. I tried multiple variants of the code to no avail. Our nuget packages are stored in Dependencies folders so I amended the path, but that did not help either - yes, the file exists on disk. I also tried the HintPath in a separate element, as other references use this syntax - no dice.

  <Choose>
    <!-- Avoid false warning about missing reference (msbuild bug) -->
    <!-- https://github.com/dotnet/msbuild/issues/9698#issuecomment-1945763467 -->
    <When Condition=" '$(EnableMSTestRunner)' == 'true' ">
      <ItemGroup>
        <Reference Include="Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter">
          <HintPath>..\..\Dependencies\MSTest.TestAdapter.3.2.1\build\net462\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll</HintPath>
        </Reference>
      </ItemGroup>
    </When>
  </Choose>

Reloading the project and restarting VS did not help.

In order to see the changes, you have to remove bin/obj folder and rebuild your solution.
Also, you can provide a problematic condition and I will try to help for your case specifically.

[Mek7]

OK, I cleaned solution, removed bin and obj folders from disk in the problematic project. Then I built solution again, and the warning was still there.
Forgot to mention that the project still uses packages.config for nuget packages instead of PackageReference, so there is no package restore. The ..\..\Dependencies\MSTest.TestAdapter.3.2.1\build\net462\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll file is always on disk, and is also in source control as part of the Dependencies folder. Could this be part of the problem?

[YuliiaKovalova]

OK, I cleaned solution, removed bin and obj folders from disk in the problematic project. Then I built solution again, and the warning was still there. Forgot to mention that the project still uses packages.config for nuget packages instead of PackageReference, so there is no package restore. The ..\..\Dependencies\MSTest.TestAdapter.3.2.1\build\net462\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll file is always on disk, and is also in source control as part of the Dependencies folder. Could this be part of the problem?

Ah, I understand now. Without releasing a new package with the changes implemented by the MSTest team, you won't be able to see the results of the suggested change.
@Evangelink, could you please let us know if you are considering publishing the suggested workaround?

[Evangelink]

Let me test out this solution. If that's working nice locally, I will publish a preview package @Mek7 would you be ok to test it? I would then release 3.2.2 with this bug fix if that's fixing it all.

Thanks for the investigation @YuliiaKovalova!!

[Mek7]

Sure, I will test what's needed if possible :) Thanks for your fast reactions, people :)

[Evangelink]

I confirm this is working well locally.

With EnableMSTestRunner enabled:

Without

[Evangelink]

@Mek7 here is the link to the preview adapter https://dnceng.visualstudio.com/public/_artifacts/feed/test-tools/NuGet/MSTest.TestAdapter/overview/3.2.2-preview.24116.10. You should not need it but here is the framework in case you need it too https://dnceng.visualstudio.com/public/_artifacts/feed/test-tools/NuGet/MSTest.TestFramework/overview/3.2.2-preview.24116.10

[Mek7]

@Evangelink Thanks, it looks like the 3.2.2-preview adapter makes the warning go away :)
FYI, the project was migrated to PackageReference in the meantime.
What I did was, downlaod the nuget package, open it in 7-zip, enter folder build/net462 and extracted all files under C:\Users<my username>.nuget\packages\mstest.testadapter\3.2.1\build\net462
I'm not sure that is the correct way to test it, but it worked, it seems :) Can you please publish an official version of this nuget package?

[Evangelink]

Can you please publish an official version of this nuget package?

Yes that's planned for this week :) I am waiting for a few more fixes to be done.

What I did was, downlaod the nuget package, open it in 7-zip, enter folder build/net462 and extracted all files under C:\Users.nuget\packages\mstest.testadapter\3.2.1\build\net462

Weird this should have worked.

[Evangelink]

@YuliiaKovalova I don't know if you want to keep the ticket open here for some future bug fix. If not, feel free to close it, I have the working workaround for MSTest so I am good :)

@Mek7 please follow release on MSTest repo.

[YuliiaKovalova]

@Evangelink , we have had internal discussion on this bug from ProjectSystem perspective and agreed to close it "by Design", since we have a workaround and applying any aggressive changes might cause unexpected consequences due to compatibility challenges.

Thank you for your time and quick responsiveness!