Semantic analysis does not work properly in .NET 5 iff executed in a test environment

[marodev]

Roslyn's semantic analysis targeting .NET 5 does not work properly when the application is executed as a test.

Version Used:

• .NET 5.0
• Microsoft.CodeAnalysis.CSharp 3.9
• Microsoft.Build.Locator 1.4.1

Steps to Reproduce:

1. Create a new solution containing 2 projects, one with an entry point (Main(...) method) and one test project:

using System;
using System.Linq;
using Microsoft.Build.Locator;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Microsoft.CodeAnalysis.MSBuild;

public class MyApp
{
    public static void Main(string[] args)
    {
        MSBuildLocator.RegisterDefaults();

        var workspace = MSBuildWorkspace.Create();
        var solution = workspace
            .OpenSolutionAsync(@"/<PATH>/SolutionToAnalyse.sln")
            .Result;

        var project = solution.Projects.First();
        var compilation = project.GetCompilationAsync().Result;
        var documents = project.Documents.ToList();
        var document = documents.First(d => d.Name.Equals("Foo.cs"));
        var syntaxTree = document.GetSyntaxTreeAsync().Result;
        var root = syntaxTree!.GetRoot();
        var semanticModel = compilation!.GetSemanticModel(syntaxTree);

        // fetch myList.Any()
        var inVocExSyn = root.DescendantNodes().OfType<InvocationExpressionSyntax>().ToList()[0];
        var iSym = semanticModel.GetSymbolInfo(inVocExSyn).Symbol;
        // prints System.Collections.Generic.IEnumerable<int>.Any<int>()
        Console.WriteLine(iSym);
    }
}

The second project (xUnit) containing a dummy test:

public class UnitTest
{
    [Fact]
    public void Test()
    {
        MyApp.Main(new[] {""});
    }
} 

2. Create a second solution (SolutionToAnalyse.sln) containing one project and a single file Foo.cs:

public class Foo
{
    public void FooMethod()
    {
        List<int> myList = new List<int>();

        if (myList.Any())
        {
            Console.WriteLine("List is not empty");
        }
    }
}

Expected Behavior:

The console output should be:

System.Collections.Generic.IEnumerable<int>.Any<int>()

Actual Behavior:

The output is as expected if invoked with dotnet run. However, if executed as a test (dotnet test), the output is an empty line as iSym is null.

Note: Changing the targeting .NET version to .NET Core 3.1 (for all 3 projects) solves the issue and the output is the same in both cases (dotnet run / dotnet test).

[Youssef1313]

Can you check if the compilation contains any diagnostics?

[marodev]

Hi!

I fetched the diagnostics like this:

var diagnostics = compilation.GetDiagnostics();

and when executed as a unit test dotnet test, the diagnostic array contains errors such as:

/<PATH TO SOLUTION>/Foo.cs(1,7): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)

/<PATH TO SOLUTION>/Foo.cs(7,18): error CS0518: Predefined type 'System.Object' is not defined or imported

However, these errors are not present if executed with dotnet run.

[wdhofmann]

Hello!

Are there any updates to this issue? I am running into the same problem. I also found out, that when looking at the MSBuildWorkspace.Diagnostics after loading my SampleProject.sln, I get a strange error message (concerning Nuget) which does not show up when running via dotnet run:

var solution = msWorkspace.OpenSolutionAsync(solutionPath).Result;
var diagnostics = msWorkspace.Diagnostics;

foreach (var diagnostic in diagnostics)
{
    OutputHelper.WriteLine(diagnostic.Message);
}

MSBuildWorkspace Diagnostic Error Message:

D:\SampleProject\SampleProject.csproj" with message: C:\Program Files\dotnet\sdk\5.0.203\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets: (63, 5): Unexpected error in ProcessFrameworkReferences-Task.
System.MissingMethodException: Method not found: "System.String NuGet.Frameworks.NuGetFramework.get_Platform()".
   at Microsoft.NET.Build.Tasks.ProcessFrameworkReferences.KnownFrameworkReferenceAppliesToTargetFramework(NuGetFramework knownFrameworkReferenceTargetFramework)
   at Microsoft.NET.Build.Tasks.ProcessFrameworkReferences.<ExecuteCore>b__105_1(KnownFrameworkReference kfr)
   at System.Linq.Enumerable.WhereEnumerableIterator`1.ToList()
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Microsoft.NET.Build.Tasks.ProcessFrameworkReferences.ExecuteCore()
   at Microsoft.NET.Build.Tasks.TaskBase.Execute()
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask).

[wdhofmann]

Hi!

I finally found a solution for this issue (running CodeAnalysis in Test Environment in .NET 5):
Reference the Nuget package "NuGet.Frameworks" in your test project.

[JoeRobich]

Hi @marodev ,

As @wdhofmann has already discovered adding a reference to "NuGet.Frameworks" resolves the project loading issues. This is an issue where Microsoft.NET.Test.Sdk depends on Microsoft.TestPlatform.TestHost which depends on Microsoft.TestPlatform.ObjectModel which depends on and older version of NuGet.Frameworks. Since you do not have a direct reference to NuGet.Frameworks, then you inherit this older version of the package. Because more recent .NET 5 SDKs require a higher version, your project does not load.

@nohwnd Are there changes we can make here to improve the experience?

[JoeRobich]

@shyamnamboodiripad The Microsoft.NET.Test.Sdk has a dependency on the NuGet.Frameworks package. This can cause issues when tests are using MSBuildWorkspace with newer versions of the .NET SDK. Is this dependency necessary? The current workaround is for those test projects to directly reference a version of NuGet.Frameworks that is compatible with the SDK being tested against.

[shyamnamboodiripad]

I believe the issue is known and we have worked around it in other places. See this internal work item https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1050903. I don't fully recall why the dependency between MS.TP.OM and Nuget.Frameworks cannot be severed, but @vritant24 @nohwnd @AbhitejJohn may remember.

[vritant24]

My memory is hazy but we ran into this when trying to work with fakes and runsettings.
The runsettings parsing used Nuget.Frameworks to parse the targetframework which is why the dependency exists. @nohwnd and @AbhitejJohn may have more input here but afaik it was done for back compat reasons as there's no single source of truth for tfm parsing

We worked around this issue in LUT (in the bug @shyamnamboodiripad linked) by ensuring that we don't load testplatform.objectmodel into our process by just removing the reference.

[earloc]

Just stumbled upon the exact same behavior in one of my projects.
A repro can be found here:
https://github.com/earloc/RoslynUnitTest

I can confirm that adding NuGet.Frameworks to the affected test-project resolves the issue.

See earloc/RoslynUnitTest#1

[jasonmalinowski]

This is a duplicate of #61454.