Can MTP continue support the Msbuild property RunSettingsFilePath of runsettings file?

[moh-hassan]

The Msbuild property RunSettingsFilePath was removed support in MTP based on this PR
Currently, the property is still used and supported by VS 2022 Test Explorer and VsCode in both VSTest and MTP modes.

In MTP mode, I have to add the runsettings file in via the commandLine like:

dotnet test -- --settings absolute/path/to/runsetting

Relative path raise error in MTP mode and also i have to reconfigure it again in VS 2022 Test Explorer.

The runSettings file provide a flexible way to configure how tests are discovered, run, and reported and it is supported by Visual Studio, vstest, dotnet test, and many CI/CD systems.

So, it is nice if we simplify its configuration as MSBuild property and continue to use RunSettingsFilePath especially MTP support new MSBuild properties.

[Youssef1313]

NOTE: The PR you mentioned is for the new dotnet test experience which is shipping for .NET 10 (currently in preview). The dotnet test experience that you see to be using is the old one (which requires the extra --). Note that supporting RunSettingsFilePath was discussed in #4636 and we decided not to do anything about it.

You may also try to do something like this in your Directory.Build.props or Directory.Build.targets:

  <Target Name="_MTPRunSettingsFilePath" BeforeTargets="ComputeRunArguments">
    <PropertyGroup>
      <!-- Make sure RunSettingsFilePath is either absolute, or relative to test executable -->
      <RunArguments Condition="'(RunSettingsFilePath)' != ''">$(RunArguments) --settings $(RunSettingsFilePath)</RunArguments>
    </PropertyGroup>
  </Target>

[moh-hassan]

Thanks @Youssef1313 for reply.
I read the comment in #4636 by @nohwnd:

This imho opens it up for having a different runsettings file per project, I would be careful with that.

This caution can also exist, if we use the new MsBuild property TestingPlatformCommandLineArguments either in the project or in Directory.Build.props like:

<TestingPlatformCommandLineArguments>$(TestingPlatformCommandLineArguments) --settings $(RunSettingsFilePath)</TestingPlatformCommandLineArguments> 

And that is my workaround solution to continue use RunSettingsFilePath and its working fine in MTP either in vs 2022 test explorer or dotnet test without options .

Also, It is allowed to use the commandLine for the different projects in solution like:

dotnet test --project myproject1  -- --settings my_1.runsettings
dotnet test --project myproject2  -- --settings my_2.runsettings

So, If RunSettingsFilePath is used per project in MTP mode, it will be passed as settings option to the executable of that project and no confusion can be exist.

[nohwnd]

I don't think there is anything particularly bad about having settings per project, but we don't have the settings categorized as "apply per project" and "apply to whole run", which can lead to confusing behavior especially when running via dotnet test vs running in VS, and push us into patterns that we might not want to have, like the one I think I mentioned in the linked issue: dotnet test running one vstest.console per project, and because we apply the per project runsettings there, including the properties that are "per whole run" we would regress if we change it to more optimal approach.

We also would like to get rid of runsettings as first class citizen in MTP and replace it with better abstractions for configuration.

[moh-hassan]

Thanks @nohwnd for your comment.
It is accepted in MTP to apply the per project runsettings with excluding that are "per whole run" as documented here
The point is runsetting include a vital setting in Nunit section which is managed by Nunit3TestAdapter to configure how tests are discovered, run, and reported.

Currently, in Nunit MTP mode, VS 2022 Test Explorer is using RunSettingsFilePath property and dotnet test is using commandline option -- --settings absolute/path.
It is nice if the Msbuild property RunSettingsFilePath can be added as a new feature to the package Microsoft.Testing.Extensions.VSTestBridge to unify the way of handling runsettings by both VS and dotnet test until the readiness of replacing runsettings.
As you said, 

We also would like to get rid of runsettings as first class citizen in MTP and replace it with better abstractions for configuration.

Currently, thanks to the new Msbuild property TestingPlatformCommandLineArguments, i reuse RunSettingsFilePath by the workaround:

<TestingPlatformCommandLineArguments>$(TestingPlatformCommandLineArguments) --settings $(RunSettingsFilePath)</TestingPlatformCommandLineArguments> 

[Youssef1313]

Moving away from VSTestBridge is a goal for me. I'd like to ship MSTest package that's purely MTP at some point. So I wouldn't add that feature to the VSTestBridge for now, especially that it's already easy to get it to work with simple MSBuild stuff.

As it seems the team is generally opposed to supporting RunSettingsFilePath out of the box, I'm going to close this issue. Using TestingPlatformCommandLineArguments MSBuild property for that sounds very reasonable to me.