Runsettings from command line (--) doesnt have capability to pass TestRunParameters

[Faizan2304]

Description

TestPlatform can handle only those settings which is of the form

<RunSettings>
    <node1>
        <node2>
             <node3>value</node3>
        </node2>
    </node1>
</RunSettings>

by running command dotnet test -- node1.node2.node3=value

It doesn't have capability to pass attribute and its value for a runsetting element.
for example:

  <TestRunParameters>  
    <Parameter name="webAppUrl" value="http://localhost" />  
    <Parameter name="webAppUserName" value="Admin" />  
    <Parameter name="webAppPassword" value="Password" />  
  </TestRunParameters>

Creating this issue to enhance command line to handle the above scenario.

[Faizan2304]

/cc @pvlakshm

[harshjain2]

For the above example, can we have something like
dotnet test -- TestRunParameters.Parameter(name=webAppUrl,value=http://localhost) TestRunParameters.Parameter(name=webAppUserName,value=Admin) TestRunParameters.Parameter(name=webAppPassword,value=Password)

Currently, if same argument is passed more than once with different values after --, last value is used and other values are ignored, so we might need a way to support passing same arguments with different values in such a way that they become multiple nodes in the runsettings.

[jhgoodwin]

Note that the addition of command line parameters should overlay the file contents passed in through the dotnet test -s <settingfile>

In this way, you can easily have default test options, but override just a few.

[jhgoodwin]

@harshjain2 Personally, I would rather error than allow multiple overrides in the command line. Odds are if I provided multiple overrides to the same key, I made a mistake.

[ebekker]

Maybe add a flag to indicate intentionally adding multiple values, for example
TestRunParameters.Parameter(name=webAppUrl,append=true,value=http://localhost)

[sbaid]

It might be better to add testrunparameters as first class parameter in the cmdline.

dotnet test --TestRunParameter:webAppUrl="http://localhost --TestRunParameter:webAppUser=faizan

[codito]

It might be better to add testrunparameters as first class parameter in the cmdline.

It is adapter specific currently, we should think if other adapters are interested in implementing this behavior (in order to make it first class in vstest.console)?

[pvlakshm]

Closing; reactivate if still required.

[jhgoodwin]

If you can show me where the unit tests are for related behaviors, and the argument parsing, I will add the feature myself.
The core problem with requiring a config/settings file to execute tests with parameters is that it makes it harder to make general purpose tests. For example, I have created a set of unit tests which allow validation of project and solution configuration. For example: hint paths for Project Dependencies must exist, warnings must be set as errors, dependencies across projects in the same solutions must be imported in the same manner, so on. Reusing this test project requires that I somehow inform the unit tests which project files to inspect. Without command line parameters, settings files must be managed in addition to invoking the test runner.

As a test runner is already a wrapper to a test framework for the purposes of execution, it seems unnecessary to wrap it again, just so I can pass in arguments. As an alternate solution, I have tried environment variables, but they don't seem to make it to the execution context of the test.

Thanks in advance for any assistance you can provide @pvlakshm

[onionhammer]

It seems like this is still unresolved - why was it closed so hastily? The thing that makes the least sense is how it works right now, which is to simply throw up a random XML error

[remiX-]

So is this possible now? :|

[mvarblow]

This still doesn't seem to be possible. Can this issue be reopened?

[zootyzoot]

I also need to be able to modify TestRunParameters from cmdln.
Please re-open this issue.

[remiX-]

Yes pleaaase

[dschob]

Please make this work!

[jhgoodwin]

For those of you struggling with this, I was able to work around it for configurations. I'm not too thrilled with the pattern, so if anyone has a better answer, I'll gladly switch.

The idea is that my test classes can ask for a configuration. Locally, I tend to use the user-secrets. On my build server, I use environment variables. So far, this has helped me:

$ cat Config.cs # static config builder
using System;
using System.IO;
using System.Reflection;
using Microsoft.Extensions.Configuration;

namespace MetadataService.Server.Oracle.Tests
{
    public static class Config
    {
        private static readonly Lazy<string> _getTestRootPath =
            new Lazy<string>(() => Path.GetDirectoryName(Assembly.GetAssembly(typeof(Config)).Location));
        public static IConfigurationBuilder CreateConfigBuilder()
        {
            var user = Environment.GetEnvironmentVariable("USER")
                ?? Environment.GetEnvironmentVariable("USERNAME")
                ?? throw new NullReferenceException($"Expected environment to have a USER or USERNAME variable");

            return new ConfigurationBuilder()
                .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                .AddJsonFile($"appsettings.{user}.json", optional: true)
                .AddUserSecrets(typeof(Config).Assembly, true)
                .AddEnvironmentVariables();
        }
    }
}
$ cat AbstractOracleTests.cs # example of me consuming the config
using System;
using System.Data;
using MetadataService.Contracts.Exceptions;
using Microsoft.Extensions.Configuration;
using Oracle.ManagedDataAccess.Client;

namespace MetadataService.Server.Oracle.Tests
{
    public abstract class AbstractOracleTests : IDisposable
    {
        protected IConfiguration Configuration { get; }
        protected IDbConnection Connection { get; }

        protected AbstractOracleTests()
        {
            Configuration = Config.CreateConfigBuilder().Build();
            var connectionString = Configuration.GetConnectionString(ConfigKeys.OracleDb)
               ?? throw new ConfigurationKeyMissingException($"Missing required connection string '{ConfigKeys.OracleDb}'");
            Connection = new OracleConnection(connectionString);
        }

        public void Dispose()
        {
            Connection.Dispose();
        }
    }
}
$ cat MetadataService.Server.Oracle.Tests.csproj # I truncated lines for brevity
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <IsPackable>false</IsPackable>
    <UserSecretsId>MetadataService.Server.Oracle.Tests</UserSecretsId>
  </PropertyGroup>
... # truncated lines here
  <ItemGroup>
    <None Update="appsettings.json">
      <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="appsettings.*.json">
      <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>
  <ItemGroup>
      <None Remove="Data\*\*.json" />
      <EmbeddedResource Include="Data\*\*.json" />
  </ItemGroup>
</Project>

[Huffman17]

@pvlakshm Why would you close this? This is something alot of people need. (Currently needing to find work arounds for this issue).

[millsy013]

please reopen this. This is crucial for our integration with Azure Dev ops

[remiX-]

???? @pvlakshm

[kanchm-zz]

A little ugly but you could do something along the lines of:
dotnet test -- TestRunParameters/Parameter[contains(@name,'SETTINGNAME')]/@value=SETTINGVALUE

Left-hand-side of the '=' can be xpath that needs to resolve to an existing attribute or element.

Note: that the xpath cannot contain '='.

[Huffman17]

For Azure Dev Ops, we have moved away from dotnet test and started using VSTest. Instead we use a dotnet publish in our build pipeline than in release, we choose a settings file when running our vstest. This is no longer an issue for us.

[ghost]

Runsetting is an option for me if there's a secure way of storing credentials in it.

[reimer-atb]

@nohwnd I am using bash on Manjaro Linux

GNU bash, Version 5.0.16(1)-release (x86_64-pc-linux-gnu)

$ dotnet --info
.NET Core SDK (gemäß "global.json"):
 Version:   3.1.103
 Commit:    6f74c4a1dd

Laufzeitumgebung:
 OS Name:     manjaro
 OS Version:  
 OS Platform: Linux
 RID:         arch-x64
 Base Path:   /usr/share/dotnet/sdk/3.1.103/

Host (useful for support):
  Version: 3.1.3
  Commit:  ed88943d24

.NET Core SDKs installed:
  3.1.103 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

[nohwnd]

@philipreimer I am currently digging into this, what I plan to do is:

• take the literal part after -- from the Environment.CommandLine to get it unparsed
• pass this to msbuild already escaped, or using single-quoted string

This should take care of both of the issues, you as the caller will just have to ensure that you don't run into the parser of the command line you are using, I cannot do anything about that. Which in the case of bash is avoided by using -- as you already do.

Would be really nice if you helped me test this 🙂

[reimer-atb]

@nohwnd I can help testing.

I also have one general question: why is the format for passing TestRunParameters the way it is? It seems very complicated. I am not familiar with the internals of the dotnet CLI but wouldn't something like this -- TestRunParameters.myParam=myValue TestRunParameters.myOtherParam=myOtherValue be much easier to handle and be just as powerful?

[nohwnd]

@philipreimer I can't answer that I don't have that context either. I started a new issue on your behalf. I think this might be coming from azdo, or maybe was always like that and was never meant to be used "interactively". I am not sure how soon we will get to the issue. It potentially has a ton of dependencies, and so it will be difficult to change.

Thanks for the testing part, I am building dotnet/sdk now, so I am not sure how feasible it will be for you to actually try it on your side 😔

[reimer-atb]

@nohwnd since this issue has been closed what is the usual process now? Will your changes be included in the next version of dotnet sdk? Do I have to wait for that and when will that happen? Or is there some preview version I can download?

[nohwnd]

Heh, I did not know that GH now matches also part of the message, not just whole line. The fix for this is not complete.

[nohwnd]

Right now there is a PR pending with the changes that I've done on vstest side, and I am waiting for them to build and merge, to test a fix that I wrote yesterday for sdk to PR it. Once that PR is merged it will be available via a local sdk build (on windows it's simple you just run build.cmd and then eng/dogfood.cmd, on linux I am not sure yet).

And then when sdk inserts into dotnet/installer it will be available to download directly as a pre-release.

dotnet/sdk#11465

[nohwnd]

I could not get it to not require any escaping, that would only be possible on Windows via PInvoke, and even that was not 100%. So instead I aimed to make it the same syntax for dotnet test, dotnet vstest and vstest.console. It needs a single escapement, and will merge your test run parameters if they contain spaces. Typically ", value or when you provide value with spaces. In powershell --% parsing terminator is the secret sauce. On bash you need to escape all spaces quotes and parentheses, but the error messages are helpful. It also aligns nicely with the current error vstest console message if you do it wrong.

# cmd
dotnet test  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\")

# powershell
dotnet test --%  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\") 

# bash
dotnet test -- TestRunParameters.Parameter\(name=\"myParam\",\ value=\"value\"\) 

At the moment I am waiting for PR to dotnet/sdk to build to see if I pass their tests. It works locally on my Windows.

[reimer-atb]

Sorry for bringing that up but bash is not the only shell used on non-windows systems. For example, zsh is the default shell in current macos versions, I think.

Will the bash version work in those other shells as well?

[nohwnd]

@philipreimer bash was an example. The point is that the syntax should be the same for dotnet vstest, dotnet test and (on windows) vstest.console, no matter how many hops there are. I am aware of zsh, and I hope it will work there as well using the same syntax as in bash, but that depends on the escaping rules that zsh has. There is a merge building in dotnet/sdk so let's hope we can have an installer for that soon.

[nohwnd]

@philipreimer I was able to validate this on bash and zsh on Ubuntu (actually Ubuntu 18.4 in WSL2).

Download and extract to dotnet folder:

mkdir dotnet
wget https://aka.ms/dotnet/net5/dev/Sdk/dotnet-sdk-linux-x64.tar.gz
tar zxvf dotnet-sdk-linux-x64.tar.gz -C dotnet

And then just run it from that folder:

/mnt/c/temp/dotnet/dotnet test --no-build -- TestRunParameters.Parameter\(name=\"myParam\",\ value=\"value\ with\ space\"\)

[NightOwl888]

I am attempting to use the syntax here using cmd.

dotnet test --logger:"trx;LogFileName=TestResults.trx"  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\")

But when I do, I get the error message:

One or more runsettings provided contain invalid token

What (if any) version of .NET Core SDK was this fixed in?

[NightOwl888]

I installed .NET Core SDK 3.1.202 and tried again. Now I am getting a different error message with the same command on the same test project (also using cmd).

dotnet test --logger:"trx;LogFileName=TestResults.trx"  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\")

The test run parameter argument 'TestRunParameters.Parameter(name=myParam,value=value)' is invalid. Please use the format below.
     Format: TestRunParameters.Parameter(name=\"<name>\", value=\"<value>\")

Since I have literally copied and pasted the command from above, it seems like this is a bug of some kind with the latest release.

[nohwnd]

This was fixed in the master of dotnet/sdk, so it will be part of the next 5.0 release. Checking with the team to get the simplest way to get it into 3.1 as well.

[nohwnd]

The vstest.console part was released in this preview https://www.nuget.org/packages/Microsoft.TestPlatform.TestHost/16.7.0-preview-20200519-01

@NightOwl888 we will have to backport to net3.1.xxx after this proves to be working because the bar is a bit higher there than getting it into master of the current dotnet. So this will take some time till it's released... Can't think of any way to get it to you faster, that would be simpler than using the above escaping workaround.

[NightOwl888]

Thanks. Since our use case is a test framework library that many others will use, getting it to "us faster" wouldn't help. We can wait for an official release, but it would be great if we can get a notification when that happens so we can add the documentation about the feature as well as the minimum prerequisite version of .NET Core SDK.

[arbitrary9]

It can be helpful while configuring VsTest step on Azure DevOps.
If u try to implement .runsettings TestRunParameters section for Azure DevOps VsTest step and you have similar log

##########################################################################
DtaExecutionHost version 18.170.30112.1.
Starting TestExecution Model...
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Result Attachments will be stored in LogStore
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Provided settings file:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <RunConfiguration>
        <MaxCpuCount>0</MaxCpuCount>
        <BatchSize>1000</BatchSize>
    </RunConfiguration>
    
    ################# HERE SHOULD BE YOUR TestRunParameters SECTION #################    
</RunSettings>
Updated Run Settings:
<RunSettings>
  <RunConfiguration>
    <MaxCpuCount>0</MaxCpuCount>
    <BatchSize>1000</BatchSize>
    </RunConfiguration>
    
    ################# HERE SHOULD BE YOUR UPDATED TestRunParameters SECTION #################    
    
</RunSettings>
**************** Starting test execution *********************

Please just try to install this nuget to project where is located your .runsettings file