[Testing Platform] Command line arguments file source

[thomhurst]

From what I can tell options are only supported currently from the command line.

It might be useful to try to find a well-known file that can store options within it. This way it can be source controlled by teams, and provide a more deterministic result between local development and CI builds without developers having to know which various flags to set.

E.g.

test-options.json

{
    "minimum-expected-tests": 200,
    "coverage": true,
    "report-trx": true
}

[Evangelink]

We have it partially done but that does need to be finished and documented (cc @MarcoRossignoli).

[MarcoRossignoli]

We have it partially done but that does need to be finished and documented (cc @MarcoRossignoli).

Actually is not yet implemented but it's doable and a good idea, maybe a section like

{
    "testingPlatform" :
    {
        "additionalCommandLine" : "--someoption file --anotherone"
    }
}

That we can parse and add, we need to understand what to do in case of duplicates (override, merge, throw) and ordering of the apply.

[thomhurst]

I'd probably take the file as the base values, and then any actual command line options override them

[nohwnd]

I'd probably take the file as the base values, and then any actual command line options override them

👍 This is how the current vstest works. It takes config file, then answer file, then commandline arguments, then inline settings (settings specified on commandline after --).

One great addition to that would imho be also applying settings file over a settings file, so we can have slightly different settings in CI, similar to my.configy.json, my.dev.config.json

e.g. --settings my-settings.json --settings my-settings.ci.json

maybe a section like additionalCommandLine

@MarcoRossignoli I think there is a good concept hidden in there, IMHO the settings and commandline parameters should be closely related and there should be a definitive step where all settings are finalized and are applied back to a "runsettings" object. This object is then the source of truth for the local run, or a remote run.

In less abstract terms we should be able to write a plugin / execution mode that takes given command line parameters and turns them into runsettings.

[MarcoRossignoli]

In less abstract terms we should be able to write a plugin / execution mode that takes given command line parameters and turns them into runsettings.

My main problem are complex configuration, in command line you cannot express too much and the configuration file in future could change the format like toml. One thing that we can do is maybe override the config with a convention "--overrideconfig-mstest-parallelize true" for scalar value

{
...
 "mstest":
 {
    "parallelize" : false  
 }
}

here in command line we have only strings without type safety.

[nohwnd]

That is the direction I am proposing. Command Line -> config file, not the other way around. So not being able to handle all settings in commandline is okay.

What I would like to avoid is the situation in VSTest where we say that runsettings are the source of truth, but we cannot easily convert from commandline parameters to runsettings, and so distributed runs don't allow all providing additional commandline parameters, and users have to convert to runsettings manually.

[thomhurst]

Or maybe allow choice of ordering like asp/generic host builder?

[thomhurst]

Or maybe allow choice of ordering like asp/generic host builder?

[MarcoRossignoli]

Or maybe allow choice of ordering like asp/generic host builder?

What you mean?

[thomhurst]

Or maybe allow choice of ordering like asp/generic host builder?

What you mean?

The Microsoft host builder model allows you to choose overriding settings by extension methods of .AddEnvVariables or .AddJsonFile. you could let the framework decide that if using automatic hook registrations, or if the framework allows the user to build their own entry main method, they could set up the order themselves.

Would be more work but give more flexibility. Just a thought.