Command line paramters not prioritised the way they are in other applications

[j4m3s]

Not sure if this is a bug or a feature request. The command-line runner is not doing as I would expect :)

Configuration

Operating system: Ubuntu 20.04

PlatformIO Version (platformio --version): PlatformIO Core, version 5.1.0

Description of problem

I would expect that command-line switches override values in the platformio.ini file, because that's the way most (all?) other packages work. E,g. with ansible, I specify config vars in an ini file that override the defaults. If I then specify a command-line argument, the argument overrides both the value in the ini file and the application's defaults. That doesn't appear to be the case with the "pio test --filter" switch at least.

Steps to Reproduce

1. Add a test_filter = abc, def, xyz value into a [env:native] section in platformio.ini
2. run pio test -e native -f xyz

Actual Results

The command-line flag is ignored/ overridden by the ini file entry - all test suites specified in the ini file are run

Expected Results

I would expect to only run the xyz test suite in the native environment.

Additional info

This is a problem because a test_filter in an env section is common, as some tests require the microprocessor so will not run natively. Being able to filter to specific tests when running them natively is very useful when adding new features.

[ivankravets]

Thanks for the report! Please re-test with pio upgrade --dev.

[dzid26]

One tradeoff is that now VScode tests explorer doesn't skip tests based on the test_filter (only based on test_ignore.
#5434
It would be best would be to take a conjunction of cli and ini filters - i.e. apply both.