Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specifying Test Selection Language filters in VSTest runsettings #669

Merged
merged 1 commit into from
Dec 13, 2019
Merged

Allow specifying Test Selection Language filters in VSTest runsettings #669

merged 1 commit into from
Dec 13, 2019

Conversation

mletterle
Copy link
Contributor

This allows using dotnet test as a drop in replacement for a
nunit-console run with the --where option.

i.e.:

nunit-console SomeTests.dll --where "cat == SomeCategory"
becomes
dotnet test SomeTest.csproj -- NUnit.TestsWhere="cat == SomeCategory"

Related: #425, #655

@OsirisTerje
Copy link
Member

OsirisTerje commented Oct 30, 2019

@mletterle The idea of adding test filters to the runsettings is a good one, I like that one!
But, I don't really understand the connection with dotnet test, which already supports the --filter. Can you elaborate a bit on this?

Also, is this really two different things in one PR?

Do you see this PR as resolving issues with #425 and #655

@mletterle
Copy link
Contributor Author

mletterle commented Oct 30, 2019

@mletterle The idea of adding test filters to the runsettings is a good one, I like that one!
But, I don't really understand the connection with dotnet test, which already supports the --filter. Can you elaborate a bit on this?

Sure, the filter used by --filter is VSTest's filter dialect which is different then NUnit's native Test Selection Language

Not only is the syntax different, but as far as I can tell VSTest's filtering is far more limited than NUnit's native filtering.

This PR allows one to use NUnit's native Test Selection Language from the dotnet test command (and also via dotnet vstest)

Also, is this really two different things in one PR?

Do you see this PR as resolving issues with #425 and #655

The two issues have a common source, they're both about selecting which tests you actually want to run under NUnit.

#425 is related because this allows you to run dotnet test -- NUnit.TestsWhere="arbitrary Test Selection Language Filter"

#655 is related because, as far as I can tell it's actually not possible to use VSTest filter to select a specific parameterized test, but it is possible with the Test Selection Language, so for example you could do:

dotnet test -- NUnit.TestsWhere="test == 'My.Name.Space.TestFixture.TestMethod(5)'"

@OsirisTerje
Copy link
Member

Thanks @mletterle ! Makes sense!

I'll add some review comments :-)

@OsirisTerje OsirisTerje self-requested a review November 1, 2019 17:57
src/NUnitTestAdapter/AdapterSettings.cs Outdated Show resolved Hide resolved
src/NUnitTestAdapter/NUnitTestFilterBuilder.cs Outdated Show resolved Hide resolved
src/NUnitTestAdapter/NUnitTestFilterBuilder.cs Outdated Show resolved Hide resolved
src/NUnitTestAdapterTests/TestExecutionTests.cs Outdated Show resolved Hide resolved
@mletterle mletterle requested a review from OsirisTerje November 8, 2019 14:37
@mletterle
Copy link
Contributor Author

@OsirisTerje Is there any further changes you'd like to see?

@OsirisTerje
Copy link
Member

I think this is fine. I would like to have this out together with the other changes in 3.16. Have you merged the latest changes from master in here? And verified there are no crashes there ?

@OsirisTerje OsirisTerje added this to the 3.16 milestone Dec 13, 2019
This allows using `dotnet test` as a drop in replacement for a
nunit-console run with the `--where` option.

i.e.:

`nunit-console SomeTests.dll --where "cat == SomeCategory"`
becomes
`dotnet vstest SomeTests.dll -- NUnit.Where="cat == SomeCategory"`
or
`dotnet test SomeTests.csproj -- NUnit.Where="cat == SomeCategory"`
@mletterle
Copy link
Contributor Author

I rebased the PR against current master, everything appears ok.

@OsirisTerje OsirisTerje merged commit 771790f into nunit:master Dec 13, 2019
@OsirisTerje
Copy link
Member

Btw, are you writing blog posts? This change could deserve a post.

@mletterle mletterle deleted the nunit-filter branch December 13, 2019 21:15
@mletterle
Copy link
Contributor Author

It's been a long time since I wrote a blog post ^_^

It's not a very complicated feature, I could try and put something together this weekend, if it would be helpful.

@OsirisTerje
Copy link
Member

It's obvious to those close to the issue at hand :-) But not so for many others. I am often a bit surprised by how well received simple explanations can be.
So it would be awesome if you did! I'll refer to it in the release notes afterward too.

@mletterle
Copy link
Contributor Author

@OsirisTerje
Copy link
Member

Awesome! Thanks a lot!

@OsirisTerje
Copy link
Member

@mletterle Version 3.16 has been published. Links to your blogpost added in release notes and on the Trick and Trips page. :-)

@goblinmaks
Copy link

@mletterle Is it possible to pass string parameters as part of filter value like
dotnet test -- NUnit.TestsWhere="test == 'My.Name.Space.TestFixture.TestMethod("myString")'"
?

@OsirisTerje
Copy link
Member

@goblinmaks Something like that should work, perhaps with some escaping. If you also set Verbosity to 5, you should see all settings being listed , can be good for checking what the adapter resolves it into :-)

@mletterle
Copy link
Contributor Author

Yeah, shell escaping shenanigans comes into play here, it might be easier to just define the filter in a .runsettings file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants