Breaking change in filter functionality between framework 2.7 and 3.11

[jbengaard]

We have a filter, --where="!(test != (something))" which returned the appropriate tests on nunit framework 2,7.0.0. On 3.11.0.0, it always returns an empty set of tests.
This behavior was observed using NUnit Console Runner 3.10.0.

I attached a small example solution containing a few tests using 2.7,0.0 and a copy of the tests using 3.11.0.0. I also attached the output from running the tests with the same filter.

nunit2example_1.zip

[CharliePoole]

This isn't entirely surprising, since the V2 test is run using the V2 framework driver extension as well as the V2 framework. IOW, an entirely separate code base for execution, even though you are using the same console version in both cases.

I assume you are also using the 3.10 version of the V2 framework driver. Have you tried to use any earlier versions of the console + engine + extension? If not, we should do that to see if the error crept i recently or was there from the beginning. I think the latter is quite possible since this is a bit of a corner case.

[jbengaard]

I tried to use nunit framework v3.0.5797 and console runner v3.0.5797, with the same result. Den man. 15. apr. 2019 kl. 16.57 skrev CharliePoole < notifications@github.com>:

…

This isn't entirely surprising, since the V2 test is run using the V2 framework driver extension as well as the V2 framework. IOW, an entirely separate code base for execution, even though you are using the same console version in both cases. I assume you are also using the 3.10 version of the V2 framework driver. Have you tried to use any earlier versions of the console + engine + extension? If not, we should do that to see if the error crept i recently or was there from the beginning. I think the latter is quite possible since this is a bit of a corner case. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/nunit/nunit-console/issues/606#issuecomment-483287342>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AsoVBZ_QPVlbF3gUADZ67FP5FYmMUOufks5vhJNlgaJpZM4cv9r7> .

[CharliePoole]

Sounds like the problem has been there for a while then. Can you avoid that specific syntax as a workaround?

[CharliePoole]

This is probably a framework error, but someone should confirm before transferring it I think.

[jbengaard]

Unfortunately, we have some tests running on framework 2.7 that need that specific type of filter. We have not found a workaround yet.

[CharliePoole]

You should be aware that !(test != X) is not the same as test == X if X happens to be marked Explicit.

That is, a top level not-filter isn't allowed to select an Explicit test. For non-explicit tests, the normal equivalence is intended to apply.

[Sharken03]

Yes, we are aware of how this works for explicit tests, this behavior is very useful as we do not want to run explicit tests.

[cyanite]

You should be aware that !(test != X) is not the same as test == X if X happens to be marked Explicit.

Well, !(test != X) is not the same as test == X in general, since in our case (I am a collegue of @jbengaard), I get 0 tests run with --where="!(test!='Some.TopLevel.Namespace')" and all tests run with the De Morgan equivalent --where="test='Some.TopLevel.Namespace'". But this must be a problem in the framework, as the console engine simply forwards the filter.

It also doesn't work if instead of Some.TopLevel.Namespace you fully qualify a test name.

Edit: Since both !(...) and != is internally converted to <not>, this means that test=X and !(!(test=X)) are also different.

[CharliePoole]

Yes, I think we understand the problem you are seeing. I am only pointing out one small exception where Explicit is involved.

Someone will need to verify that the generated filter is correct before possibly moving the issue to the framework repo.

My guess is that it's a framework issue related to how top level not is handled. That could be tested by producing your expression with "test == junk ||..."

[CharliePoole]

I can confirm this bug. See build at https://ci.appveyor.com/project/CharliePoole/nunit-console/builds/24263259

In this branch I added tests to

1. Verifiy that the filter is generated correctly, for example, !(test != X) gives <not><not><test>X</test></not></not>. These tests pass.

2. Verify that a filter beginning with <not><not> performs as expected. Those tests fail.

I'm going to go ahead and prepare a PR to add the first tests, but not the second ones. We can't have those tests since they will fail unless we first update the framework. We can either leave them out entirely or comment them until the framework version used is updated.

[CharliePoole]

@nunit/engine-team I'm waiting for someone to confirm my view that this issue should be moved to the framework repo before doing it. See the above build.

[jnm2]

@CharliePoole I agree.

[ChrisMaddock]

Me to!