Extensible Test Cancellation

[thomhurst]

If I wanted to write an extension that could cancel the test session based on some logic, I don't believe that's currently possible.

I can see there's an (internal) ITestSessionContext in the ServiceProvider. But that only exposed a cancellation token.

Exposing a CancellationTokenSource, or just a Cancel() method could be helpful.

I can (and do) create my own linked cancellation token source in TUnit, but a generic extension would have no concept of that.

[nohwnd]

Do you have some examples of why would the extension trigger cancellation?

Right now there are 2 cancellations basically, app cancellation token that should behave more like an abort, and then per request cancellation that is handled by the test framework.

[thomhurst]

One example I was thinking of is a "Fail Fast" extension.

If you have a long running test suite, you may not want to keep it running if a test has failed.

A fail fast extension could listen to test node updates, if any are failed, cancel the test run.

[nohwnd]

Makes sense. Without thinking too much about if such extension can be truly generic and test framework agnostic. I think it would also be useful for it to receive a source and reason for cancellation. Which is true also for IDE interaction where it could be VSClient and "user cancelled in the UI".

[thomhurst]

Sounds good to me!

Probably better to expose an interface with a Cancel(reason) method then

[Youssef1313]

A fail fast extension could listen to test node updates, if any are failed, cancel the test run.

Isn't that achievable simply by --maximum-failed-tests 1?

If your real scenario is more than what --maximum-failed-tests provide, then yes we will need to open some extensibility point for that. It's good to understand more so that we can have better idea on the needs when designing it.

[thomhurst]

It is but that was just one example. I was thinking the more generically so you can cancel it from any logic path, not just a failed test.

Another example if it helps demonstrate a use case:

I could create an extension that lets users define an attribute on their class something like [TestCollectionTimeout(300_000)]

That extension then listens to test nodes, and each time it receives one from that class, adds the time to a stored value.

Then if it goes over that defined threshold, you could cancel the test run.