StaFact facts breaks the concurrency limitations of Xunit

[lg2de]

Xunit has an internally implemented limitation of tests to be executed concurrently. This is implemented based on fixed number of threads executing the tests.

All Fact implementations of StaFact packages creating its own thread, e.g. UIFact, will break the Xunit limitation because the Xunit thread is released after UIFact test execution has been started. I think this causes unexpected high CPU load while test execution resulting in accidental failed test execution, e.g. fluentassertions/fluentassertions#1391 and https://ci.appveyor.com/project/dennisdoomen/fluentassertions/builds/37160306

I've created and attached a sample project showing the problem. For demonstration I've configured the limitation to 2 concurrent tests. All standard Fact tests completing successfully. Most UIFact tests will fail because the maximum number of tests is exceeded.

StaFactTests.zip

[aesalazar]

We occasionally see our tests fails or even hang on our build servers which is mentioned in one of those threads. Have you seen this as well?

[lg2de]

Yes, they have been failed if they rely on timing (assertion of execution time).
They did NOT hang, at least we haven't noticed that so far.

[AArnott]

Thanks for reporting. Any idea how we would fix this? I have no experience with the xunit parallel execution throttle. The xunit methods are mostly async, so it seems designed that tests not always block a thread, so if xunit's throttling assumes the test blocks a thread, that already seems broken without UIFact's help. Can you repro this problem with any [Fact]-based async Task returning method that yields right away with .ConfigureAwait(false)?

[lg2de]

I think it should be ok to block the xunit thread while UIFact thread (and all other thread in this extension) is executing. Synchronous thread will "block" the thread too.
Let say, the UIFact thread replaces temporary the xunit thread.

Can you repro this problem with any [Fact]-based async Task returning method that yields right away with ConfigureAwait(false)?

I could not really understand your question.
And, what is the background of this question?

[AArnott]

what is the background of this question?

I am trying to establish how xunit throttles test runs when tests are asynchronous. Because whatever that mechanism is, we should reuse it for UIFact.

[lg2de]

Xunit starts as many threads as tests should be in parallel. All testing work is distributed to these worker threads.

With the additional threads of StaFact the mechanism cannot be reused, I guess.
Instead I think the the Xunit thread should be blocked while StaFact thread is running.

Maybe @xunit can help?

[lg2de]

@AArnott , when do you plan to create a new release with this issue fixed?

[AArnott]

I'll try to get it out later today.

[AArnott]

This is now available as https://github.com/AArnott/Xunit.StaFact/releases/tag/v1.1.5-alpha

[lg2de]

Hey @AArnott, could you please create a new public release?
I have a new project which must not use alpha or beta releases.

[lg2de]

We found another issue... :(
Please wait with the release.