Some unit tests fail occasionally when run alongside other tests #1022
I have looked into this already and will create a pull request.
Certain unit tests seem to occasionally pass and occasionally fail, when running all tests together.
All unit tests should pass reliably, independent from one another.
Occasionally some tests will fail when run together.
Complete minimal example reproducing the issue
Run all of the tests, either using the PowerShell build script or through Visual Studio.
Thanks very much for pointing me in the direction of that pull request, I hadn't seen that and I think that looks like a good approach :)
Some of the transient test failures are related to methods such as ExecutionTime as depending on how many tests are currently running, and plenty of other factors, the machine running a test might run the method slower than usual and it might be below the threshold. A form of abstraction might help with that, so I might look into those particular failures if and when this other PR is merged.
Some of the other tests are related to timing as well. Things like NotThrowAfter are occasionally failing due to the methods being run slower than usual. I'm still investigating but right now it looks like a quick win is to increase the waitTime on some of these tests from small values (i.e 100ms) to larger values (i.e. 2s), like some of the more reliable tests. We can then do subsequent work to use timer abstraction in a further PR in the near future.
I'm still investigating this but interestingly it looks like removing the redundant multithreading as described in my other issue is actually helping to remove these transient test errors.
I think a reason for this is that Stopwatch isn't thread safe, yet it's being used in tests which may share instances of it across threads as the action being completed uses the same references to stopwatches. I think there's a combination of different issues at work here though.
Will keep updating this issue when I find more.
@jnyrup my mistake, I looked into it further and it looks like I was wrong about instances being shared.
In general it looks like a lot of factors are at work here, related to performance. If I run the tests on a more powerful machine, they pass 99% of the time. On a slower machine it's rare for me to be able to run them all without at least one failure.
They only seem to fail when run together and it looks like it's only affecting tests which test methods which have precise timings, i.e. ones that compare execution time, or call methods at specific intervals, either using Thread.Sleep or Task.Delay.
We have no performance guarantees when using either method of waiting a specific amount of time, and speed is affected by things like CPU load, process priorities and number of concurrent threads, even from other processes. This does help to explain why the tests only fail when run all together.
Maybe we should look into a similar strategy to what you mentioned previously - mocking the timings for the sake of testing the logic of some of these methods, without actually relying on actual times which vary from machine to machine?