Timeout failures on MacOS

[rprouse]

After setting up the macOS build on VSTS, I am seeing the following test failures occasionally,

1) Failed : NUnit.Framework.Attributes.RepeatableTestsWithTimeoutAttributesTests.ShouldPassAfter2RepeatsAndTimeoutIsResetEachTime
Test exceeded Timeout value of 85ms

2) Failed : NUnit.Framework.Attributes.RepeatableTestsWithTimeoutAttributesTests.ShouldPassAfter3RetriesAndTimeoutIsResetEachTime
Test exceeded Timeout value of 85ms

https://dev.azure.com/nunit/NUnit/_build/results?buildId=731&view=logs

[rprouse]

I thought this might be a difference in the resolution of the system clock on macOS vs Windows, but the tests use a sleep of 30 ms. The resolution on macOS is 10ms and on Windows 15.6 ms. so that isn't it. Possibly just slower macOS machines in the VSTS build pool?

[CharliePoole]

Running the test three times, with a sleep of 30ms and a timeout of 85ms seems like a very chancy way to verify that the timeout is reset correctly for each execution. The other one with two repeats sounds like it has a better chance of success, but my own experience has been that sleeps can easily end up taking two or three times longer than the value you set. It may be that the entire approach to testing this feature needs to be reviewed.

[oznetmaster]

It may be that the parallelization issue I discussed previously is part of the problem.

I had to disable parallel execution on those tests to get my build to pass in CF.

[DaiMichael]

Well I can feel a little guilty with this, I wrote them 😞. I have always hated writing tests that use Thread.Sleep or something like that, they are always brittle. As this is all about Timeout and Repeat and Retry, I didn't really try to come up with another solution as the tests are in a way replication to the usage even if a bit blunt. I tried to keep the sleep values to allow margins whilst not adding too long to the unit tests run time.

These tests should work with #1638 when I finish up on that. I only paused as I had a corrupt branch, so I deleted it and I'm hoping to get on with it this weekend. The upside to avoid politics, is that I deleted the example code (in branch) provided and I can redo it myself and not worry about the contribution thing. (I digress).

I'm trying to think of away around this but currently I can't... at the end of the day we need a Timeout to be less than the total of repeats time as this was the issue #2786, i.e. The Timeout wasn't resetting on each Repeat or Retry.

I'm happy to pick this up but if anyone has any ideas, I'm all 👂. I also have the problem of no MacOS but that is less of a worry to start with.

[mikkelbu]

The tests also fail quite often on Travis (especially on OSX).