Flakey tests in FrameworkControllerTests

[ChrisMaddock]

Several tests in FrameworkControllerTests have recently failed in a flaky manner on the CI builds. We need to look into why that is, and make a fix.

The failures don't appear to be platform-specific, here are a few sample failure messages:

Errors, Failures and Warnings
1) Failed : NUnit.Framework.Api.FrameworkControllerTests.RunTestsAction_AfterLoad_ReturnsRunnableSuite("<filter/>")
  Expected: not null and String starting with "ID"
  But was:  "8601"
  at NUnit.Framework.Api.FrameworkControllerTests.RunTestsAction_AfterLoad_ReturnsRunnableSuite (System.String filter) [0x00000] in <1df9fb68b85648378ea6dd2e640b0ed4>:0 

1) Failed : NUnit.Framework.Api.FrameworkControllerTests.ExploreTestsAction_FilterCategory_ReturnsTests
  Expected: not null and String starting with "ID"
  But was:  "7680"
   at NUnit.Framework.Api.FrameworkControllerTests.ExploreTestsAction_FilterCategory_ReturnsTests() in /home/travis/build/nunit/nunit/src/NUnitFramework/tests/Api/FrameworkControllerTests.cs:line 316

1) Failed : NUnit.Framework.Api.FrameworkControllerTests.RunTestsAction_AfterLoad_ReturnsRunnableSuite("<filter/>")
  Expected: not null and String starting with "ID"
  But was:  "8508"
  at NUnit.Framework.Api.FrameworkControllerTests.RunTestsAction_AfterLoad_ReturnsRunnableSuite (System.String filter) [0x00000] in <9278af9c5d3e455d8858b76816878cf6>:0 

As always, contributions welcome!

[CharliePoole]

@ChrisMaddock Are you saying the same tests sometimes pass for the same code? I had seen this error and figured someone had messed up the setting of the prefix for the ids but if it's only failing intermittently, that explanation wouldn't apply.

[ChrisMaddock]

@CharliePoole Pass - I haven't had a chance to even look at the code, and see what these tests do! I just wanted to record the error messages as an issue.

[mikkelbu]

And again (from https://travis-ci.org/nunit/nunit/jobs/416897254)

1) Failed : NUnit.Framework.Api.FrameworkControllerTests.RunTestsAction_AfterLoad_ReturnsRunnableSuite("")
  Expected: not null and String starting with "ID"
  But was:  "8483"
  at NUnit.Framework.Api.FrameworkControllerTests.RunTestsAction_AfterLoad_ReturnsRunnableSuite (System.String filter) [0x00000] in <4da919d604294820b300e383ceca81a7>:0 

[CharliePoole]

@mikkelbu Link doesn't show that failure.

[mikkelbu]

@CharliePoole Sorry. I forgot that restarting the job cleans the log, and I restarted the job after reporting here.

[CharliePoole]

@mikkelbu I see. And it apparently succeeded when restarted.

The prefix used for all ids generated is stored in Test.IdPrefix, which is static. Maybe some other test is tromping on it.

[CharliePoole]

We might want to rethink the use of a static to store the prefix.

[mikkelbu]

I was thinking the same regarding the static, but I've only skimmed the code very quickly.

Edit: @CharliePoole the build failed again at (https://travis-ci.org/nunit/nunit/jobs/416897254) and this time I'll not rerun it immediately 😄

1) Failed : NUnit.Framework.Api.FrameworkControllerTests.ExploreTestsAction_FilterCategory_ReturnsTests
  Expected: not null and String starting with "ID"
  But was:  "7680"
   at NUnit.Framework.Api.FrameworkControllerTests.ExploreTestsAction_FilterCategory_ReturnsTests() in /Users/travis/build/nunit/nunit/src/NUnitFramework/tests/Api/FrameworkControllerTests.cs:line 316

[ChrisMaddock]

Just cleared this on another PR.

A little concerned what's changed recently to cause these - they seem very frequent.

[rprouse]

I just saw this again on another PR. It is a bit worrying...