The Integration Tests Time out

[haacked]

Rather than increase the QED timeout, how can we make our integration tests faster. Part of the problem is that for many test classes, each test has to set up the "entire world."

What if we did one of the following:

1. Sequence our integration tests (I would never do this for unit tests) so that they build on each other and do not need to create the world for each test. This adds a lot of complexity to our tests.
2. Have each test only create the parts of the world it needs and not clean up the world. We would need a way to clean up at the end of each set of tests (as opposed to after each individual test).

Thoughts?

[johnduhart]

Have each test only create the parts of the world it needs and not clean up the world.

This is no good either as it could lead to unintentional contamination between tests.

[shiftkey]

Nudge nudge, wink wink: https://xunit.codeplex.com/workitem/9747

[shiftkey]

Also, this is still a thing to be wary of with our current Appveyor setup...

[haagenson]

On large test suites like this in the past where setting up was important, I've seen and used "the world" caching. Essentially, a test creates something if it needs it and then caches it. The cache is checked before each test's data setup to see if it was already cached during that test session. I've only used this with the built in mstest, but it might work with xunit as well?

[shiftkey]

I've looked into enabling parallel builds with xUnit 2 but am hitting some issues with the integration tests running in parallel. My psychic debugger thinks this is related to contention with making requests to the same remote server...

[haacked]

We could run the unit tests in parallel, but not the integration tests.

[shiftkey]

Just tested this out, and it does shave off a couple of seconds:

Before

=== TEST EXECUTION SUMMARY ===
   Octokit.Tests  Total: 1026, Errors: 0, Failed: 0, Skipped: 0, Time: 8.756s
   Octokit.Tests-NetCore45  Total: 588, Errors: 0, Failed: 0, Skipped: 0, Time: 5.373s
   Octokit.Tests-Portable  Total: 588, Errors: 0, Failed: 0, Skipped: 0, Time: 5.332s

UnitTests   00:00:41.2409233

After

=== TEST EXECUTION SUMMARY ===
   Octokit.Tests  Total: 1026, Errors: 0, Failed: 0, Skipped: 0, Time: 5.372s
   Octokit.Tests-NetCore45  Total: 588, Errors: 0, Failed: 0, Skipped: 0, Time: 3.219s
   Octokit.Tests-Portable  Total: 588, Errors: 0, Failed: 0, Skipped: 0, Time: 3.240s

UnitTests   00:00:40.7780810

[shiftkey]

Running the tests in the new VS test runner suffers the same problem as the command line, so I'd like to get to the bottom of this issue. But I can work around it in the meantime and run batches of tests...

[github-actions]

👋 Hey Friends, this issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Please add the pinned label if you feel that this issue needs to remain open/active. Thank you for your contributions and help in keeping things tidy!