I've been looking for a while for a test framework that would let me run browser based tests in parallel and it looks like xUnit has most of the required features. I blogged a detailed analysis here (http://www.pabich.eu/2014/06/how-to-run-selenium-tests-in-parallel.html) but I will include all important information here so the discussion is self-contained. The purpose of this discussion is to see whether those missing features fit into the vision of the xUnit project and if so, how they can be implemented. Personally I believe that the famous extensibility capabilites of xUnit could help here a lot (e.g. custom test execution pipeline via a custom implementation of XUnitTestCase). And having one, instead of many frameworks, has its obvious adventages.
I forked xUnit https://github.com/pawelpabich/xunit to see if I can fill the gaps. This was easier than I thought it would be thanks to great condition the xUnit codebase is in. The whole excercise is more a Proof of Concept then a fully blown implementation but it works and it is used successfully in production.
I split the missing features into 3 groups:
Tests need to know when they fail so a screenshot of the web browser can be taken. Current solution checks whether the test class implements certain interfaces, e.g. INeedToKnowTestFailure.
Current design splits the execution pipline into mutliple tasks which means that Dispose method of an already executed test might be run with some delay which means more browser instances will be required than what MaxParallelThreads would suggest. In my tests I very often ended up with 2 or 3 x MaxParallelThreads number of browsers. Current solution is to execute the whole pipleline as a single Task
At the moment the smalles unit of work for parallel processing is class. For long running tests this is not optimal as a class with a few long running tests can significantly slow down the whole build. Current solution introduces a new, per method, collection bahaviour.
Assuming that we can have a test collection per test methed then the next logical step to minimize the exectuion time would be to have ability to order test collections to make sure the failed tests or/and tests with longest execution time are run first. Current solution is based on the existing extensibilty point for ordering tests within a test collection.
At the moment global initialization needs to happen in the instance constructor with global lock.
Current workaround relies on subscription to AppDomain.Unload event which works most of the time but not always.
I really hope we can make it work :). Thoughts?
I think it is important to have those features, or at least to add some extensibility points so it is easier to change its default behaviour without having to branch it.
+1 as well, considering all other unit testing frameworks look archaic by not supporting async / task and running tests in parallel, this would give us xUnit users another reason to look nowhere else, making this a de facto unit and functional testing framework in .NET.
It would be great to have those features in xUnit especially now, when we cannot use any other framework for proper parallel tests run.
All of these requirements tell me you're going to want to implement your own test framework (at least ITestFrameworkExecutor). You should be able to leverage existing code in base classes, especially since the big runner split out that started in Beta 2 and continues in Beta 3.
I will have a look what I can do with this interface. I will be on leave for couple of weeks so it might take me some time.
@bradwilson one more question, I understand that the whole package of changes sounds like a new framework but what is your thinking about features that are extensions of existing features (e.g test collection per method) or conceptually are based on existing concepts (TestCollectionOrderer which is based on TestOrderer)?
I had a look what it would take to create a custom test framework based on xunit that would help me solve my problem. This is that what I ended up with pawelpabich@0494e78.
I'm sure I've missed a few things but the idea was to subclass as much as possible and avoid copying and pasting code and then plug my custom test framework via TestFrameworkAttribute. The end result is that It looks like without additional abstractions in the framework I won't be able to achieve what I'm looking for. I would love to be proven wrong :).
Taking this all into account, would you guys be willing to:
As far as I can see they can be implemented without introducing any breaking changes. I'm more than happy to provide pull requests for them all of them.
We've opened #174 to track the test collection orderer.
As for PreMethodTestCollection, we have solidified on the design of the test collection being the boundary for parallelization. Allowing test methods in the same class to be in different test collections is a major breaking change with regard to fixtures.
We could do your third request (virtualize RunAsync), but it seems like the only reason you'd need that would be to parallelize test methods. Is there any other reason to make that virtual?
👍 Just running into this. Deterministic disposal and any easy way to globally setup and teardown state are my biggest issues.
The demands at the top of this issue are not things we are planning to deal with; our focus with xUnit.net is to keep the core focused on TDD, with extensibility to let other people adapt it to their needs. Our recommendation is still to write your own test framework (implementation of ITestFramework). There is an example of this here: https://github.com/xunit/samples.xunit/tree/master/ObservationExample
That's how I ended up solving it. Abstracted away TestFramework and TestExecutor to handle IIS Express spin up. If anyone else in this thread is interested in building a general purpose selenium test framework on top of xUnit using the Xunit.Sdk, let me know.
@RichiCoder1 would love to have a look
@pawelpabich @RichiCoder1 me too - at this stage I've just subclassed the XUnitTestCaseRunner.
Reading over the goals you were aiming for @pawelpabich, I think we had slightly different use scenarios. I was most interested in per-assembly Setup and Teardown, and I prefer the class-as-collection and per-collection behavior. That being said, I'm cleaning up and de-companyizing my code and will push it out to GitHub in a little. After that, we can make a few issues and see about building an all-encompassing, nuget-pluggable solution. Sound good?
Sounds like a good start @RichiCoder1 :)
Dropped a very basic framework here: https://github.com/RichiCoder1/Browser.xUnit
Going to add the custom setup and teardown extensibility points I added elsewhere here in a little. Feel free to create issues and start discussions. My goals are very simplistic and I want to make sure we come up with a good solution :)
Edit: And it is using Dnx. Ensure you have dnx installed and run dnvm upgrade and well as dnu restore on the repository before building.
@RichiCoder1 looks good, thanks!