-
Notifications
You must be signed in to change notification settings - Fork 0
Parallel test execution and test isolation
When the tests are running slowly in a project, sooner or later someone will come up with the idea of running the tests in parallel instead of using a single thread. This can make a lot of sense in certain situations, but not always. For example in case your tests are relying on some shared resources or have a chance of leaving some side effects, most probably adding the second thread to the runner will generate more headaches than desired while only bringing a potential 2x performance uplift. Implementing proper synchronization and context switches in these cases can be easier said than done.
Using JVM forks is one of the tools solving this isolation problem as it is perfectly separating the tests running in one fork from the ones running in the other. Opting for this solution can be a quick win as long as you only need isolation and test cases don't depend on shared resources altered by both.
If you can't use forks or you are not willing to accept the limitations coming with them, you can most likely figure out some clever way to remove the dependency between the tests by somehow stopping the use of the aforementioned shared resource. This can be hard, sometimes it is impossible without changing the production code as well to allow us writing proper tests. This is recommended either way if you are following F.I.R.S.T. as your tests need to be independent in this case. Of course, we are not necessarily talking about unit tests here, still having independent tests is not only a noble goal, but will let you get away without using forks at the same time.
There are many ways to implement/configure parallel tests, depending on the test framework of your choice. They tend to provide options to let you define the number of threads you can use, the approach you want to take, like using parallel runs on the method/class/suite level etc. The benefit is having some potential for speeding things up up to the number of threads used. Let's take a look at this topic then.
As the number of threads means how many things we are doing at the same time, we need to assume, that the ideal scenario will reduce the time needed based on this number. Unfortunately the time needed will not really decrease at the same rate as we are adding more and more threads, due to the fact that we need to divide the time needed by the number of threads. Illustrating this, given a project which takes 10 minutes (600 seconds) to test, increasing the number of threads can give us only the benefit indicated by the table below:
| # threads | Time needed |
|---|---|
| 1 | 600 secs |
| 2 | 300 secs |
| 3 | 200 secs |
| 4 | 150 secs |
| 5 | 120 secs |
| 6 | 100 secs |
| 7 | 86 secs |
| 8 | 75 secs |
| 9 | 67 secs |
| 10 | 60 secs |
Looking at this table, it must be clear how little it matters to add a new thread if you have a bunch already. If I were you, I would be not going above 4-5 threads at all as it is not that beneficial compared to the cost.
Unfortunately, running your tests in parallel is not giving you the kind of guarantees you would hope speaking of the execution time. The framework will not always utilize all of the threads 100% efficiently as this is a very complex thing to do. Also, you need to set up test classes, merge results, schedule the work etc. which will use some time on top of the test execution. If you are using a modern CPU, most likely it will already support single-threaded workload better, as both AMD and Intel have solutions that boost higher if only a limited set of cores are used.
Given all these, if you decided to take it easy and only go for 2-3 threads, you are still "paying the full price" in terms of test design and making sure that your tests are fully isolated. It will simply not work reliably otherwise, it is worth taking a look, how good your test implementation is when you start. If your tests are all isolated already, you have nothing to fear, just set a few properties and fidget with your runner. If it really runs faster, you are all set, if it isn't then at least you tried, no harm in seeing it for sure.
Abort-Mission is unfortunately a shared resource used and changed by test class and method executions all over the place. I imagine it is not a great look, lucky for us, there is a silver lining. Even though Abort-Mission context is shared between tests and is written quite heavily, there are ways to mitigate this issue.
For example, you can use multiple named contexts if needed, this way, your tests don't need to access the same resource. Furthermore, this limitation was seen early in the planning phase, therefore thread safety is handled as mission-critical. We are using thread based resources both in Jupiter and TestNG and the core parts are using components which are capable of handling the challenges coming from these multi-threaded use cases.
Forks are no longer supported by Abort-Mission. Unfortunately Strongback functionality (fork support) had to be removed as it was slow and hard to maintain.
Reporting is using a single file at the moment for output. If your setup results in splitting your tests into multiple test suites and you are running those one by one in separate threads, the reporting output will be likely incomplete. You might get lucky, but it is strongly recommended that you use a single test suite in TestNG or a single test launch from Jupiter for the best results. If this is not what you would like to do, you will need to solve the reporting issue as well.