When your test suite grows to include a large number of tests, target platforms and devices, it's no longer feasible to run them one at a time. Learn how to run multiple tests at once to speed up your build.
It's important to be able to run multiple tests at once. Eventually your testsuite will grow large enough that waiting for them to complete one by one just doesn't make sense. But running tests in parallel comes with a number of complications. Let's talk about everything that you'll need in order to successfully start running your testsuite in parallel.
| Requirements for Parallel Testing |
|---|
Test runner that can run multiple tests at the same time. Pytest can do this through the use of a plugin called pytest-xdist |
| Multiple test browsers or devices (at least one per parallel execution thread). Can be tricky to scale easily especially with mobile; cloud services help with this problem. |
| (Maybe) multiple webdriver servers. E.g., one Geckodriver instance per parallel execution thread. |
| Partitioning of system resources required for a test. E.g., each Android session requires the exclusive use of a system port, so simultaneous sessions need to be configured not to use the same port. |
| Ensure total independence of tests (tests cannot assume state from any other tests) |
- The first thing you need to run your tests in parallel is a test runner that can run multiple tests at the same time. If your test runner can't run multiple tests at the same time, you're kind of dead in the water. And unfortunately, there are some common test runners out there that don't have this ability, so if you end up using one of those, you'll have to do some pretty intense gymnastics to figure out how to break your testsuite up into different processes. But luckily for us, Pytest's test runner can parallelize tests within a suite no problem, through the use of a special plugin called pytest-xdist.
- The second thing you need is a number of test devices, or browsers. If you want to run 5 tests at a time in your iOS test suite, you need 5 iOS devices or simulators that can be independently used. Mobile devices pose the biggest challenge here, primarily because it's more computationally and financially expensive to run a mobile device than a web browser. Also, it's quite easy to run a number of different instances of the same browser on your system, but interacting with a number of identically-configured iOS or Android devices involves knowing some unique information about each of them, like an ID. This creates complexity in our test code, as we'll see. This is the area where a device or browser cloud can make life a lot easier. With a device or browser cloud, you don't have to figure out how to provision and maintain each browser or device. Instead, you just use capabilities to request devices that make sense, and as long as your account has access to them, you're good!
- There's a third thing you might need, which is multiple webdriver servers to handle your multiple parallel sessions. Some webdriver servers, like Geckodriver, for example, are only able to handle one session at a time. Thus if you want to run multiple Firefox tests at the same time on a single computer, you'll need to first be running multiple Geckodriver servers on different ports, and figure out how to make sure each parallel test is only talking to one Geckodriver server. Other servers, like Appium or Selenium if you choose to run Selenium in front of your browser drivers, can handle multiple requests at the same time, so you don't need to worry about this.
- Fourth, you need to make sure that each of your parallel sessions aren't competing for resources. What do I mean by resources? Well, one kind of resource could be a device. If two simultaneous sessions are trying to use the same device, you're going to get some very odd behavior and your tests will not work. But there are other kinds of resources that are more invisible to us as well. For example, Appium uses various means of communication in order to work with its drivers. For both the XCUITest and the UiAutomator2 drivers, Appium connects over a local network to talk to a special server running on the device. To facilitate this network connection, Appium uses a local port. A port is just a channel for communication that can be used by one process at a time. Normally when you're running just one test a time, you don't have to think about ports. Appium will just use the default port, and you'll be fine. But if you're running two tests at once, and Appium tries to talk to two devices over the same port, very weird things could happen. Luckily you can tell Appium which port to use via a special capability, so you can make sure that each of your sessions uses a different port. That way they don't step on each other's toes. Or shout in each other's ears, I guess would be a better analogy! We'll look at these capabilities momentarily.
- Finally, you need to make sure that your tests are designed in such a way that they don't step on each other's toes either. What do I mean? Well, when we're initially designing a testsuite, and running tests one after the other, it's not too hard to get into a situation where some of the later tests depend on something that some of the earlier tests did. Imagine that you have a test that performs a login, and a test that runs after the login test that assumes the user is already logged in. This assumption might work when you run all your tests one after the other, but if the tests are all run in parallel on different devices, this assumption would cause the latter test to fail! What we want to strive for is tests that are totally independent from one another. What this means is that each test needs to set up its own state, which is a practice we'll talk about more later on.
| Requirements for Parallel Testing of Browsers |
|---|
Multiple browser drivers: handled automatically if you use browser-specific classes (webdriver.Firefox()). Otherwise if you use webdriver.Remote(), you need to start as many browser drivers as you want parallel execution threads (running each server on a different port: geckodriver -p 4444 and geckodriver -p 4445 for example) |
| Alternatively: use the Selenium Standalone server as a single WebDriver endpoint for tests (the Standalone server can handle multiple simultaneous sessions and will take care of starting up as many driver browsers as necessary) |
| Alternatively: Selenium Grid is a great tool for getting a single WebDriver endpoint in front of many different kinds of browsers running potentially on many different machines. |
Let's zoom in a little bit on requirements for parallel testing of browsers. Of course, you could use a cloud service for this, but assuming you are going to set things up on your own, you have a few options. You can continue to run your own browser drivers, but some browser drivers only support one session at a time. To work around this fact, you can either use the browser-specific driver classes in the Python client, which will automatically start a separate driver server for each session for you. Or, if you want to run multiple sessions using the webdriver.Remote class, you'll need to spin up multiple instances of the drivers you're using. This looks like using the command line flag for each driver to ensure that there are no port conflicts, that they all use a different port. For example, geckodriver -p 4444 for one server and geckodriver -p 4445 for another server. If you don't want this complexity, you could instead download something called the Selenium Standalone server. It's a Java binary that will act as a front end for any of the browser drivers. All your tests would just point to the Selenium server instead of individual driver servers. This is nice especially if you would otherwise be running different drivers for different browsers. If even this doesn't get you the flexibility you need, or you need to run on more browsers, or you don't want your work on your computer to be interrupted by browsers popping up all the time, then you can also check out something called Selenium Grid. Setting it up can be a bit challenging, but it's basically a way to have a single WebDriver server that other WebDriver servers on other hosts can hide behind.
| Requirements for Parallel Testing of Mobile Devices |
|---|
Knowledge about how to identify each device uniquely (either a UDID or a deviceName which is guaranteed to target only one device) |
| Optional: run multiple Appium servers (one for each parallel execution thread), since it makes reading session logs easier than having logs from parallel sessions all jumbled together |
Ensure that each parallel session uses different internal ports as the drivers require them. On iOS, set a unique port with the wdaLocalPort capability. On Android, use the systemPort capability. This will prevent simultaneous sessions from clobbering each other's communication to the device. |
The cleanest way to achieve this is by using the --default-capabilities CLI param from Appium. It allows setting capabilities which will always be in force even if not included in the call to create a new session. Allows easily creating a 1:1 mapping of Appium server to driver ports. |
Now let's talk about getting parallel mobile devices running more specifically. For mobile devices, you need to make sure that you have some way to uniquely identify any particular device. If you have two iOS simulators up and running, for example, you'll need to make sure each of your two parallel tests is only targeting one simulator. You could do this by setting up your capabilities in such a way that they target unique devices. You could also use the udid capability and get the UDID of each simulator from Xcode. The same goes for Android. If you have two devices or emulators running, you can use the udid capability to make sure a given test is targeting a specific device. To figure out the ID of an Android device, you can run the adb devices command, and it will show you the IDs for each connected device. In terms of Appium, you don't need to have two Appium servers running, but you might find that reading the Appium logs for two or more sessions at once could be a bit difficult, so you can run as many Appium servers as you need as long as they're on different ports. This is my preferred strategy since it makes reading the logs quite a bit easier than using just one Appium server.
The other thing we need to do is make sure that the internal ports used by Appium don't conflict across sessions. We do that through the use of two capabilities, one for iOS and one for Android. On iOS it's called wdaLocalPort, and on Android it's called systemPort. We can make sure to use these capabilities in such a way that every simultaneous test session uses a dedicated port. But there's an even more foolproof way of making sure these ports aren't clashing, and that's to assign a particular wdaLocalPort or systemPort to an Appium server when we start the server. Appium comes with a command line option called --default-capabilities, and it lets us specify a JSON string on the command line. This JSON string will be taken to be a set of capabilities that will apply to every session that gets started with this server. It's a great way to set capabilities for a server so that the client doesn't need to remember to set them. So in our case, we could start two Appium servers like so:
appium -p 4700 --default-capabilities '{"wdaLocalPort": 8100, "systemPort": 8200}'
appium -p 4701 --default-capabilities '{"wdaLocalPort": 8101, "systemPort": 8201}'
First we run one Appium server on port 4700, and give it a set of default capabilities that causes wdaLocalPort to be 8100, and systemPort to be 8200. Now we can run another Appium server in another terminal window, on port 4701, with a wdaLocalPort of 8101 and a systemPort of 8201. We could keep adding running servers in this way, making sure that none of these ports will clash across server instances. Now, in our actual test code capabilities, we don't have to think about wdaLocalPort or systemPort. We just have to ensure that each parallel test targets a different Appium server, and the server will already be configured with its own unique internal ports! And we'll talk about how to make this happen with Pytest in a bit. But it's worth pointing again before we do that, if we use a cloud service for our Appium hosts and devices, we don't have to worry about running Appium servers ourselves, or any of these port conflicts, because that is all managed by the service itself.
| Requirements for Parallel Testing with Pytest |
|---|
Install the necessary plugin: pip install pytest-xdist |
When you run your suite, include the number of parallel execution threads with the -n option: pytest-n 5 |
Let's talk about how to make our Pytest suite run in parallel. This is managed with a Pytest plugin called pytest-xdist. To make this plugin available, we just need to install it via pip (or pip3 if that's the command you use on your system), the way we install all our Python packages:
pip install pytest-xdist
So go ahead and run this command now in the same Python environment you've been using all along. It should install pretty quickly. Now, all we need to do is pass a special command line flag, -n, to our test runner, and include a number after that:
pytest -n 5
That number is the number of tests that will be run in parallel. It's pretty simple! But how does it work? Getting into all of those details would be unnecessary complexity, but it is important to know that whenever we tell Pytest-xdist to run with a certain number of parallel tests, it will provision a series of what are known as workers to handle each parallel thread. Pytest-xdist takes care of finding all the tests and deciding which tests go to which workers; we don't need to worry about any of that. But we do need to make sure that different Pytest workers are not trying to use the same devices, or the same servers, if we have a model where each server can only run one test at a time.
We can think about the problem this way. Imagine we have a party with a bunch of people at it, and we need to cook them all a lot of food. We could hire one chef to cook it all using one pot and one stove burner, recipe after recipe, but this would take a long time.
It would be much more efficient if we had a bunch of chefs and a bunch of stove burners, each with its pot. In an ideal scenario, we can imagine that each chef would come to a box of recipes that need to be prepared, take a recipe, and head to a stove burner to cook that recipe. When that chef is done, they can come back, pick up another recipe, and head to another free burner to cook that next recipe. And on and on until all the recipes are finished.
What Pytest-xdist does is take care of hiring as many chefs as we tell it to, and handing a recipe to each chef whenever that chef needs another job. But it's our job to tell each chef which stovetop burner is OK to use. Unraveling the analogy here, each chef is a Pytest worker, in other words a Python process that is ready to run one of our testcases all by itself. And each recipe is a testcase that we need to execute. Finally, the stovetop burner and pot correspond to an Appium or Selenium server and its connected device or browser. So we're responsible for coming up with the recipes, or testcases, and making sure we have enough stovetops and pots, or WebDriver servers and devices. Pytest will take care of the rest. Well, mostly. Pytest doesn't know anything about WebDriver servers or devices. So it wouldn't know how to tell each chef which stovetop to take their recipe to. And we could imagine lots of kinds of problems if the chefs don't do it right! Two chefs could each try to go to the same stovetop and make a different recipe at the same time in the same place, leading to disaster! Or we could have more chefs than stovetops, leaving some chefs sitting and doing nothing. We need to make sure there's one stovetop for each chef, and that each chef has a dedicated stovetop that it always returns to, so the different chefs aren't stepping on each other's toes.
Let's talk conceptually about how we might do this in our framework. The key will be that we can detect, for any given run of a testcase, which Pytest worker will be running that testcase. Each Pytest worker will have its own special ID. So we can set up a system in our conftest.py, where we make sure that a given Pytest worker with a given ID will always get the same WebDriver environment. If we do that, we guarantee that different workers won't clash with one another. We can also make sure that we haven't tried to use more workers than we have environments for them to work with, by checking if there are no free environments for a given worker. If that's the case, we can fail the test suite right away and avoid any potentially odd behavior.
That's it for the theory behind how we're going to get multiple tests running at a time.