Login Tests Advanced
The Login Tests - Basic page introduced some the of the easier E-gAT concepts found in the tests/test_login.py module. In this section we will delve into some of more difficult and powerful concepts like parallelizing your tests, running them in multiple environments, and customizing their output.
If we look at the methods on the TestLogin class the first two that we see are the setup and teardown methods. These are special methods that will be recognized by the test runner and run before and after our tests are executed. Because this is a SequentialTestSet the test runner will call setup once, then run all of the test methods sequentially, and then call teardown. If we were using the UnorderedTestSet, setup and teardown would be called before and after every test method.
In this particular case we are using our setup and teardown methods to boot up our browser using Selenium, and to shut it down when the tests have finished. These methods will be run whether or not tests pass or fail, so we can be sure that we will not be left with browsers hanging open if exceptions are raised during test execution.
Our setup method contains a few more interesting pieces. We can see that it too has a decorator, and in the first line of the method body we are accessing the test environment. You can read about SharedResource decorators and test environments in the sections below.
Executing tests in parallel is one of the primary design considerations for E-gAT. The longer a test suite takes to run the less useful it becomes. E-gAT seeks to help the test writer easily write thread-safe tests to reduce test execution time by leaps and bounds.
Telling E-gAT to run tests in multiple threads is pretty easy. Just pass the -t flag with the number of threads you wish to use to the egatest command. For example, to run the login tests in three threads you would execute egatest -t 3 test_login. Now, because TestLogin is a SequentialTestSet, running in three threads will not give us any speed boost, because each test method in TestLogin has to wait for the previous method to finish before executing. But throughout the wiki you will see many examples of the -t option and a number of features designed to make it easier to use with your tests.
One of the hardest parts of thread-safe programming is making sure that access to shared memory is controlled, so that memory is not being read in one thread and written to simultaneously in another. Similarly when writing tests there are often various resources that multiple tests need to use but not all of them can use a given resource at the same time. In a perfect testing environment every test should be executed in a completely sterile environment and should not be sharing any resources with other tests, but often resource or time limitations create dependencies that make the environment less than perfect. When that happens, E-gAT is there to help.
One good example of this is found in the LoginTest class. Selenium works by booting up a full instance of a web browser to simulate a real user. But booting up multiple instances of a web browser simultaneously can cause some web browsers to throw errors. It happens on various platforms and with various browsers, and the error message usually looks something like this:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/matt/dev/egat_example_project/env/lib/python2.7/site-packages/egat/test_runner_helpers.py", line 71, in run
self.run_tests_for_node(cur_node)
File "/Users/matt/dev/egat_example_project/env/lib/python2.7/site-packages/egat/auto_threaded_test_runner.py", line 225, in run_tests_for_node
instance.setup()
File "/Users/matt/dev/egat_example_project/tests/test_login.py", line 18, in setup
self.browser = browser_helper.get_browser(browser_type)
File "/Users/matt/dev/egat_example_project/tests/test_helpers/browser_helper.py", line 23, in get_browser
return webdriver.Chrome()
[Some of the traceback has been removed]
URLError: <urlopen error [Errno 61] Connection refused>
So how can we solve this? We want to run our tests in multiple threads but booting up multiple browsers at the same time is problematic. The answer is with E-gAT's SharedResource class. The Selenium webdriver class, which represents the web browser, is a resource that many of our tests share. Specifically, the browser startup process is a resource that only one thread at a time can have access to. The SharedResource class represents exactly that: a resource that only one thread gets to use at a time. By telling the test runner which tests need to use this resource it can make sure that only one thread at a time gets to use it.
In the case of our TestLogin class, we have subclassed SharedResource in a class called BrowserStartupResource. You can take a look at it in the egat_example_project/test_helpers/browser_helper.py module. It is pretty simple, but now that we have created our own SharedResource we need to show the test runner which methods need it. We do this with the SharedResource.decorator method. If you look at the setup method on the TestLogin class, you will see that we have done exactly that. By decorating the setup method with the BrowserStartupResource have told the test runner that setup cannot be executed while any other threads are using the BrowserStartupResource.
Open up test_customers.py. You will see that its setup method is also decorated with BrowserStartupResource.decorator. In fact all the tests that use Selenium to start up the browser will be decorated with this, to ensure that running any of these tests concurrently will not cause the error we saw earlier. If you want, you can execute the test suite again and see this in action. Run egatest -c test_configs/suite.json and you will see multiple browsers open up, but not all simultaneously.
E-gAT makes it fairly easy to set global "environment variables" using the configuration dictionary. But in modern day webapps we have the universal issue of browser compatibility. Testing a webapp in one browser is simply not enough. With E-gAT we could easily create multiple configuration files, one for each browser, and run them separately. E-gAT makes this a little bit easier however, by allowing the tester to define multiple sets of environment variables in one configuration file.
This is done through the 'environments' key in the configuration file. The 'environments' key points to a list of dictionaries, where each dictionary is it's own set of environment variables. Open up the egat_example_project/tests/test_configs/advanced_login_config.json configuration file to see an example.
The 'environments' list works alongside the 'configuration' dictionary to create two sets of environment variables that you can use in your tests, a local environment and global environment. As we have already seen, the global environment is accessible to each test in self.configuration. Likewise, a test can access its local environment using self.environment. In this case we are using the local environment to specify which browser the tests should be executed in. Obviously the local environment can be used for any purpose, not just executing front-end webapp tests in multiple browsers.
In TestLogin's setup method, we are calling out to the browser_helper module to instantiate our browser based on the 'browser' key in the self.environment dictionary. Feel free to look at that code in the egat_example_project/tests/test_helpers/browser_helper.py file. Looking at our configuration file again, we have defined two environments: one where "browser" = "Firefox" and one where "browser" = "Chrome". When we run tests using this configuration file every test will be executed twice, once in the Firefox environment and once in the Chrome environment. If we had 5 environments defined then the tests would be executed 5 times. If we left the -t option out, or set -t 1, our tests would still run in one thread, one after the other. But in this case we are executing our tests in two threads, and the test runner will run in the two environments simultaneously. Go ahead and see for yourself (egatest -c test_configs/advanced_login_config.json).
You may have noticed after you executed that last test that no output was printed to the command line. That's because the results were written to an HTML file in the /tests/test_results directory. This was done because we specified the -l option in our configuration file. Remember you can get information about all the valid options by running egatest -h. The -l option specifies a directory where test output should be written, instead of the command line. If you look in the test_results folder at the "Test Run" folder that was generated you will find a results.html file that contains the test results. If you open it up and click the "+" to the left of the "test_login.TestLogin" label you can view a lot of information about the tests that were executed. You can see the thread each method was executed in, the environment it was executed in, and in the details section you can even see a test's Shared Resources, Execution Groups, and timing information. If a test fails its traceback will also be in the details section. You can customize this page with your own CSS file using the --css-path option, or even even write a completely custom TestLogger class.
If you have had any test errors up to this point you might see some .png and .html files lying around. This is a nifty little feature that makes using E-gAT and Selenium together a little nicer. Whenever a test fails, E-gAT will check a test for a self.browser variable which, if defined, it'll automatically take a screenshot of the browser and capture the DOM. This is a huge timesaver when it comes to debugging failed tests, because often a quick look at a picture of the browser is more informative than a traceback.
Now you might be wondering if E-gAT and Selenium are bound a little too closely. E-gAT is a functional testing framework that can be used for writing tests with any tools you choose. However, since browser-based testing is one of the scenarios where E-gAT's features shine the most, Selenium gets a little bit of special treatment.
Looking at the TestLogin class, we can see how E-gAT makes it easy to set up and tear-down resources, run our tests in multiple threads, multiple environments, and make pretty HTML reports that management will understand.
If you'd like to see some more examples of tests using E-gAT, check out the CRUD Tests. If you'd like to read about the test suite as a whole, go to The Test Suite.
Table of Contents