Skip to content

The Configuration File

mplorentz edited this page May 7, 2015 · 1 revision

Format

The configuration file is a JSON file that allows you to specify more about how your tests will run than is possible from the command-line. The configuration file will be always be used instead of any command line arguments you pass when you run egatest -c path_to_your_config_file (excluding the -c argument). The configuration file is designed to be flexible, so you can use it to include all your configuration parameters if you want. There are a few reserved keys in the JSON dictionary that will have special meaning to the test runner. These special keys can all be seen in this empty example configuration:

{
	"options": {},
	"tests": [],
	"configuration": {},
	"environments": []
}

Each section below will explain one of these reserved keywords and how they act.

Options

The "options" key in the configuration file is used to configure various aspects of how the tests will be run by the test runner. The available options for the "options" key are exactly the same as the flags for the command-line test runner, egatest. If present, "options" must be a JSON object (dictionary), where each key is one of the command line flags (including the "-" or "--"), and the associated value is the argument to be passed to that flag. If the command-line flag does not take an argument, the value should be an empty string. Take a look at this example:

"options": {
    "-l": "/home/you",
    "--log-level": "ERROR"
    "-u": "",
}

The first key/value pair specifies that the logs should be written in /home/you, the second pair specifies that the log level should be ERROR, and the third signifies that the user has defined the threads in this configuration file.

Tests

The "tests" key in the configuration file is the only required key. It specifies the Python packages, modules, classes, and functions you wish to run. It's format depends on whether you intend to use Auto Threading or User Defined Threading.

Auto Threading

Auto Threading is the default threading mode for the egatest test runner. In Auto Threading mode the test runner will automatically assign tests to threads if you specify multiple threads using the "-t" or "--number-of-threads" option. You can read more about how Auto Threading works on the Threading Modes page.

The "tests" key should have a list as its value when using Auto Threading. Each object in the list is allowed to be one of two types: either a test descriptor string that specifies a Python package, module, class, or function to be run, or a "test object", a JSON dictionary that can specify it's own configuration, environment, and list of tests.

Most of the time the "tests" list will just contain a list of strings. The configuration file would end up looking something like this:

{
	"options": {
		"-t": 4
	}
	"tests": [
		"test_module1",
		"test_module2",
		"test_module3.ThisTestClass",
		"test_module3.ThatTestClass",
		"test_module4",
		"test_module5",
		"test_module6",
		"very_big_test_module.BigTest.one_big_test_func"
	],
	"configuration": {
		"base_url": "http://localhost/test_app",
		"port": 8000
	}
}

Running this configuration file would run all the tests specified in the "tests" list in 4 concurrent threads. As stated above, the "tests" list can also contain "test objects" for more advanced configuration. A test object is a JSON dictionary that can contain its own "tests", "configuration", and "environments". In the above example we have configured our port to be 8000. This is a configuration parameter that all of the tests have access too. Say that one_big_test_func needs to use port 9000 instead. Specifying a new test object gives us the flexibility to do that like so:

{
	"options": {
		"-t": 4
	}
	"tests": [
		"test_module1",
		"test_module2",
		"test_module3.ThisTestClass",
		"test_module3.ThatTestClass",
		"test_module4",
		"test_module5",
		"test_module6",
		{
			"tests": ["very_big_test_module.BigTest.one_big_test_func"],
			"configuration": {
				"port": 9000
			}
		}
	],
	"configuration": {
		"base_url": "http://localhost/test_app",
		"port": 8000
	}
}

In this configuration file we have replaced the "one_big_test_func" line with a "test object". When "one_big_test_func" is run from this configuration file, it's configuration will tell it to use port 9000. It will also still have access to the base_url configuration parameter from its "parent's" configuration dictionary. Test objects can be nested indefinitely, but don't get carried away.

User Defined Threading

If you choose to use User Defined Threading (by passing the "-u" or "--user-defined-threads" options in the "options" dictionary) then "tests" should be a two-dimensional list of test names. Each sublist represents a thread, the number of threads of execution will equal the number of sublists. Each thread will be assigned one sublist and will run the tests specified in that sublist in order. The "environments" key is not allowed when running with user defined threads, and Resource Groups will be ignored. An example of a configuration file running with User Defined Threads would look like:

{
	"options": {
		"--user-defined-threads": ""
	}
	"tests": [
		[
			"test_module1",
			"test_module2"
			"test_module3.ThisTestClass",
			"test_module3.ThatTestClass",
		],
		[
			"test_module4",
			"test_module5",
			"test_module6",
		],
		[
			"very_big_test_module.BigTest.one_big_test_func"
		]
	],
	"configuration": {
		"base_url": "http://localhost/test_app",
		"port": 8000
	}
}

Running this configuration file would execute tests in three threads. The first thread would execute test_module1, test_module2, test_module3.ThisTestClass, and test_module3.ThatTestClass. The second thread would execute test_modules 4-6, and the third thread would execute one_big_test_func from the BigTest class.

Configuration

The "configuration" key is optional and can be used to define a number of parameters that will be available to tests during execution. The "configuration" dict will be parsed by Python's JSON library and the parsed result will be added as a variable named "configuration" to every TestSet subclass the test runner runs. This means that the configuration dictionary can contain any JSON-compatible data you wish to include. A typical configuration dictionary looks something like:

"configuration": {
	"base_url": "http://staging",
	"port": 8000,
	"username": "test_user",
	"password": "pass1word",
	"urls_to_check": ["/home", "/about", "/contact"]
}

If you are running in Auto Threaded mode, it is possible to have multiple configuration dictionaries in one file. The nested configuration dictionaries inherit and override each other much like a subclass inherits and overrides a superclass. The root configuration will be passed down to all tests. If any tests objects have their own configuration dictionary the unique keys will be added to the configuration and when duplicate keys are found the child's value overrides the parent's.

Environments

The "environments" key is only allowed when running in Auto Threaded mode. "environments" should be a list of dictionaries, where each dictionary defines an environment that the tests should be run in. The test runner will run all the tests you specified once in every environment. Each dictionary in the "environments' list will be provided to every TestSet class as a variable named "environment", much like the configuration dictionary. For example, if we had a package of Selenium tests we had written and we wanted to run those tests in the each of the four major browsers we could do so:

{
	"options": {
		"-t": 4
	},
	"tests": ["selenium_tests"],
	"configuration": {
		"base_url": "http://localhost:8000"
	},
	"environments": [
		{
			"browser": "Chrome"
		},
		{
			"browser": "Firefox",
		},
		{
			"browser": "Safari",
		},
		{
			"browser": "IE"
		}
	]
}

If we update the tests in our "selenium_tests" package to check self.environment["browser"] to determine what browser they should run in, then running this configuration file will run our selenium_tests in each browser. We've even specified "-t": 4 so that each environment can be run in parallel.

Running tests in parallel and in multiple environments is very powerful, but the programmer should take extra care to use the proper Resource Group decorators, and realize that running tests in multiple environments can result in a single test being executed in multiple threads at the same time.

Environments can also be nested inside of each other when using test objects in Auto Threading mode. This behavior is best illustrated with an example:

{
	"tests": [
		{
			"tests": ["test_create_object"],
			"environments": [
				{
					"username": "create_user",
					"password": "pass1word"
				},
				{
					"username": "create_manager",
					"password": "pass1word"
				}
			]
		},
		{
			"tests": ["test_delete_object"],
			"environments": [
				{
					"username": "delete_user",
					"password": "pass1word"
				},
				{
					"username": "delete_manager",
					"password": "pass1word"
				}
			]
		}
	],
	"environments": [
		{
			"browser": "Chrome"
		},
		{
			"browser": "Firefox"
		}
	]
}

In this configuration file we have two tests we would like to run: "test_create_object" and "test_delete_object". We want to run each of these tests with two users, the test_create_object tests with the create_user and create_manager and the test_delete_object with the delete_user and the delete_manager. We also want to run all of these combinations in Chrome and Firefox. The above configuration file will accomplish this. The test runner will run the test_create_object tests four times: With the create_user in Firefox, with the create_manager in Firefox, with the create_user in Chrome, and the create_manager in Chrome. The test_delete_object tests will be run similarly.

This functionality gives the tester the flexibility to run tests in as many permutations as they may need. However, if running tests in many permutations is required, it may be better to dynamically create tests in a TestSet subclass and make them available via a custom load_tests() method.

You can’t perform that action at this time.