The driver.classic
module's main contribution is to provides a TestDriver
convenience subclass: ClassicTestDriver
, that test driver implementations are invited to derive from.
It starts with an assumption, considered to be common to most real world use cases: testcases are atomic, meaning that the execution of each testcase is a single chunk of work that produces a single test result. This assumption allows to provide a simpler framework compared to the base TestDriver
API, so that test drivers are easier to write.
First, there is no need to create fragments <test_fragments>
and handle dependencies: the minimal requirement for ClassicTestDriver
subclasses is to define a run
method. As you have probably guessed, its sole responsibility is to proceed to testcase execution: build what needs to be built, spawn subprocesses as needed, etc.
ClassicTestDriver
considers that most drivers will need to create the temporary directories, and thus make it the default: before running the testcase, this driver will copy the test directory to the working directory. Subclasses can override this behavior overriding the copy_test_directory
property. For instance, to disable this copy unconditionally:
class MyDriver(ClassicTestDriver):
copy_test_directory = False
Alternatively, to disable it only if the test.yaml
file contains a no-copy
entry:
class MyDriver(ClassicTestDriver):
@property
def copy_test_directory(self):
return not self.test_env.get("no-copy")
Although the concept of "test output" is not precisely defined here, ClassicTestDriver
has provisions for the very common pattern of drivers that build a string (the test output) and that, once the test has run, analyze of the content of this output determines whether the testcase passed or failed. For this reason, the self.output
attribute contains a Log
instance (see result_log
).
Although drivers generally want to deal with actual strings (str
in Python3, a valid sequence of Unicode codepoints), at the OS level, process outputs are mere sequences of bytes (bytes
in Python3), i.e. binary data. Such drivers need to decode the sequence of bytes into strings, and for that they need to pick the appropriate encoding (UTF-8, ISO-8859-1, ...).
The default_encoding
property returns the name of the default encoding used to decode process outputs (as accepted by the str.encode()
method: utf-8
, latin-1
, ...). If it returns binary
, outputs are not decoded and self.output
is set to a Log
instance that holds bytes.
The default implementation for this property returns the encoding
entry from the self.test_env
dict. If there is no such entry, it returns utf-8
(the most commonly used encoding these days).
Spawning subprocesses is so common that this driver class provides a convenience method to do it:
def shell(self, args, cwd=None, env=None, catch_error=True,
analyze_output=True, timeout=None, encoding=None):
This will run a subprocess given a list of command-line arguments (args
); its standard input is redirected to /dev/null
while both its standard output/error streams are collected as a single stream. shell
returns a ProcessResult
instance once the subprocess exitted. ProcessResult
is just a holder for process information: its status
attribute contains the process exit code (an integer) while its out
attribute contains the captured output.
Note that the shell
method also automatically appends a description of the spawned subprocess (arguments, working directory, exit code, output) to the test result log <api_test_result_log>
.
Its other arguments give finer control over process execution:
cwd
Without surprise for people familiar with process handling APIs: this argument controls the directory in which the subprocess is spawned. When left to
None
, the processed is spawned in the working directory.env
Environment variables to pass to the subprocess. If left to
None
, the subprocess inherit the Python interpreter's environment.catch_error
If true (the default),
shell
will check the exit status: if it is 0, nothing happen, however if it is anything else,shell
raises an exception to abort the testcase with a failure (seeclassic_exceptions
for more details). If set to false, nothing special happens for non-0 exit statuses.analyze_output
Whether to append the subprocess output to
self.output
(seeclassic_output_encodings
). This is for convenience in test drivers based on output comparison (seeapi_diff
).timeout
Number of seconds to allow for the subprocess execution: if it lasts longer, the subprocess is aborted and its status code is set to non-zero.
If left to
None
, use instead the timeout that thedefault_process_timeout
property returns. TheClassicTestDriver
implementation for that property returns either thetimeout
entry fromself.test_env
(if present) or 300 seconds (5 minutes). Of course, subclasses are free to override this property if needed.encoding
Name of the encoding used to decode the subprocess output. If left to
None
, use instead the encoding that thedefault_encoding
property returns (seeclassic_output_encodings
). Here, too, the default implementation returns theencoding
entry fromself.test_env
(if present) orutf-8
. Again, subclasses are free to override this property if needed.truncate_logs_threshold
Natural number, threshold to truncate the subprocess output that
shell
logs in thetest result log <api_test_result_log>
. This threshold is interpreted as half the number of output lines allowed before truncation, and 0 means that truncation is disabled. If left toNone
, use the testsuite's--truncate-logs
option.
The common organization for test driver execution has four parts:
- Initialization: make sure input is valid: required files must be present (test program sources, input files), metadata is valid, start a server, and so on.
- Execution: the meat happens here: run the necessary programs, write the necessary files, ...
- Analysis: look at the test output and decide whether the test passed.
- Finalization: free resources, shut down the server, ..
ClassicTestDriver
defines four overridable methods, one for each step: set_up
, run
, analyze
and tear_down
. First, the set_up
method is called, then the run
one and then the analyze
one. So far, any unhandled exception in these methods would prevent the next ones to run. Except for the tear_down
method, which is called no matter what happens as long as the set_up
method was called.
The following example shows how this is useful. Imagine a testsuite for a database server. We want some test drivers only to start the server (leaving the rest to testcases) while we want other test drivers to perform more involved server initialization.
class BaseDriver(ClassicTestDriver):
def set_up(self):
super().set_up()
self.start_server()
def run(self):
pass # ...
def tear_down(self):
self.stop_server()
super().tear_down()
class FixturesDriver(BaseDriver):
def set_up(self):
super(FixturesDriver, self).set_up()
self.install_fixtures()
The install_fixtures()
call has to happen after the start_server()
one, but before the actual test execution (run()
). If initialization, execution and finalization all happened in BaseDriver.run
, it would not be possible for FixturesDriver
to insert the call at the proper place.
Note that ClassicTestDriver
provide valid default implementations for all these methods except run
, which subclasses have to override.
The analyze
method is interesting: its default implementation calls the compute_failures
method, which returns a list of error messages. If that list is empty, it considers that there is no test failure, and thus that the testcase passed. Otherwise, it considers that the test failed. In both cases, it appropriately set the status/message in self.result
and pushes it to the testsuite report.
That means that in practice, test drivers only need to override this compute_failures
method in order to properly analyze test output. For instance, let's consider a test driver whose run
method spawns a supbrocess and must consider that the test succeeds iff the SUCCESS
string appears in the output. The following would do the job:
class FooDriver(ClassicTestDriver):
def run(self):
self.shell(...)
def compute_failures(self):
return (["no match for SUCCESS in output"]
if "SUCCESS" not in self.output
else [])
Deciding whether to skip a testcase, or expecting a test failure are both so common that ClassicTestDriver
provides a mechanism which makes it possible to control testcase execution thanks to metadata in that testcase.
By default, it is based on metadata from the test environment (self.test_env
, i.e. from the test.yaml
file), but each driver can customize this. This mechanism is described extensively in api_control
.
The e3.testsuite.driver.classic
module defines several exceptions that ClassicTestDriver
subclasses can use to control the execution of testcases. These exceptions are expected to be propagated from the set_up
, run
and analyze
methods when appropriate. When they are, this stops the execution of the testcase (next methods are not run). Please refer to api_test_status
for the meaning of test statuses.
TestSkip
Abort the testcase and push a
SKIP
test result.TestAbortWithError
Abort the testcase and push an
ERROR
test result.TestAbortWithFailure
Abort the testcase and push a
FAIL
test result, orXFAIL
if a failure is expected (seeapi_control
).
Long raw text logs can be difficult to read quickly. Light formatting (color, brightness) can help in this area, revealing the structure of text logs. Since it relies on the e3-core
project, e3-testsuite
already has the colorama project in its dependencies.
ClassicTestDriver
subclasses can use self.Fore
and self.Style
attributes as "smart" shortcuts for colorama.Fore
and colorama.Style
: if there is a single chance for text logs to be redirected to a text file (rather than everything to be printed in consoles), colors support is disable and these two attributes yield empty strings instead of the regular console escape sequences.
The shell
method already uses them to format the logging of subprocesses in self.result.log
:
self.result.log += (
self.Style.RESET_ALL + self.Style.BRIGHT
+ "Status code" + self.Style.RESET_ALL
+ ": " + self.Style.DIM + str(p.status) + self.Style.RESET_ALL
)
This will format Status code
in bright style and the status code in dim style if formatting is enabled, and will just return Status code: 0
` without formatting when disabled.
Even though each testcase using a ClassicTestDriver
subclass has a single test fragment, it can be useful for drivers to know which slot
<test_fragment_slot>
they are being run on. The slot is available in the self.slot
driver attribute.