testreporting.rst

The tracability ===========

The events

The execution of a test is divided into events, all of these events are stored and can be viewed afterwards. An event can represent: - an action performed by the test framework - an action performed by the test - a data item received by the system to be tested or checked. - data to send to the system to test or control.

Event execution makes it possible to have robust tests thanks to the definition of observation intervals. The approach is to write the tests with the following formalism: - I perform an action in my test. - During a given interval, I look and compare all the events received with an expected. - I decide on the next action * after receiving the event I was waiting for * or when the observation interval is over.

During the execution of a test, the framework captures all the events generated by the tested or piloted system. The events are then converted and stored in a message called template.

A template is split into one or more layers. A layer is defined by a set of key/value. The value of a layer can be another layer too.

Creating a template ~~~~~~~~~~~~~

Creating a template can be done using the TestTemplates test framework.

This template indicates that the event should contain:

• a layer called response and containing the key` code` and msg
• the key code must be strictly equal to the value 500
• the msg key must be strictly equal to the text hello world.

Example of an expected message visible from the graphical client:

The operators

Operators are available to facilitate comparison of the models received.

Name	Description
Any	Accepts all values ​​
Contains	Check if the string of characters contains
Endswith	Check if the string ends with
Startswith	Check if the string starts with
GreaterThan	Check if an integer is larger than
LowerThan	Check if an integer is smaller than
RegEx	Check if the string matches the regular expression

Sample template using comparison operators:

tpl = TestTemplates.TemplateMessage()
layer = TestTemplates.TemplateLayer(name='response')
layer.addKey(name='code', data=TestOperators.LowerThan(x=500)))
layer.addKey(name='msg', data=TestOperators.Contains(x="hello"))
tpl.addLayer(layer=layer) 

This template indicates that the event should contain:

• a layer called response and containing the key` code` and msg
• the code key must be less than 500
• the msg key must contain the text hello.

Visualization

The client can graphically display the comparison made by the framework.

Definition of the color code:

Green	Perfect match between the value received and expected
Red	The value received does not correspond to the expected value
Yellow	The expected value has not been verified

Test reports

After each run of a test, the framework automatically generates the associated test reports.

There are 2 type reports:

• An advanced report
• A basic report (accessible by default from the graphical client)

The reports are accessible from the client, the web interface or from the API.

Advanced report ~~~~~~~~~~~~~~

The advanced report displays information such as:

• the execution time of each test case
• the complete description of the test steps.
• performance statistics.
• the test parameters.

It is possible to display variables in the test report by prefixing the variables:

• SUT_ Variables describing the version of the system to be tested or piloted
• DATA_ Variables describing specific data
• USER_ User variables

This feature can be useful for increasing the level of traceability in reports.

Basic report

The basic report summarizes the result of all test cases and reports.

Color code:

Green	The test case is valid
Red	The test case is in error
Orange	The result of the test case is not determined
Gray	The test case was not executed

Tip

You must click on the test cases to display the steps.

Note

The messages displayed by the test with the Trace (self) .info () function are available in the report by clicking on the [logs details] link.

Errors are also displayed by clicking on the [errors details] link.

The logs ----

The framework allows you to save logs while running a test and make them available quickly to the uses. All additional logs are zipped and accessible from the client or the API.

Note

For more details, read the chapter The fundamentals >> Data.