The pykiso-pytest plugin allows to run pykiso test suites with pytest without any migration effort. Simply run pytest my_test_configuration_file.yaml
and enjoy!
Still, to benefit from all of pytest's features, your existing test suites can be migrated from the pykiso (unittest) style to the pytest style.
Let us take as example the following test configuration file named example.yaml
:
auxiliaries:
aux1:
connectors:
com: chan1
config: null
type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
aux2:
connectors:
com: chan2
type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
connectors:
chan1:
config: null
type: pykiso.lib.connectors.cc_example:CCExample
chan2:
type: pykiso.lib.connectors.cc_example:CCExample
test_suite_list:
- suite_dir: ./
test_filter_pattern: 'test_suite.py'
test_suite_id: 1
Along with the following test case as a python module named test_suite.py
, located in the same folder as the test configuration file:
import pykiso
from pykiso.auxiliaries import aux1, aux2
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteSetup(pykiso.BasicTestSuiteSetup):
def test_suite_setUp(self):
pass
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
def test_suite_tearDown(self):
pass
@pykiso.define_test_parameters(suite_id=1, case_id=1, aux_list=[aux1], tag={"variant": ["v2"]})
class MyTest1(pykiso.BasicTest):
def setUp(self):
self.side_effect = iter([False, True])
@pykiso.retry_test_case(max_try=2)
def test_run(self):
# the retry feature is also available out of the box
self.assertTrue(next(self.side_effect))
@pykiso.define_test_parameters(suite_id=1, case_id=2, aux_list=[aux2], tag={"variant": ["v1"]})
class MyTest2(pykiso.BasicTest):
def test_run(self):
self.assertTrue(aux2.is_instance)
Then, executing pytest example.yaml -v
will produce the following output:
One of Pytest's main features are test fixtures. These allow you to create a context for your test cases, by specifying a test setup, teardown or more generally by initializing resources that are then provided to the test cases.
Pykiso's pytest plugin uses fixtures to provide test auxiliaries to test cases. Similarly to the importable auxiliary aliases in pykiso, the plugin will create fixtures available under this alias.
For example, taking as reference the previous test configuration file and pykiso test suite, the pykiso test case MyTest2
could be rewritten to:
@pytest.mark.tags(variant=["var1"])
def test_mytest2(aux2):
assert aux2.is_instance == True
Just like pykiso, an auxiliary will be started as soon as it is used for the first time. It will then keep running until the end of the test session. To change this behaviour, please refer to pytest_auxiliary_scope.
Note
When writing test cases with pytest, always use plain assert
statements instead of e.g. assertTrue
. Otherwise pytest does not provide assertion introspection.
In order to select a subset of the tests to run, 2 options are available:
- Use pytest's builtin option
-k
. By modifying the previous command to e.g.
pytest example.yaml -v -k MyTest1
, onlyMyTest1
will be run. - Use pykiso's test tags feature. By modifying the previous command to e.g.
pytest example.yaml -v --tags variant=v1
, onlyMyTest1
will be skipped
as it is only meant for a variant tagv2
. For more information regarding
this pykiso feature, please refer totest_tags
.
test_requirements
is also supported by the pykiso pytest plugin:
@pykiso.define_test_parameters(test_ids={"Variant1": ["Requirement123"]})
class MyTest2(pykiso.BasicTest):
def test_run(self):
self.assertTrue(aux2.is_instance)
@pytest.mark.test_ids(Variant1=["Requirement123"])
def test_mytest2(aux2):
assert aux2.is_instance == True
In order to generate a JUnit report for your test session, simply use pytest's built-in option:
pytest ./example.yaml --junit-xml=./report.xml
As stated previously, the test auxiliary fixtures that are generated from the test configuration file as scoped session-widely.
However, this can be changed by setting the value auxiliary_scope
to one of the following scope values:
function
: an auxiliary will be stopped and restarted between each test caseclass
: an auxiliary will be stopped and restarted between each test class containing test casesmodule
: an auxiliary will be stopped and restarted between each test module
This value only needs to be added to the pytest configuration file (pytest.ini
or pyproject.toml
). For more information refer to the pytest documentation.
Pytest allows to 'overwrite' existing features in order to customize then. The fixture only needs to have the same name as the auxiliary defined in your YAML configuration file and will wrap the default behaviour of the auxiliary fixture.
Depending on your use-case, this can be done directly within your test module or within pytest's conftest.py file.
Consider a test module were we would need a CommunicationAuxiliary <communication_auxiliary>
named my_aux
within the test configuration file to send data over some communication protocol. At startup the auxiliary has to send hello
and at teardown goodbye
.
With fixtures, this can easily be achieved with:
import pytest
@pytest.fixture(scope="module")
def my_aux(my_aux):
# my_aux.create_instance() has been called from the base my_aux fixture
my_aux.send_message(b'hello')
yield my_aux
my_aux.send_message(b'goodbye')
# my_aux.delete_instance() will now be called from the base my_aux fixture
Warning
The customized auxiliary fixture cannot have a higher scope than the wrapping one.
The pykiso pytest plugin adds two hooks for the startup and teardown of all test auxiliaries: :pypykiso.pytest.hooks.pytest_auxiliary_start
and :pypykiso.pytest.hooks.pytest_auxiliary_stop
. In very specific use cases, these hooks can be implemented within a conftest.py
file. The execution of registered hook functions will stop when one hook function returns a value different from None
.
For more information about pytest's hook functions, please refer to Writing hook functions <https://docs.pytest.org/en/7.1.x/how-to/writing_hook_functions.html>.
This section aims to provide a non-exhaustive list of potentially interesting features provided by pytest and by 3rd party pytest plugins:
Feature | Plugin | Option |
---|---|---|
Stop test execution at the first failure | native | -x, --exitfirst |
Re-run the last failed tests | native | --lf, --last-failed |
Enter the Python debugger (pdb) on test failure | native | -s --pdb |
Enter the Python debugger (pdb) on test start | native | -s --trace |
Use another Python debugger (here pudb) | native | --pdbcls=pudb.debugger:Debugger |
Generate an HTML test report | pytest-html | --html=report.html --self-contained-html |
Repeating one or multiple tests for a certain amount of times | pytest-repeat | --count=N |
Rerun failing tests for a maximum amount of times | pytest-rerunfailures | --reruns=N |