-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
/
nonpython.rst
102 lines (73 loc) · 3.73 KB
/
nonpython.rst
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
.. _`non-python tests`:
Working with non-python tests
====================================================
.. _`yaml plugin`:
A basic example for specifying tests in Yaml files
--------------------------------------------------------------
.. _`pytest-yamlwsgi`: https://pypi.org/project/pytest-yamlwsgi/
Here is an example ``conftest.py`` (extracted from Ali Afshar's special purpose `pytest-yamlwsgi`_ plugin). This ``conftest.py`` will collect ``test*.yaml`` files and will execute the yaml-formatted content as custom tests:
.. include:: nonpython/conftest.py
:literal:
You can create a simple example file:
.. include:: nonpython/test_simple.yaml
:literal:
and if you installed :pypi:`PyYAML` or a compatible YAML-parser you can
now execute the test specification:
.. code-block:: pytest
nonpython $ pytest test_simple.yaml
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y
rootdir: /home/sweet/project/nonpython
collected 2 items
test_simple.yaml F. [100%]
================================= FAILURES =================================
______________________________ usecase: hello ______________________________
usecase execution failed
spec failed: 'some': 'other'
no further details known at this point.
========================= short test summary info ==========================
FAILED test_simple.yaml::hello
======================= 1 failed, 1 passed in 0.12s ========================
.. regendoc:wipe
You get one dot for the passing ``sub1: sub1`` check and one failure.
Obviously in the above ``conftest.py`` you'll want to implement a more
interesting interpretation of the yaml-values. You can easily write
your own domain specific testing language this way.
.. note::
``repr_failure(excinfo)`` is called for representing test failures.
If you create custom collection nodes you can return an error
representation string of your choice. It
will be reported as a (red) string.
``reportinfo()`` is used for representing the test location and is also
consulted when reporting in ``verbose`` mode:
.. code-block:: pytest
nonpython $ pytest -v
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y -- $PYTHON_PREFIX/bin/python
cachedir: .pytest_cache
rootdir: /home/sweet/project/nonpython
collecting ... collected 2 items
test_simple.yaml::hello FAILED [ 50%]
test_simple.yaml::ok PASSED [100%]
================================= FAILURES =================================
______________________________ usecase: hello ______________________________
usecase execution failed
spec failed: 'some': 'other'
no further details known at this point.
========================= short test summary info ==========================
FAILED test_simple.yaml::hello
======================= 1 failed, 1 passed in 0.12s ========================
.. regendoc:wipe
While developing your custom test collection and execution it's also
interesting to just look at the collection tree:
.. code-block:: pytest
nonpython $ pytest --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y
rootdir: /home/sweet/project/nonpython
collected 2 items
<Package nonpython>
<YamlFile test_simple.yaml>
<YamlItem hello>
<YamlItem ok>
======================== 2 tests collected in 0.12s ========================