-
-
Notifications
You must be signed in to change notification settings - Fork 281
/
running-tests.texy
239 lines (161 loc) · 9.16 KB
/
running-tests.texy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
Running Tests
*************
.[perex]
The most visible part of Nette Tester is the command-line test runner. It is extremely fast and robust because it starts automatically all tests as separate processes in parallel in multiple threads. It can also run itself in the so-called watch mode.
The Nette Tester test runner is invoked from the command-line. As a parameter, we will pass the test directory. For the current directory just enter a dot:
/--pre .[terminal]
vendor/bin/tester .
\--
When invoked, the test runner will scan the specified directory and all subdirectories and looks for tests, which are the `*.phpt` and `*Test.php` files. It also reads and evaluates their [annotations|test-annotations] to know which ones and how to run them.
It will then execute the tests. For each done test, the runner prints one character to indicate progress:
- <code style="color: #CCC; background-color: #000">.</code> – test passed
- <code style="color: #CCC; background-color: #000">s</code> – test has been skipped
- <code style="color: #FFF; background-color: #900">F</code> – test failed
The output may look like:
/--pre .[terminal]
_____ ___ ___ _____ ___ ___
|_ _/ __)( __/_ _/ __)| _ )
|_| \___ /___) |_| \___ |_|_\ v2.5.2
Note: No php.ini is used.
PHP 8.3.2 (cli) | php -n | 8 threads
........s..........................
<span style="color: #FFF; background-color: #090">OK (35 tests, 1 skipped, 1.7 seconds)</span>
\--
When you run it again, it first runs tests that failed during the previous run, so you'll know right away if you fixed the error.
Tester's exit code is zero if no one fails. Non-zero otherwise.
.[warning]
The Tester runs PHP processes without `php.ini`. More details in the [#Own php.ini] section.
Command-Line Options
====================
We obtain command-line options overview by running the Tester without parameters or with option `-h`:
/--pre .[terminal]
_____ ___ ___ _____ ___ ___
|_ _/ __)( __/_ _/ __)| _ )
|_| \___ /___) |_| \___ |_|_\ v2.5.2
Usage:
tester [options] [<test file> | <directory>]...
Options:
-p <path> Specify PHP interpreter to run (default: php).
-c <path> Look for php.ini file (or look in directory) <path>.
-C Use system-wide php.ini.
-d <key=value>... Define INI entry 'key' with value 'value'.
-s Show information about skipped tests.
--stop-on-fail Stop execution upon the first failure.
-j <num> Run <num> jobs in parallel (default: 8).
-o <console|console-lines|tap|junit|log|none> (e.g. -o junit:output.xml)
Specify one or more output formats with optional file name.
-w | --watch <path> Watch directory.
-i | --info Show tests environment info and exit.
--setup <path> Script for runner setup.
--temp <path> Path to temporary directory. Default by sys_get_temp_dir().
--colors [1|0] Enable or disable colors.
--coverage <path> Generate code coverage report to file.
--coverage-src <path> Path to source code.
-h | --help This help.
\--
-p <path> .[filter]
-------------------
Specifies the PHP binary that will be used to run tests. By default it is `php`.
/--pre .[terminal]
tester -p /home/user/php-7.2.0-beta/php-cgi tests
\--
-c <path> .[filter]
-------------------
Specifies which `php.ini` will be used when running tests. By default, no php.ini is used. See [#Own php.ini] for more information.
-C .[filter]
------------
A system-wide `php.ini` is used. So on UNIX platform, all the `/etc/php/{sapi}/conf.d/*.ini` files too. See [#Own php.ini] section.
-d <key=value> .[filter]
------------------------
Sets the value of the PHP configuration directive for tests. The parameter can be used multiple times.
/--pre .[terminal]
tester -d max_execution_time=20
\--
-s
---
Information about skipped tests will be shown.
--stop-on-fail .[filter]
------------------------
Tester stops testing upon the first failing test.
-j <num> .[filter]
------------------
Tests run in a `<num>` parallel precesses. Default value is 8. If we wish to run tests in series, we use value 1.
-o <console|console-lines|tap|junit|log|none> .[filter]
-------------------------------------------------------
Output format. Default is the console format. You can specify the name of the file into which the output will be written (e.g., `-o junit:output.xml`). The `-o` option can be repeated multiple times to generate multiple formats at once.
- `console`: the same as default, but the ASCII logo is not printed in this case
- `console-lines`: similar to the console, but the result of each test is listed on a separate line with more information
- `tap`: [TAP format |https://en.wikipedia.org/wiki/Test_Anything_Protocol] appropriate for machine processing
- `junit`: JUnit XML format, appropriate for machine processing too
- `log`: Outputs testing progress. All failed, skipped and also successful tests
- `none`: nothing is printed
''-w | --watch <path>'' .[filter]
---------------------------------
The Tester doesn't end after tests are completed but it keeps running and watching PHP files in given directory. When changed, it runs the tests again. Parameter can be used multiple times if we want to monitor multiple directories.
It is handy during refactoring a library or tests debugging.
/--pre .[terminal]
tester --watch src tests
\--
''-i | --info'' .[filter]
-------------------------
It shows information about a test running environment. For example:
/--pre .[terminal]
tester -p /usr/bin/php7.1 -c tests/php.ini --info
<span style="color: #0F0">PHP binary:</span>
/usr/bin/php7.1
<span style="color: #0F0">PHP version:</span>
7.1.7-1+0~20170711133844.5+jessie~1.gbp5284f4 (cli)
<span style="color: #0F0">Code coverage engines:</span>
(not available)
<span style="color: #0F0">Loaded php.ini files:</span>
/var/www/dev/demo/tests/php.ini
<span style="color: #0F0">PHP temporary directory:</span>
/tmp
<span style="color: #0F0">Loaded extensions:</span>
Core, ctype, date, dom, ereg, fileinfo, filter, hash, ...
\--
--setup <path> .[filter]
------------------------
The Tester loads given PHP script on start. Variable `Tester\Runner\Runner $runner` is available in it. Let's assume file `tests/runner-setup.php`:
```php
$runner->outputHandlers[] = new MyOutputHandler;
```
and we run the Tester:
/--pre .[terminal]
tester --setup tests/runner-setup.php tests
\--
--temp <path> .[filter]
-----------------------
Sets a path to directory for temporary files of Tester. Default value is returned by `sys_get_temp_dir()`. When default value is not valid, you will be noticed.
If we are not sure which directory is used, we can run Tester with the `--info` parameter.
--colors 1|0 .[filter]
----------------------
The Tester detects a color-able terminal in default and colorizes its output. This option is over the autodetection. We can set coloring globally by a system environment variable `NETTE_TESTER_COLORS`.
--coverage <path> .[filter]
---------------------------
Tester will generate a report with overview how much is the source code coveraged by tests. This option requires PHP extension [Xdebug |https://xdebug.org/] or [PCOV |https://github.com/krakjoe/pcov] enabled, or PHP 7 with the PHPDBG SAPI, which is faster. The destination file extension determines the format of the content. HTML or Clover XML.
/--pre .[terminal]
tester tests --coverage coverage.html # HTML report
tester tests --coverage coverage.xml # Clover XML report
\--
Priority to choose collecting mechanism is following:
1) PCOV
2) PHPDBG
3) Xdebug
Extensive tests may fail during run by PHPDBG due to memory exhaustion. Coverage data collecting is memory consuming operation. In that case, calling the `Tester\CodeCoverage\Collector::flush()` inside a test may help. It will flush collected data into file and frees memory. When data collection is not in progress, or Xdebug is used, calling has no effect.
"An example of HTML report":https://files.nette.org/tester/coverage.html with code coverage.
--coverage-src <path> .[filter]
-------------------------------
We use it with the option `--coverage` simultaneously. The `<path>` is a path to source code for which we generate the report. It can be used repeatedly.
Own php.ini
===========
The Tester runs PHP processes with `-n` option, which means no `php.ini` is loaded (not even that from `/etc/php/conf.d/*.ini` in UNIX). It ensures the same environment for the tests run, but it also deactivates all the external PHP extensions commonly loaded by system PHP.
If you want to keep system configuration, use the `-C` parameter.
If you need some extensions or some special INI settings, we recommend to create own `php.ini` file and distribute it among the tests. Then we run Tester with `-c` option, e.g. `tester -c tests/php.ini`. The INI file may look like:
```ini
[PHP]
extension=php_pdo_mysql.dll
extension=php_pdo_pgsql.dll
memory_limit=512M
```
Running the Tester with a system `php.ini` in UNIX, e.g. `tester -c /etc/php/cgi/php.ini`, does not load other INI from `/etc/php/conf.d/*.ini`. That's the PHP behavior, not the Tester's.