Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 308 lines (253 sloc) 11.57 kb
08fead5 Added README.TESTING
Yasuo Ohgaki authored
1 [IMPORTANT NOTICE]
2 ------------------
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
3 Do _not_ ask to developers why some or all tests are failed under
08fead5 Added README.TESTING
Yasuo Ohgaki authored
4 your environment! Let us know if you find why it fails. Thank you.
5
6
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
7 [Testing Basics]
8 ----------------
9 To execute test scripts, you must build PHP with some SAPI, then you
10 type "make test" to execute all or some test scripts saved under
08fead5 Added README.TESTING
Yasuo Ohgaki authored
11 "tests" directory under source root directory.
12
13 Usage:
14 make test
15
16 "make test" basically executes "run-tests.php" script
8bfb6c5 Some english fixes
Melvyn Sopacua authored
17 under the source root (parallel builds will not work). Therefore you
18 can execute the script as follows:
08fead5 Added README.TESTING
Yasuo Ohgaki authored
19
8bfb6c5 Some english fixes
Melvyn Sopacua authored
20 TEST_PHP_EXECUTABLE=sapi/cli/php \
21 sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
08fead5 Added README.TESTING
Yasuo Ohgaki authored
22
23
24 [Which "php" executable "make test" look for]
25 ---------------------------------------------
96f7bcc Update info and synch example test.
Marcus Boerger authored
26 You must use TEST_PHP_EXECUTABLE environment variable to explicitly
27 select the php executable to be used to run the tests. That can either
28 be the CLI or CGI executable.
29
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
30 "make test" executes "run-tests.php" script with "php" binary. Some
31 test scripts such as session must be executed by CGI SAPI. Therefore,
32 you must build PHP with CGI SAPI to perform all tests.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
33
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
34 NOTE: PHP binary executing "run-tests.php" and php binary used for
35 executing test scripts may differ. If you use different PHP binary for
36 executing "run-tests.php" script, you may get errors.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
37
38
39 [Which php.ini is used]
40 -----------------------
8bfb6c5 Some english fixes
Melvyn Sopacua authored
41 "make test" uses the same php.ini file as it would once installed.
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
42 The tests have been written to be independent of that php.ini file,
8bfb6c5 Some english fixes
Melvyn Sopacua authored
43 so if you find a test that is affected by a setting, please report
44 this, so we can address the issue.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
45
46
47 [Which test scripts are executed]
48 ---------------------------------
8bfb6c5 Some english fixes
Melvyn Sopacua authored
49 "run-tests.php" ("make test"), without any arguments executes all
c95256b no more looking
Melvyn Sopacua authored
50 test scripts by extracting all directories named "tests"
8bfb6c5 Some english fixes
Melvyn Sopacua authored
51 from the source root and any subdirectories below. If there are files,
52 which have a "phpt" extension, "run-tests.php" looks at the sections
53 in these files, determines whether it should run it, by evaluating
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
54 the 'SKIP' section. If the test is eligible for execution, the 'FILE'
096d8da additions
Marcus Boerger authored
55 section is extracted into a ".php" file (with the same name besides
56 the extension) and gets executed.
8bfb6c5 Some english fixes
Melvyn Sopacua authored
57 When an argument is given or TESTS environment variable is set, the
58 GLOB is expanded by the shell and any file with extension "*.phpt" is
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
59 regarded as a test file.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
60
96f7bcc Update info and synch example test.
Marcus Boerger authored
61 Tester can easily execute tests selectively with as follows.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
62
96f7bcc Update info and synch example test.
Marcus Boerger authored
63 Examples:
8bfb6c5 Some english fixes
Melvyn Sopacua authored
64 ./sapi/cli/php run-tests.php ext/mbstring/*
65 ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
08fead5 Added README.TESTING
Yasuo Ohgaki authored
66
67
68 [Test results]
69 --------------
70 Test results are printed to standard output. If there is a failed test,
8bfb6c5 Some english fixes
Melvyn Sopacua authored
71 the "run-tests.php" script saves the result, the expected result and the
72 code executed to the test script directory. For example, if
73 ext/myext/tests/myext.phpt fails to pass, the following files are created:
08fead5 Added README.TESTING
Yasuo Ohgaki authored
74
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
75 ext/myext/tests/myext.php - actual test file executed
96f7bcc Update info and synch example test.
Marcus Boerger authored
76 ext/myext/tests/myext.log - log of test execution (L)
77 ext/myext/tests/myext.exp - expected output (E)
78 ext/myext/tests/myext.out - output from test script (O)
79 ext/myext/tests/myext.diff - diff of .out and .exp (D)
08fead5 Added README.TESTING
Yasuo Ohgaki authored
80
8bfb6c5 Some english fixes
Melvyn Sopacua authored
81 Failed tests are always bugs. Either the test is bugged or not considering
82 factors applying to the tester's environment, or there is a bug in PHP.
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
83 If this is a known bug, we strive to provide bug numbers, in either the
8bfb6c5 Some english fixes
Melvyn Sopacua authored
84 test name or the file name. You can check the status of such a bug, by
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
85 going to: http://bugs.php.net/12345 where 12345 is the bug number.
86 For clarity and automated processing, bug numbers are prefixed by a hash
87 sign '#' in test names and/or test cases are named bug12345.phpt.
96f7bcc Update info and synch example test.
Marcus Boerger authored
88
89 NOTE: The files generated by tests can be selected by setting the
90 environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
91 generated use the character in brackets as shown above (default is LEOD).
9da3c88 update and correct
Marcus Boerger authored
92 The php file will be generated always.
96f7bcc Update info and synch example test.
Marcus Boerger authored
93
94 NOTE: You can set environment variable TEST_PHP_DETAILED to enable
95 detailed test information.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
96
8bfb6c5 Some english fixes
Melvyn Sopacua authored
97 [Automated testing]
98 If you like to keep up to speed, with latest developments and quality
99 assurance, setting the environment variable NO_INTERACTION to 1, will not
100 prompt the tester for any user input.
101
102 Normally, the exit status of "make test" is zero, regardless of the results
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
103 of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
2322917 statis->status
Melvyn Sopacua authored
104 and "make test" will set the exit status ("$?") to non-zero, when an
105 individual test has failed.
8bfb6c5 Some english fixes
Melvyn Sopacua authored
106
c1caad6 Provide an example cron script, to invite people to test.
Melvyn Sopacua authored
107 Example script to be run by cron(1):
108 ========== qa-test.sh =============
109 #!/bin/sh
110
111 CO_DIR=$HOME/cvs/php4
112 MYMAIL=qa-test@domain.com
113 TMPDIR=/var/tmp
114 TODAY=`date +"%Y%m%d"`
115
116 # Make sure compilation enviroment is correct
117 CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
118 export MAKE=gmake
119 export CC=gcc
120
121 # Set test environment
122 export NO_INTERACTIVITY=1
123 export REPORT_EXIT_STATUS=1
124
125 cd $CO_DIR
126 cvs update . >>$TMPDIR/phpqatest.$TODAY
127 ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
128 $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
129 if test $? -gt 0
130 then
131 cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
132 fi
133 ========== end of qa-test.sh =============
134
8bfb6c5 Some english fixes
Melvyn Sopacua authored
135 NOTE: the exit status of run-tests.php will be 1 when
136 REPORT_EXIT_STATUS is set. The result of "make test" may be higher
137 than that. At present, gmake 3.79.1 returns 2, so it is
138 advised to test for non-zero, rather then a specific value.
139
08fead5 Added README.TESTING
Yasuo Ohgaki authored
140
141 [Creating new test files]
142 -------------------------
9307fa7 update
Marcus Boerger authored
143 Writing test file is very easy if you are used to PHP.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
144
9307fa7 update
Marcus Boerger authored
145 Here is an actual test file from standard module.
146
147 ===== ext/standard/tests/strings/strtr.phpt =======
08fead5 Added README.TESTING
Yasuo Ohgaki authored
148 --TEST--
9307fa7 update
Marcus Boerger authored
149 strtr() function
08fead5 Added README.TESTING
Yasuo Ohgaki authored
150 --FILE--
9307fa7 update
Marcus Boerger authored
151 <?php
152 /* Do not change this test it is a REATME.TESTING example. */
153 $trans = array("hello"=>"hi", "hi"=>"hello", "a"=>"A", "world"=>"planet");
154 var_dump(strtr("# hi all, I said hello world! #", $trans));
155 ?>
08fead5 Added README.TESTING
Yasuo Ohgaki authored
156 --EXPECT--
9307fa7 update
Marcus Boerger authored
157 string(32) "# hello All, I sAid hi planet! #"
158 ===== ext/standard/tests/strings/strtr.phpt =======
08fead5 Added README.TESTING
Yasuo Ohgaki authored
159
9307fa7 update
Marcus Boerger authored
160 As you can see the file is devided into several sections. Below is a
161 list of all possible sections:
96f7bcc Update info and synch example test.
Marcus Boerger authored
162
cc4c063 -introduce EXPECTREGEX
Marcus Boerger authored
163 "--TEST--" is title of the test (required).
164 "--SKIPIF--" is condition when to skip this test (optional).
165 "--POST--" is POST variable passed to test script (optional).
166 "--GET--" is GET variable passed to test script (optional).
167 "--INI--" each line contains an ini setting e.g. foo=bar (optional).
168 "--FILE--" is the test script (required).
169 "--EXPECT--" is the expected output from the test script (required).
170 "--EXPECTF--" is an alternative of --EXPECT--. The difference is that
171 this form uses sscanf for output validation (alternative).
172 "--EXPECTREGEX--" is an alternative of --EXPECT--. This form allows the tester
173 to specify the result in a regular expression (alternative).
9307fa7 update
Marcus Boerger authored
174
175 A test must at least contain the sections TEST, FILE and either EXPECT
176 or EXPECTF. When a test is called run-test.php takes the name from the
177 TEST section and writes the FILE section into a ".php" file with the
178 same name as the ".phpt" file. This ".php" file will then be executed
179 and its output compared to the contents of the EXPECT section. It is a
180 good idea to generate output with var_dump() calls.
181
182 /ext/standard/tests/strings/str_shuffle.phpt is a good example for using
183 EXPECTF instead of EXPECT. From time to time the algorithm used for shuffle
9da3c88 update and correct
Marcus Boerger authored
184 changed and sometimes the machine used to execute the code has influence
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
185 on the result of shuffle. But it always returns a three character string
9307fa7 update
Marcus Boerger authored
186 detectable by %s. Other scan-able forms are %i for integers, %d for numbers
cc4c063 -introduce EXPECTREGEX
Marcus Boerger authored
187 only, %f for floating point values, %c for single characters and %x for
188 hexadecimal values.
9da3c88 update and correct
Marcus Boerger authored
189
9307fa7 update
Marcus Boerger authored
190 ==== /ext/standard/tests/strings/str_shuffle.phpt ===
9da3c88 update and correct
Marcus Boerger authored
191 --TEST--
192 Testing str_shuffle.
193 --FILE--
194 <?php
9307fa7 update
Marcus Boerger authored
195 /* Do not change this test it is a REATME.TESTING example. */
9da3c88 update and correct
Marcus Boerger authored
196 $s = '123';
197 var_dump(str_shuffle($s));
198 var_dump($s);
199 ?>
200 --EXPECTF--
201 string(3) %s
202 string(3) "123"
9307fa7 update
Marcus Boerger authored
203 ==== end of /ext/standard/tests/strings/str_shuffle.phpt ===
204
cc4c063 -introduce EXPECTREGEX
Marcus Boerger authored
205 /ext/standard/tests/strings/strings001.phpt is a good example for using
6857e10 Only in EXPECTREGEX
Marcus Boerger authored
206 EXPECTREGEX instead of EXPECT. This test also shows that in EXPECTREGEX
207 some characters need to be escaped since otherwise they would be
208 interpreted as a regular expression.
cc4c063 -introduce EXPECTREGEX
Marcus Boerger authored
209
210 ==== /ext/standard/tests/strings/strings001.phpt ===
211 --TEST--
212 Test whether strstr() and strrchr() are binary safe.
213 --FILE--
214 <?php
215 /* Do not change this test it is a REATME.TESTING example. */
216 $s = "alabala nica".chr(0)."turska panica";
217 var_dump(strstr($s, "nic"));
218 var_dump(strrchr($s," nic"));
219 ?>
220 --EXPECTREGEX--
221 string\(18\) \"nica\x00turska panica\"
222 string\(19\) \" nica\x00turska panica\"
223 ==== end of /ext/standard/tests/strings/strings001.phpt ===
224
9307fa7 update
Marcus Boerger authored
225 Some tests depend on modules or functions available only in certain versions
226 or they even require minimum version of php or zend. These tests should be
227 skipped when the requirement cannot be fullfilled. To achieve this you can
228 use the SKIPIF section. To tell run-test.php that your test should be skipped
229 the SKIPIF section must print out the word "skip" followed by a reason why
230 the test should skip.
231
232 ==== /ext/exif/tests/exif005.phpt ===
233 --TEST--
234 Check for exif_read_data, unusual IFD start
235 --SKIPIF--
236 <?php
237 if (!extension_loaded('exif')) print 'skip exif extension not available';
238 ?>
239 --FILE--
240 <?php
241 /* Do not change this test it is a REATME.TESTING example.
242 * test5.jpg is a 1*1 image that contains an Exif section with ifd = 00000009h
243 */
244 $image = exif_read_data('./ext/exif/tests/test5.jpg','',true,false);
245 var_dump($image['IFD0']);
246 ?>
247 --EXPECT--
248 array(2) {
249 ["ImageDescription"]=>
250 string(11) "Ifd00000009"
251 ["DateTime"]=>
252 string(19) "2002:10:18 20:06:00"
253 }
254 ==== end of /ext/exif/tests/exif005.phpt ===
255
256 Test script and SKIPIF code should be directly written into *.phpt. However,
257 it is recommended to use include files when more test scripts depend on the
258 same SKIPIF code or when certain test files need the same values for some
259 input. But no file used by any test should have one of the following
260 extensions: ".php", ".log", ".exp", ".out" or ".diff".
261
262 Tests should be named according to the following list:
263
264 Class: Name: Example:
265 Tests for bugs bug<bugid>.phpt bug17123.phpt
266 Tests for functions <functionname>.phpt dba_open.phpt
267 General tests for extensions <extname><no>.phpt dba3.phpt
268
269 When you use an include file for the SKIPIF section it should be named
270 "skipif.inc" and an include file used in the FILE section of many tests
271 should be named "test.inc".
9da3c88 update and correct
Marcus Boerger authored
272
fde7dff Enough discussion on display_erros it seems
Marcus Boerger authored
273 NOTE: All tests should run correctly with error_reporting(E_ALL) and
274 display_errors=1. This is the default when called from run-test.php.
275 If you have a good reason for lowering the error reporting, use --INI--
276 section and comment this in your testcode.
277
278 NOTE: If your test intentionally generates a PHP warning message use
279 $php_errormsg variable, which you can then output. This will result
280 in a consistent error message output across all platforms and PHP
281 configurations, preventing your test from failing due inconsistencies
b262301 Added note about error reporting in tests.
Ilia Alshanetsky authored
282 in the error message content.
283
08fead5 Added README.TESTING
Yasuo Ohgaki authored
284 [How to help us]
285 ----------------
286 If you find bug in PHP, you can submit bug report AND test script
287 for us. You don't have to write complete script, just give us test
288 script with following format. Please test the script and make sure
289 you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
290 submit.
291
292 <?php
293 /*
294 Bug #12345
295 substr() bug. Do not return expected string.
296
297 ACTUAL OUTPUT
298 XYXA
299
300 EXPECTED OUTPUT
301 ABCD
302 */
303
304 $str = "XYZABCD";
305 echo substr($str,3,7);
306
307 ?>
Something went wrong with that request. Please try again.