Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 180 lines (141 sloc) 6.509 kb
08fead5 Added README.TESTING
Yasuo Ohgaki authored
1 [IMPORTANT NOTICE]
2 ------------------
702ca7f be a bit more explanatory ...
Hartmut Holzgraefe authored
3 Failed tests usualy indicate a problem with your local system setup
4 and not within PHP itself (at least for official PHP release versions).
5 You may decide to automaticaly submit a test summary to our QA workflow
6 at the end of a test run.
7 Please do *not* submit a failed test as a bug or ask for help on why
8 it failed on your system without providing substantial backup information
9 on *why* the test failed on your special setup. Thank you :-)
08fead5 Added README.TESTING
Yasuo Ohgaki authored
10
11
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
12 [Testing Basics]
13 ----------------
702ca7f be a bit more explanatory ...
Hartmut Holzgraefe authored
14 The easiest way to test your PHP build is to run "make test" from the
15 command line after successfully compiling. This will run the complete
16 tests for all enabled functionalities and extensions using the PHP
17 CLI binary (
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
18 To execute test scripts, you must build PHP with some SAPI, then you
19 type "make test" to execute all or some test scripts saved under
08fead5 Added README.TESTING
Yasuo Ohgaki authored
20 "tests" directory under source root directory.
21
22 Usage:
23 make test
24
25 "make test" basically executes "run-tests.php" script
8bfb6c5 Some english fixes
Melvyn Sopacua authored
26 under the source root (parallel builds will not work). Therefore you
27 can execute the script as follows:
08fead5 Added README.TESTING
Yasuo Ohgaki authored
28
8bfb6c5 Some english fixes
Melvyn Sopacua authored
29 TEST_PHP_EXECUTABLE=sapi/cli/php \
30 sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
08fead5 Added README.TESTING
Yasuo Ohgaki authored
31
32
33 [Which "php" executable "make test" look for]
34 ---------------------------------------------
96f7bcc Update info and synch example test.
Marcus Boerger authored
35 You must use TEST_PHP_EXECUTABLE environment variable to explicitly
36 select the php executable to be used to run the tests. That can either
37 be the CLI or CGI executable.
38
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
39 "make test" executes "run-tests.php" script with "php" binary. Some
40 test scripts such as session must be executed by CGI SAPI. Therefore,
41 you must build PHP with CGI SAPI to perform all tests.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
42
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
43 NOTE: PHP binary executing "run-tests.php" and php binary used for
44 executing test scripts may differ. If you use different PHP binary for
45 executing "run-tests.php" script, you may get errors.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
46
47
48 [Which php.ini is used]
49 -----------------------
8bfb6c5 Some english fixes
Melvyn Sopacua authored
50 "make test" uses the same php.ini file as it would once installed.
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
51 The tests have been written to be independent of that php.ini file,
8bfb6c5 Some english fixes
Melvyn Sopacua authored
52 so if you find a test that is affected by a setting, please report
53 this, so we can address the issue.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
54
55
56 [Which test scripts are executed]
57 ---------------------------------
8bfb6c5 Some english fixes
Melvyn Sopacua authored
58 "run-tests.php" ("make test"), without any arguments executes all
c95256b no more looking
Melvyn Sopacua authored
59 test scripts by extracting all directories named "tests"
8bfb6c5 Some english fixes
Melvyn Sopacua authored
60 from the source root and any subdirectories below. If there are files,
61 which have a "phpt" extension, "run-tests.php" looks at the sections
62 in these files, determines whether it should run it, by evaluating
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
63 the 'SKIP' section. If the test is eligible for execution, the 'FILE'
096d8da additions
Marcus Boerger authored
64 section is extracted into a ".php" file (with the same name besides
65 the extension) and gets executed.
8bfb6c5 Some english fixes
Melvyn Sopacua authored
66 When an argument is given or TESTS environment variable is set, the
67 GLOB is expanded by the shell and any file with extension "*.phpt" is
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
68 regarded as a test file.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
69
96f7bcc Update info and synch example test.
Marcus Boerger authored
70 Tester can easily execute tests selectively with as follows.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
71
96f7bcc Update info and synch example test.
Marcus Boerger authored
72 Examples:
8bfb6c5 Some english fixes
Melvyn Sopacua authored
73 ./sapi/cli/php run-tests.php ext/mbstring/*
74 ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
08fead5 Added README.TESTING
Yasuo Ohgaki authored
75
76
77 [Test results]
78 --------------
79 Test results are printed to standard output. If there is a failed test,
8bfb6c5 Some english fixes
Melvyn Sopacua authored
80 the "run-tests.php" script saves the result, the expected result and the
81 code executed to the test script directory. For example, if
82 ext/myext/tests/myext.phpt fails to pass, the following files are created:
08fead5 Added README.TESTING
Yasuo Ohgaki authored
83
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
84 ext/myext/tests/myext.php - actual test file executed
96f7bcc Update info and synch example test.
Marcus Boerger authored
85 ext/myext/tests/myext.log - log of test execution (L)
86 ext/myext/tests/myext.exp - expected output (E)
87 ext/myext/tests/myext.out - output from test script (O)
88 ext/myext/tests/myext.diff - diff of .out and .exp (D)
08fead5 Added README.TESTING
Yasuo Ohgaki authored
89
8bfb6c5 Some english fixes
Melvyn Sopacua authored
90 Failed tests are always bugs. Either the test is bugged or not considering
91 factors applying to the tester's environment, or there is a bug in PHP.
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
92 If this is a known bug, we strive to provide bug numbers, in either the
8bfb6c5 Some english fixes
Melvyn Sopacua authored
93 test name or the file name. You can check the status of such a bug, by
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
94 going to: http://bugs.php.net/12345 where 12345 is the bug number.
95 For clarity and automated processing, bug numbers are prefixed by a hash
96 sign '#' in test names and/or test cases are named bug12345.phpt.
96f7bcc Update info and synch example test.
Marcus Boerger authored
97
98 NOTE: The files generated by tests can be selected by setting the
99 environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
100 generated use the character in brackets as shown above (default is LEOD).
9da3c88 update and correct
Marcus Boerger authored
101 The php file will be generated always.
96f7bcc Update info and synch example test.
Marcus Boerger authored
102
103 NOTE: You can set environment variable TEST_PHP_DETAILED to enable
104 detailed test information.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
105
8bfb6c5 Some english fixes
Melvyn Sopacua authored
106 [Automated testing]
107 If you like to keep up to speed, with latest developments and quality
108 assurance, setting the environment variable NO_INTERACTION to 1, will not
109 prompt the tester for any user input.
110
111 Normally, the exit status of "make test" is zero, regardless of the results
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
112 of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
2322917 statis->status
Melvyn Sopacua authored
113 and "make test" will set the exit status ("$?") to non-zero, when an
114 individual test has failed.
8bfb6c5 Some english fixes
Melvyn Sopacua authored
115
c1caad6 Provide an example cron script, to invite people to test.
Melvyn Sopacua authored
116 Example script to be run by cron(1):
117 ========== qa-test.sh =============
118 #!/bin/sh
119
9b31c51 @jparise Update documentation for PHP 5.
jparise authored
120 CO_DIR=$HOME/cvs/php5
c1caad6 Provide an example cron script, to invite people to test.
Melvyn Sopacua authored
121 MYMAIL=qa-test@domain.com
122 TMPDIR=/var/tmp
123 TODAY=`date +"%Y%m%d"`
124
125 # Make sure compilation enviroment is correct
126 CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
127 export MAKE=gmake
128 export CC=gcc
129
130 # Set test environment
d15a641 Correcting example to use proper variable - NO_INTERACTION
David Hill authored
131 export NO_INTERACTION=1
c1caad6 Provide an example cron script, to invite people to test.
Melvyn Sopacua authored
132 export REPORT_EXIT_STATUS=1
133
134 cd $CO_DIR
135 cvs update . >>$TMPDIR/phpqatest.$TODAY
136 ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
137 $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
138 if test $? -gt 0
139 then
140 cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
141 fi
142 ========== end of qa-test.sh =============
143
8bfb6c5 Some english fixes
Melvyn Sopacua authored
144 NOTE: the exit status of run-tests.php will be 1 when
145 REPORT_EXIT_STATUS is set. The result of "make test" may be higher
146 than that. At present, gmake 3.79.1 returns 2, so it is
147 advised to test for non-zero, rather then a specific value.
148
08fead5 Added README.TESTING
Yasuo Ohgaki authored
149
150 [Creating new test files]
151 -------------------------
9307fa7 update
Marcus Boerger authored
152 Writing test file is very easy if you are used to PHP.
5eb9125 - Moved to http://qa.php.net/write-test.php
Derick Rethans authored
153 See the HOWTO at http://qa.php.net/write-test.php
08fead5 Added README.TESTING
Yasuo Ohgaki authored
154
b262301 Added note about error reporting in tests.
Ilia Alshanetsky authored
155
08fead5 Added README.TESTING
Yasuo Ohgaki authored
156 [How to help us]
157 ----------------
158 If you find bug in PHP, you can submit bug report AND test script
159 for us. You don't have to write complete script, just give us test
160 script with following format. Please test the script and make sure
161 you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
162 submit.
163
164 <?php
165 /*
166 Bug #12345
167 substr() bug. Do not return expected string.
168
169 ACTUAL OUTPUT
170 XYXA
171
172 EXPECTED OUTPUT
173 ABCD
174 */
175
176 $str = "XYZABCD";
177 echo substr($str,3,7);
178
179 ?>
Something went wrong with that request. Please try again.