Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 181 lines (142 sloc) 6.677 kb
08fead5b »
2002-03-05 Added README.TESTING
1 [IMPORTANT NOTICE]
2 ------------------
702ca7f5 »
2004-04-15 be a bit more explanatory ...
3 Failed tests usualy indicate a problem with your local system setup
4 and not within PHP itself (at least for official PHP release versions).
5 You may decide to automaticaly submit a test summary to our QA workflow
6 at the end of a test run.
7 Please do *not* submit a failed test as a bug or ask for help on why
8 it failed on your system without providing substantial backup information
9 on *why* the test failed on your special setup. Thank you :-)
08fead5b »
2002-03-05 Added README.TESTING
10
11
aba1396f »
2002-03-12 Catch up with recent test scripts changes
12 [Testing Basics]
13 ----------------
702ca7f5 »
2004-04-15 be a bit more explanatory ...
14 The easiest way to test your PHP build is to run "make test" from the
15 command line after successfully compiling. This will run the complete
16 tests for all enabled functionalities and extensions using the PHP
1d444f9a »
2004-04-15 typo
17 CLI binary.
aba1396f »
2002-03-12 Catch up with recent test scripts changes
18 To execute test scripts, you must build PHP with some SAPI, then you
19 type "make test" to execute all or some test scripts saved under
08fead5b »
2002-03-05 Added README.TESTING
20 "tests" directory under source root directory.
21
22 Usage:
23 make test
24
25 "make test" basically executes "run-tests.php" script
8bfb6c58 »
2002-10-23 Some english fixes
26 under the source root (parallel builds will not work). Therefore you
27 can execute the script as follows:
08fead5b »
2002-03-05 Added README.TESTING
28
8bfb6c58 »
2002-10-23 Some english fixes
29 TEST_PHP_EXECUTABLE=sapi/cli/php \
30 sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
08fead5b »
2002-03-05 Added README.TESTING
31
32
33 [Which "php" executable "make test" look for]
34 ---------------------------------------------
9705e310 »
2009-03-20 clarification of wwwhether 'make test' needs the PHP xecutable to be set
35 If you are running the run-tests.php script from the command line (as above)
36 you must set the TEST_PHP_EXECUTABLE environment variable to explicitly
37 select the PHP executable that is to be tested, that is, used to run the test scripts.
96f7bcce »
2002-09-12 Update info and synch example test.
38
9705e310 »
2009-03-20 clarification of wwwhether 'make test' needs the PHP xecutable to be set
39 If you run the tests using make test, the PHP CLI and CGI executables are
40 automatically set for you. "make test" executes "run-tests.php" script with the CLI binary. Some
aba1396f »
2002-03-12 Catch up with recent test scripts changes
41 test scripts such as session must be executed by CGI SAPI. Therefore,
42 you must build PHP with CGI SAPI to perform all tests.
08fead5b »
2002-03-05 Added README.TESTING
43
aba1396f »
2002-03-12 Catch up with recent test scripts changes
44 NOTE: PHP binary executing "run-tests.php" and php binary used for
45 executing test scripts may differ. If you use different PHP binary for
46 executing "run-tests.php" script, you may get errors.
08fead5b »
2002-03-05 Added README.TESTING
47
48
49 [Which php.ini is used]
50 -----------------------
8bfb6c58 »
2002-10-23 Some english fixes
51 "make test" uses the same php.ini file as it would once installed.
9f5af091 »
2002-10-23 Correct mispellings and some word misuse.
52 The tests have been written to be independent of that php.ini file,
8bfb6c58 »
2002-10-23 Some english fixes
53 so if you find a test that is affected by a setting, please report
54 this, so we can address the issue.
08fead5b »
2002-03-05 Added README.TESTING
55
56
57 [Which test scripts are executed]
58 ---------------------------------
8bfb6c58 »
2002-10-23 Some english fixes
59 "run-tests.php" ("make test"), without any arguments executes all
c95256b9 »
2002-10-23 no more looking
60 test scripts by extracting all directories named "tests"
8bfb6c58 »
2002-10-23 Some english fixes
61 from the source root and any subdirectories below. If there are files,
62 which have a "phpt" extension, "run-tests.php" looks at the sections
63 in these files, determines whether it should run it, by evaluating
9f5af091 »
2002-10-23 Correct mispellings and some word misuse.
64 the 'SKIP' section. If the test is eligible for execution, the 'FILE'
096d8dad »
2002-10-24 additions
65 section is extracted into a ".php" file (with the same name besides
66 the extension) and gets executed.
8bfb6c58 »
2002-10-23 Some english fixes
67 When an argument is given or TESTS environment variable is set, the
68 GLOB is expanded by the shell and any file with extension "*.phpt" is
9f5af091 »
2002-10-23 Correct mispellings and some word misuse.
69 regarded as a test file.
08fead5b »
2002-03-05 Added README.TESTING
70
96f7bcce »
2002-09-12 Update info and synch example test.
71 Tester can easily execute tests selectively with as follows.
08fead5b »
2002-03-05 Added README.TESTING
72
96f7bcce »
2002-09-12 Update info and synch example test.
73 Examples:
8bfb6c58 »
2002-10-23 Some english fixes
74 ./sapi/cli/php run-tests.php ext/mbstring/*
75 ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
08fead5b »
2002-03-05 Added README.TESTING
76
77
78 [Test results]
79 --------------
80 Test results are printed to standard output. If there is a failed test,
8bfb6c58 »
2002-10-23 Some english fixes
81 the "run-tests.php" script saves the result, the expected result and the
82 code executed to the test script directory. For example, if
83 ext/myext/tests/myext.phpt fails to pass, the following files are created:
08fead5b »
2002-03-05 Added README.TESTING
84
9f5af091 »
2002-10-23 Correct mispellings and some word misuse.
85 ext/myext/tests/myext.php - actual test file executed
96f7bcce »
2002-09-12 Update info and synch example test.
86 ext/myext/tests/myext.log - log of test execution (L)
87 ext/myext/tests/myext.exp - expected output (E)
88 ext/myext/tests/myext.out - output from test script (O)
89 ext/myext/tests/myext.diff - diff of .out and .exp (D)
08fead5b »
2002-03-05 Added README.TESTING
90
8bfb6c58 »
2002-10-23 Some english fixes
91 Failed tests are always bugs. Either the test is bugged or not considering
92 factors applying to the tester's environment, or there is a bug in PHP.
9f5af091 »
2002-10-23 Correct mispellings and some word misuse.
93 If this is a known bug, we strive to provide bug numbers, in either the
8bfb6c58 »
2002-10-23 Some english fixes
94 test name or the file name. You can check the status of such a bug, by
9f5af091 »
2002-10-23 Correct mispellings and some word misuse.
95 going to: http://bugs.php.net/12345 where 12345 is the bug number.
96 For clarity and automated processing, bug numbers are prefixed by a hash
97 sign '#' in test names and/or test cases are named bug12345.phpt.
96f7bcce »
2002-09-12 Update info and synch example test.
98
99 NOTE: The files generated by tests can be selected by setting the
100 environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
101 generated use the character in brackets as shown above (default is LEOD).
9da3c884 »
2002-10-23 update and correct
102 The php file will be generated always.
96f7bcce »
2002-09-12 Update info and synch example test.
103
104 NOTE: You can set environment variable TEST_PHP_DETAILED to enable
105 detailed test information.
08fead5b »
2002-03-05 Added README.TESTING
106
8bfb6c58 »
2002-10-23 Some english fixes
107 [Automated testing]
108 If you like to keep up to speed, with latest developments and quality
109 assurance, setting the environment variable NO_INTERACTION to 1, will not
110 prompt the tester for any user input.
111
112 Normally, the exit status of "make test" is zero, regardless of the results
9f5af091 »
2002-10-23 Correct mispellings and some word misuse.
113 of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
2322917a »
2002-10-23 statis->status
114 and "make test" will set the exit status ("$?") to non-zero, when an
115 individual test has failed.
8bfb6c58 »
2002-10-23 Some english fixes
116
c1caad68 »
2002-10-23 Provide an example cron script, to invite people to test.
117 Example script to be run by cron(1):
118 ========== qa-test.sh =============
119 #!/bin/sh
120
9b31c51d »
2003-07-03 Update documentation for PHP 5.
121 CO_DIR=$HOME/cvs/php5
c1caad68 »
2002-10-23 Provide an example cron script, to invite people to test.
122 MYMAIL=qa-test@domain.com
123 TMPDIR=/var/tmp
124 TODAY=`date +"%Y%m%d"`
125
126 # Make sure compilation enviroment is correct
127 CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
128 export MAKE=gmake
129 export CC=gcc
130
131 # Set test environment
d15a6412 »
2003-03-19 Correcting example to use proper variable - NO_INTERACTION
132 export NO_INTERACTION=1
c1caad68 »
2002-10-23 Provide an example cron script, to invite people to test.
133 export REPORT_EXIT_STATUS=1
134
135 cd $CO_DIR
136 cvs update . >>$TMPDIR/phpqatest.$TODAY
137 ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
138 $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
139 if test $? -gt 0
140 then
141 cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
142 fi
143 ========== end of qa-test.sh =============
144
8bfb6c58 »
2002-10-23 Some english fixes
145 NOTE: the exit status of run-tests.php will be 1 when
146 REPORT_EXIT_STATUS is set. The result of "make test" may be higher
147 than that. At present, gmake 3.79.1 returns 2, so it is
148 advised to test for non-zero, rather then a specific value.
149
08fead5b »
2002-03-05 Added README.TESTING
150
151 [Creating new test files]
152 -------------------------
9307fa75 »
2002-10-25 update
153 Writing test file is very easy if you are used to PHP.
5eb91255 »
2003-01-19 - Moved to http://qa.php.net/write-test.php
154 See the HOWTO at http://qa.php.net/write-test.php
08fead5b »
2002-03-05 Added README.TESTING
155
b2623015 »
2002-10-25 Added note about error reporting in tests.
156
08fead5b »
2002-03-05 Added README.TESTING
157 [How to help us]
158 ----------------
159 If you find bug in PHP, you can submit bug report AND test script
160 for us. You don't have to write complete script, just give us test
161 script with following format. Please test the script and make sure
162 you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
163 submit.
164
165 <?php
166 /*
167 Bug #12345
168 substr() bug. Do not return expected string.
169
170 ACTUAL OUTPUT
171 XYXA
172
173 EXPECTED OUTPUT
174 ABCD
175 */
176
177 $str = "XYZABCD";
178 echo substr($str,3,7);
179
180 ?>
Something went wrong with that request. Please try again.