Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 171 lines (132 sloc) 5.979 kB
08fead5 Added README.TESTING
Yasuo Ohgaki authored
1 [IMPORTANT NOTICE]
2 ------------------
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
3 Do _not_ ask to developers why some or all tests are failed under
08fead5 Added README.TESTING
Yasuo Ohgaki authored
4 your environment! Let us know if you find why it fails. Thank you.
5
6
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
7 [Testing Basics]
8 ----------------
9 To execute test scripts, you must build PHP with some SAPI, then you
10 type "make test" to execute all or some test scripts saved under
08fead5 Added README.TESTING
Yasuo Ohgaki authored
11 "tests" directory under source root directory.
12
13 Usage:
14 make test
15
16 "make test" basically executes "run-tests.php" script
8bfb6c5 Some english fixes
Melvyn Sopacua authored
17 under the source root (parallel builds will not work). Therefore you
18 can execute the script as follows:
08fead5 Added README.TESTING
Yasuo Ohgaki authored
19
8bfb6c5 Some english fixes
Melvyn Sopacua authored
20 TEST_PHP_EXECUTABLE=sapi/cli/php \
21 sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
08fead5 Added README.TESTING
Yasuo Ohgaki authored
22
23
24 [Which "php" executable "make test" look for]
25 ---------------------------------------------
96f7bcc Update info and synch example test.
Marcus Boerger authored
26 You must use TEST_PHP_EXECUTABLE environment variable to explicitly
27 select the php executable to be used to run the tests. That can either
28 be the CLI or CGI executable.
29
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
30 "make test" executes "run-tests.php" script with "php" binary. Some
31 test scripts such as session must be executed by CGI SAPI. Therefore,
32 you must build PHP with CGI SAPI to perform all tests.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
33
aba1396 Catch up with recent test scripts changes
Yasuo Ohgaki authored
34 NOTE: PHP binary executing "run-tests.php" and php binary used for
35 executing test scripts may differ. If you use different PHP binary for
36 executing "run-tests.php" script, you may get errors.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
37
38
39 [Which php.ini is used]
40 -----------------------
8bfb6c5 Some english fixes
Melvyn Sopacua authored
41 "make test" uses the same php.ini file as it would once installed.
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
42 The tests have been written to be independent of that php.ini file,
8bfb6c5 Some english fixes
Melvyn Sopacua authored
43 so if you find a test that is affected by a setting, please report
44 this, so we can address the issue.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
45
46
47 [Which test scripts are executed]
48 ---------------------------------
8bfb6c5 Some english fixes
Melvyn Sopacua authored
49 "run-tests.php" ("make test"), without any arguments executes all
c95256b no more looking
Melvyn Sopacua authored
50 test scripts by extracting all directories named "tests"
8bfb6c5 Some english fixes
Melvyn Sopacua authored
51 from the source root and any subdirectories below. If there are files,
52 which have a "phpt" extension, "run-tests.php" looks at the sections
53 in these files, determines whether it should run it, by evaluating
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
54 the 'SKIP' section. If the test is eligible for execution, the 'FILE'
096d8da additions
Marcus Boerger authored
55 section is extracted into a ".php" file (with the same name besides
56 the extension) and gets executed.
8bfb6c5 Some english fixes
Melvyn Sopacua authored
57 When an argument is given or TESTS environment variable is set, the
58 GLOB is expanded by the shell and any file with extension "*.phpt" is
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
59 regarded as a test file.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
60
96f7bcc Update info and synch example test.
Marcus Boerger authored
61 Tester can easily execute tests selectively with as follows.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
62
96f7bcc Update info and synch example test.
Marcus Boerger authored
63 Examples:
8bfb6c5 Some english fixes
Melvyn Sopacua authored
64 ./sapi/cli/php run-tests.php ext/mbstring/*
65 ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
08fead5 Added README.TESTING
Yasuo Ohgaki authored
66
67
68 [Test results]
69 --------------
70 Test results are printed to standard output. If there is a failed test,
8bfb6c5 Some english fixes
Melvyn Sopacua authored
71 the "run-tests.php" script saves the result, the expected result and the
72 code executed to the test script directory. For example, if
73 ext/myext/tests/myext.phpt fails to pass, the following files are created:
08fead5 Added README.TESTING
Yasuo Ohgaki authored
74
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
75 ext/myext/tests/myext.php - actual test file executed
96f7bcc Update info and synch example test.
Marcus Boerger authored
76 ext/myext/tests/myext.log - log of test execution (L)
77 ext/myext/tests/myext.exp - expected output (E)
78 ext/myext/tests/myext.out - output from test script (O)
79 ext/myext/tests/myext.diff - diff of .out and .exp (D)
08fead5 Added README.TESTING
Yasuo Ohgaki authored
80
8bfb6c5 Some english fixes
Melvyn Sopacua authored
81 Failed tests are always bugs. Either the test is bugged or not considering
82 factors applying to the tester's environment, or there is a bug in PHP.
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
83 If this is a known bug, we strive to provide bug numbers, in either the
8bfb6c5 Some english fixes
Melvyn Sopacua authored
84 test name or the file name. You can check the status of such a bug, by
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
85 going to: http://bugs.php.net/12345 where 12345 is the bug number.
86 For clarity and automated processing, bug numbers are prefixed by a hash
87 sign '#' in test names and/or test cases are named bug12345.phpt.
96f7bcc Update info and synch example test.
Marcus Boerger authored
88
89 NOTE: The files generated by tests can be selected by setting the
90 environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
91 generated use the character in brackets as shown above (default is LEOD).
9da3c88 update and correct
Marcus Boerger authored
92 The php file will be generated always.
96f7bcc Update info and synch example test.
Marcus Boerger authored
93
94 NOTE: You can set environment variable TEST_PHP_DETAILED to enable
95 detailed test information.
08fead5 Added README.TESTING
Yasuo Ohgaki authored
96
8bfb6c5 Some english fixes
Melvyn Sopacua authored
97 [Automated testing]
98 If you like to keep up to speed, with latest developments and quality
99 assurance, setting the environment variable NO_INTERACTION to 1, will not
100 prompt the tester for any user input.
101
102 Normally, the exit status of "make test" is zero, regardless of the results
9f5af09 @jparise Correct mispellings and some word misuse.
jparise authored
103 of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
2322917 statis->status
Melvyn Sopacua authored
104 and "make test" will set the exit status ("$?") to non-zero, when an
105 individual test has failed.
8bfb6c5 Some english fixes
Melvyn Sopacua authored
106
c1caad6 Provide an example cron script, to invite people to test.
Melvyn Sopacua authored
107 Example script to be run by cron(1):
108 ========== qa-test.sh =============
109 #!/bin/sh
110
111 CO_DIR=$HOME/cvs/php4
112 MYMAIL=qa-test@domain.com
113 TMPDIR=/var/tmp
114 TODAY=`date +"%Y%m%d"`
115
116 # Make sure compilation enviroment is correct
117 CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
118 export MAKE=gmake
119 export CC=gcc
120
121 # Set test environment
122 export NO_INTERACTIVITY=1
123 export REPORT_EXIT_STATUS=1
124
125 cd $CO_DIR
126 cvs update . >>$TMPDIR/phpqatest.$TODAY
127 ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
128 $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
129 if test $? -gt 0
130 then
131 cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
132 fi
133 ========== end of qa-test.sh =============
134
8bfb6c5 Some english fixes
Melvyn Sopacua authored
135 NOTE: the exit status of run-tests.php will be 1 when
136 REPORT_EXIT_STATUS is set. The result of "make test" may be higher
137 than that. At present, gmake 3.79.1 returns 2, so it is
138 advised to test for non-zero, rather then a specific value.
139
08fead5 Added README.TESTING
Yasuo Ohgaki authored
140
141 [Creating new test files]
142 -------------------------
9307fa7 update
Marcus Boerger authored
143 Writing test file is very easy if you are used to PHP.
5eb9125 - Moved to http://qa.php.net/write-test.php
Derick Rethans authored
144 See the HOWTO at http://qa.php.net/write-test.php
08fead5 Added README.TESTING
Yasuo Ohgaki authored
145
b262301 Added note about error reporting in tests.
Ilia Alshanetsky authored
146
08fead5 Added README.TESTING
Yasuo Ohgaki authored
147 [How to help us]
148 ----------------
149 If you find bug in PHP, you can submit bug report AND test script
150 for us. You don't have to write complete script, just give us test
151 script with following format. Please test the script and make sure
152 you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
153 submit.
154
155 <?php
156 /*
157 Bug #12345
158 substr() bug. Do not return expected string.
159
160 ACTUAL OUTPUT
161 XYXA
162
163 EXPECTED OUTPUT
164 ABCD
165 */
166
167 $str = "XYZABCD";
168 echo substr($str,3,7);
169
170 ?>
Something went wrong with that request. Please try again.