Details of PHPT format
+-
+
- --TEST-- +
- title of the test. (required) + +
- --SKIPIF-- +
- a condition when to skip this test. (optional) + +
- --POST-- +
- POST variables to be passed to the test script. This section forces the +use of the CGI binary instead of the usual CLI one. (optional) + +
- --GZIP_POST-- +
- When this section exists, the POST data will be gzencode()'d. (optional) + +
- --DEFLATE_POST-- +
- When this section exists, the POST data will be gzcompress()'ed. (optional) + +
- --POST_RAW-- +
- RAW POST data to be passed to the test script. This differs from the section +above because it doesn't set the Content-Type, which can be set manually in +this section. This section forces the use of the CGI binary instead of the +usual CLI one. (optional) + +
- --GET-- +
- GET variables to be passed to the test script. This section forces the +use of the CGI binary instead of the usual CLI one. (optional) + +
- --COOKIE-- +
- Cookies to be passed to the test script. This section forces the +use of the CGI binary instead of the usual CLI one. (optional) + +
- --STDIN-- +
- data to be fed to the test script's standard input. (optional) + +
- --INI-- +
- to be used if you need a specific php.ini setting for the test. + Each line contains an ini setting e.g. foo=bar. (optional) + +
- --ARGS-- +
- a single line defining the arguments passed to php. (optional) + +
- --ENV-- +
- configures the environment to be used for php. (optional) + +
- --FILE-- +
- the test source-code. (required) + +
- --FILEEOF-- +
- an alternative to --FILE-- where any trailing line break is omitted. + +
- --FILE_EXTERNAL-- +
- an alternative to --FILE--. This is used to specify that an external +file should be used as the contents of the test file, and is designed +for running the same test file with different ini, environment, post/get +or other external inputs. The file must be in the same directory as the +test file, or a subdirectory. + +
- --EXPECT-- +
- the expected output from the test script. (required) + +
- --UEXPECT-- +
- same as above, but for Unicode mode (PHP >= 6 only, optional) + +
- --EXPECTF-- +
- an alternative of --EXPECT--. The difference is that this form uses +sscanf for output validation. (alternative to --EXPECT--) + +
- --UEXPECTF-- +
- same as above, but for Unicode mode (PHP >= 6 only, optional) + +
- --EXPECTREGEX-- +
- an alternative of --EXPECT--. This form allows the tester to specify the +result in a regular expression. (alternative to --EXPECT--) + +
- --UEXPECTREGEX-- +
- same as above, but for Unicode mode (PHP >= 6 only, optional) + +
- --REDIRECTTEST-- +
- this block allows to redirect from one test to a bunch of other tests. +(alernative to --FILE--) + +
- --HEADERS-- +
- header to be used when sending the request. Currently only available with +server-tests.php (optional) + +
- --EXPECTHEADERS-- +
- the expected headers. Any header specified here must exist in the +response and have the same value. Additional headers do not matter. (optional) + + +
- --CLEAN-- +
- code that is executed after the test has run. Using run-tests.php switch +--no-clean you can prevent its execution to inspect generated data/files that +were normally removed after the test. (optional) + +
- ===DONE=== +
- This is only available in the --FILE-- section. Any part after this line +is not going into the actual test script (see below for more). +
Note: The Uxx sections (such as UEXPECT) are only needed if +the output of the test differs in Unicode and non-Unicode mode.
diff --git a/styles.css b/styles.css index 8694ad48..f43ff3a1 100644 --- a/styles.css +++ b/styles.css @@ -90,6 +90,11 @@ h4 { font-size: 8pt; } +A:link {text-decoration: none; color: blue;} +A:visited {text-decoration: none; color: blue;} +A:active {text-decoration: none} +A:hover {text-decoration: underline; color: blue;} + /* Use this w/spans to hack past Netscape's li css etc bug. */ .lihack { diff --git a/write-test.php b/write-test.php index 7bd231b6..616a8184 100644 --- a/write-test.php +++ b/write-test.php @@ -19,6 +19,13 @@phpt Test Basics
+The first thing you need to know about tests is that we need more!!! Although PHP works just great +99.99% of the time, not having a very comprehensive test suite means that we take more risks every time +we add to or modify the PHP implementation. The second +thing you need to know is that if you can write PHP you can write tests. Thirdly - we are a friendly +and welcoming community, don't be scared about writing to + (php-qa@lists.php.net) - we won't bite! +
-
So what are phpt tests?
@@ -42,10 +49,12 @@
php functions available. You can write a test on a basic language function (a string
function or an array function) , or a function provided by one of PHP's numerous extensions
(a mysql function or a image function or a mcrypt function). You can find out what functions
-already have phpt tests on them by looking in the html
-version of the cvs (php-src/ext/standard/tests/ is a good place to start looking - though
-all the tests currently written are NOT in there). After that - you
-can pick any function you want. If you want more guidance than that - you can always ask
+already have phpt tests by looking in the html
+version of the cvs (php-src/ext/standard/tests/ is a good place to start looking - though not
+all the tests currently written are in there). If you look at the gcov pages you
+can see which functions have lots of tests and which need more, although these pages only
+show which lines of code are covered by test cases so even if the coverage looks good there may
+be more interesting tests to write - for example covering error cases. If you want more guidance than that you can always ask
the PHP Quality Assurance Team on their mailing list
(php-qa@lists.php.net) where they
would like you to direct your attentions.
@@ -63,7 +72,7 @@ function or an array function) , or a function provided by one of PHP's numerous
What should a phpt test do?
Basically - it should try and break the PHP function. It should check not only the functions normal parameters, but it should also check edge cases. Intentionally generating -an error is allowed.
+an error is allowed and encouraged.
bug<bugid>.phpt (bug17123.phpt)
+
+ + <functionname>_basic.phpt (dba_open_basic.phpt)
+ + <functionname>_error.phpt (dba_open_error.phpt)
- <functionname>.phpt (dba_open.phpt)
<extname><no>.phpt (dba_003.phpt)
The convention of using _basic, _error and _variation was introduced when we +found that writing a single test case for each function resulted in unacceptably large +test cases. It's quite hard to debug problems when the test case generates 100s of lines of output. +
The "basic" test case for a function should just address the single most simple +thing that the function is designed to do. For example, if writing a test for the sin() function +a basic test would just be to check that sin() returns the correct values for some known angles +- eg 30, 90, 180.
+The "error" tests for a function are test cases which are designed to provoke errors, warnings or notices. +There can be more than one error case, if so the convention is to name the test cases mytest_error1.phpt, +mytest_error2.phpt and so on.
+
The "variation" tests are any tests that don't fit into "basic" or "error" tests. For example one might +use a variation tests to test boundary conditions.
+How big is a test case?
+Small. Really - the smaller the better, a good guide is no more than 10 lines of output. The reason +for this is that if we break something in PHP and it breaks your test case we need to be able to find +out quite quickly what we broke, going through 1000s of line of test case output is not easy. Having said that +it's sometimes just not practical to stay within the 10 line guideline, in this case you can help a lot +by commenting the output. You may find plenty of much longer tests in PHP - the small tests message is something +that we learnt over time, in fact we are slowly going through and splitting tests up when we need to. +
+Comments
+Comments help. Not an essay - just a couple of lines on what the objective of the test is. It may seem completely +obvious to you as you write it, but it might not be to someone looking at it later on
Basic Format
A test must contain the sections TEST, FILE and either EXPECT or EXPECTF at a minimum. The example @@ -90,10 +128,10 @@ function or an array function) , or a function provided by one of PHP's numerous ext/standard/tests/strings/strtr.phpt
--TEST--
-strtr() function
+strtr() function - basic test for strstr()
--FILE--
<?php
-/* Do not change this test it is a REATME.TESTING example. */
+/* Do not change this test it is a README.TESTING example. */
$trans = array("hello"=>"hi", "hi"=>"hello", "a"=>"A", "world"=>"planet");
var_dump(strtr("# hi all, I said hello world! #", $trans));
?>
@@ -101,171 +139,47 @@ function or an array function) , or a function provided by one of PHP's numerous
string(32) "# hello All, I sAid hi planet! #"
-As you can see the file is divided into several sections. The TEST section holds a one line comment -about the phpt test. The phpt files name is used when generating a .php file. The FILE section is used +
As you can see the file is divided into several sections. The TEST section holds a one line title +of the phpt test, this shoudl be a simple description and shouldn't ever excede one line, if you need to write more explanation +add comments in the body of the test case. The phpt files name is used when generating a .php file. The FILE section is used as the body of the .php file, so don't forget to open and close your php tags. The EXPECT section is the part used as a comparison to see if the test passes. It is a good idea to generate output with var_dump() calls.
-Cleaning up after writing a test
-Sometimes test cases create files or directories as part of the test case and it's important to remove these after the test ends, the --CLEAN-- -section is provided to help with this.
-The PHP code in the --CLEAN-- section is executed separately from the code in the --FILE-- section. For example, this code:
----TEST-- -Will fail to clean up ---FILE-- -<?php - $temp_filename = "fred.tmp"; - $fp = fopen($temp_filename, "w"); - fwrite($fp, "Hello Boys!"); - fclose($fp); -?> ---CLEAN-- -<?php - unlink($temp_filename); -?> ---EXPECT-- --
will not remove the temporary file because the variable $filename is not defined in the --CLEAN-- section.
-Here is a better way to write the code: -
---TEST-- -This will remove temporary files ---FILE-- -<?php - $temp_filename = dirname(__FILE__)."/fred.tmp"; - $fp = fopen($temp_filename, "w"); - fwrite ($fp, "Hello Boys!\n"); - fclose($fp); -?> ---CLEAN-- -<?php - $temp_filename = dirname(__FILE__)."/fred.tmp"; - unlink($temp_filename); -?> ---EXPECT-- --
Note the use of the dirname(__FILE__) construct which will ensure that the temporary file is created in the same directory as -the phpt test script.
- -When creating temporary files it is a good idea to use an extension that indicates the use of the file, eg .tmp. It's also a good -idea to avoid using extensions that are already used for other purposes, eg .inc, .php. Similarly, it is helpful to give the temporary file a name -that is clearly related to the test case. For example, mytest.phpt should create mytest.tmp (or mytestN.tmp, N=1, 2,3,...) then if by any -chance the temporary file isnt't removed properly it will be obvious which test case created it.
- -When writing and debugging a test case with a --CLEAN-- section it is helpful to remember that the php code in the --CLEAN-- section -is executed separately from the code in the --FILE-- section. For example, in a test case called mytest.phpt, code from the --FILE-- -section is run from a file called mytest.php and code from the --CLEAN-- section is run from a file called mytest.clean.php. If the test passes, -both the .php and .clean.php files are removed by run-tests.php. You can prevent the removal by using the --keep option of run-tests.php, -this is a very useful option if you need to check that the --CLEAN-- section code is working as you intended. -
Finally - if you are using CVS it's helpful to add the extension that you use for test-related temporary files to the .cvsignore file - -this will help to prevent you from accidentally checking temporary files into CVS.
-All Sections
+PHPT structure details
A phpt test can have many more parts than just the minimum. In fact some of the mandatory parts have -alternatives that may be used if the situation warrants it. --
-
- --TEST-- -
- title of the test. (required) - -
- --SKIPIF-- -
- a condition when to skip this test. (optional) - -
- --POST-- -
- POST variables to be passed to the test script. This section forces the -use of the CGI binary instead of the usual CLI one. (optional) - -
- --GZIP_POST-- -
- When this section exists, the POST data will be gzencode()'d. (optional) - -
- --DEFLATE_POST-- -
- When this section exists, the POST data will be gzcompress()'ed. (optional) - -
- --POST_RAW-- -
- RAW POST data to be passed to the test script. This differs from the section -above because it doesn't set the Content-Type, which can be set manually in -this section. This section forces the use of the CGI binary instead of the -usual CLI one. (optional) - -
- --GET-- -
- GET variables to be passed to the test script. This section forces the -use of the CGI binary instead of the usual CLI one. (optional) - -
- --COOKIE-- -
- Cookies to be passed to the test script. This section forces the -use of the CGI binary instead of the usual CLI one. (optional) - -
- --STDIN-- -
- data to be fed to the test script's standard input. (optional) - -
- --INI-- -
- to be used if you need a specific php.ini setting for the test. - Each line contains an ini setting e.g. foo=bar. (optional) - -
- --ARGS-- -
- a single line defining the arguments passed to php. (optional) - -
- --ENV-- -
- configures the environment to be used for php. (optional) - -
- --FILE-- -
- the test source-code. (required) - -
- --FILEEOF-- -
- an alternative to --FILE-- where any trailing line break is omitted. - -
- --FILE_EXTERNAL-- -
- an alternative to --FILE--. This is used to specify that an external -file should be used as the contents of the test file, and is designed -for running the same test file with different ini, environment, post/get -or other external inputs. The file must be in the same directory as the -test file, or a subdirectory. - -
- --EXPECT-- -
- the expected output from the test script. (required) - -
- --UEXPECT-- -
- same as above, but for Unicode mode (PHP >= 6 only, optional) - -
- --EXPECTF-- -
- an alternative of --EXPECT--. The difference is that this form uses -sscanf for output validation. (alternative to --EXPECT--) - -
- --UEXPECTF-- -
- same as above, but for Unicode mode (PHP >= 6 only, optional) - -
- --EXPECTREGEX-- -
- an alternative of --EXPECT--. This form allows the tester to specify the -result in a regular expression. (alternative to --EXPECT--) - -
- --UEXPECTREGEX-- -
- same as above, but for Unicode mode (PHP >= 6 only, optional) - -
- --REDIRECTTEST-- -
- this block allows to redirect from one test to a bunch of other tests. -(alernative to --FILE--) - -
- --HEADERS-- -
- header to be used when sending the request. Currently only available with -server-tests.php (optional) - -
- --EXPECTHEADERS-- -
- the expected headers. Any header specified here must exist in the -response and have the same value. Additional headers do not matter. (optional) - - -
- --CLEAN-- -
- code that is executed after the test has run. Using run-tests.php switch ---no-clean you can prevent its execution to inspect generated data/files that -were normally removed after the test. (optional) - -
- ===DONE=== -
- This is only available in the --FILE-- section. Any part after this line -is not going into the actual test script (see below for more). -
Note: The Uxx sections (such as UEXPECT) are only needed if -the output of the test differs in Unicode and non-Unicode mode.
+alternatives that may be used if the situation warrants it. The phpt sections are documented here. +There is also a very useful set of slides, written by Marcus Boerger here. +Look at the talk entitled "The need for speed, ERM testing".
+ +Autogenerating test cases
+It isn't possible (or even sensible) to try and generate complete test cases for PHP. However +there is a script in PHP5.3 which will help to generate the framework. It can save you +some typing and ensure that you get a good basic format. +See test case generation for instructions on how to use it.
+ + +Testing your test cases
+Most people who write tests for PHP don't have access to a huge number of operating systems but the tests are run +on every system that runs PHP. It's good to test your test on as many platforms as you can - Linux and Windows +are the most important, it's increasingly important to make sure that tests run on 64 bit as well as 32 bit platforms. If you only +have access to one operating system - don't worry, if you have karma, commit the test but watch php-qa@lists.php.net for reports +of failures on other platforms. If you don't have karma to commit have a look at the next section.
+When you are testing your test case it's really important to make sure that you +clean up any temporary resources (eg files) that you used in the test. There is a special --CLEAN-- section +to help you do this - see here. +
Another good check is to look at what lines of code in the PHP source your test case covers. +This is easy to do, there are some instructions on the PHP Wiki.
+ +What should I do with my test case when I've written and tested it?
+The next step is to get someone to review it. If it's short you can paste it into a note and +send it to php-qa@lists.php.net. If the test is a bit too +long for that then put it somewhere were people can download it (pastebin is +sometimes used). Appending tests to notes as files doesn't work well - so please don't do that. Your +note to php-qa@lists.php.net should say +what level of PHP you have tested it on and what platform(s) you've run it on. Someone from +the PHP QA group will review your test and reply to you. They may ask for some changes +or suggest better ways to do things, or they may commit it to PHP.
Examples
@@ -363,6 +277,62 @@ function or an array function) , or a function provided by one of PHP's numerous FILE section of many tests should be named "test.inc".Final Notes
+Cleaning up after running a test
+Sometimes test cases create files or directories as part of the test case and it's important to remove these after the test ends, the --CLEAN-- +section is provided to help with this.
+ +The PHP code in the --CLEAN-- section is executed separately from the code in the --FILE-- section. For example, this code:
++--TEST-- +Will fail to clean up +--FILE-- +<?php + $temp_filename = "fred.tmp"; + $fp = fopen($temp_filename, "w"); + fwrite($fp, "Hello Boys!"); + fclose($fp); +?> +--CLEAN-- +<?php + unlink($temp_filename); +?> +--EXPECT-- ++
will not remove the temporary file because the variable $filename is not defined in the --CLEAN-- section.
+Here is a better way to write the code: +
+--TEST-- +This will remove temporary files +--FILE-- +<?php + $temp_filename = dirname(__FILE__)."/fred.tmp"; + $fp = fopen($temp_filename, "w"); + fwrite ($fp, "Hello Boys!\n"); + fclose($fp); +?> +--CLEAN-- +<?php + $temp_filename = dirname(__FILE__)."/fred.tmp"; + unlink($temp_filename); +?> +--EXPECT-- ++
Note the use of the dirname(__FILE__) construct which will ensure that the temporary file is created in the same directory as +the phpt test script.
+ +When creating temporary files it is a good idea to use an extension that indicates the use of the file, eg .tmp. It's also a good +idea to avoid using extensions that are already used for other purposes, eg .inc, .php. Similarly, it is helpful to give the temporary file a name +that is clearly related to the test case. For example, mytest.phpt should create mytest.tmp (or mytestN.tmp, N=1, 2,3,...) then if by any +chance the temporary file isnt't removed properly it will be obvious which test case created it.
+ +When writing and debugging a test case with a --CLEAN-- section it is helpful to remember that the php code in the --CLEAN-- section +is executed separately from the code in the --FILE-- section. For example, in a test case called mytest.phpt, code from the --FILE-- +section is run from a file called mytest.php and code from the --CLEAN-- section is run from a file called mytest.clean.php. If the test passes, +both the .php and .clean.php files are removed by run-tests.php. You can prevent the removal by using the --keep option of run-tests.php, +this is a very useful option if you need to check that the --CLEAN-- section code is working as you intended. + +
Finally - if you are using CVS it's helpful to add the extension that you use for test-related temporary files to the .cvsignore file - +this will help to prevent you from accidentally checking temporary files into CVS.
Redirecting tests
Using --REDIRECTTEST-- it is possible to redirect from one test to a bunch of other tests. That way multiple extensions can refer to the same set of @@ -407,6 +377,28 @@ function or an array function) , or a function provided by one of PHP's numerous When doing so run-tests.php does not execute the line containing the exit call as that would suppress leak messages. Actually run-tests.php ignores any part after a line consisting only of "===DONE===".
+Here is an example:
++--TEST-- +Test hypot() - dealing with mixed number/character input +--INI-- +precision=14 +--FILE-- +<?php + $a="23abc"; + $b=-33; + echo "$a :$b "; + $res = hypot($a, $b); + var_dump($res); +?> +===DONE=== +<?php exit(0); ?> +--EXPECTF-- +23abc :-33 float(40.224370722238) +===DONE=== ++
If executed as PHP script the output will stop after the code on the --FILE-- section +has been run.