Skip to content

Commit

Permalink
Some english fixes
Browse files Browse the repository at this point in the history
Added docs for automated testing.
Now telling the current version of the truth about ini settings :)
Be more verbose about QA goals intentions and the tester's part in the
process.
  • Loading branch information
Melvyn Sopacua committed Oct 23, 2002
1 parent 1de45a0 commit 8bfb6c5
Showing 1 changed file with 46 additions and 23 deletions.
69 changes: 46 additions & 23 deletions README.TESTING
Expand Up @@ -14,10 +14,11 @@ Usage:
make test make test


"make test" basically executes "run-tests.php" script "make test" basically executes "run-tests.php" script
under source root. Therefore you can execute the script under the source root (parallel builds will not work). Therefore you
as follows can execute the script as follows:


./sapi/cli/php -c /path/to/php.ini/ run-tests.php [ext/some_extension_name] TEST_PHP_EXECUTABLE=sapi/cli/php \
sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]




[Which "php" executable "make test" look for] [Which "php" executable "make test" look for]
Expand All @@ -37,46 +38,53 @@ executing "run-tests.php" script, you may get errors.


[Which php.ini is used] [Which php.ini is used]
----------------------- -----------------------
"make test" force to use php.ini-dist as default config file. If "make test" uses the same php.ini file as it would once installed.
you would like to test with other configuration file, user The tests have been written to be independant of that php.ini file,
"run-tests.php" script. so if you find a test that is affected by a setting, please report

this, so we can address the issue.
Example:
./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/standard

If you use php.ini other than php.ini-dist, you may see more failed
tests.




[Which test scripts are executed] [Which test scripts are executed]
--------------------------------- ---------------------------------
"run-tests.php" ("make test") executes all test scripts by default "run-tests.php" ("make test"), without any arguments executes all
by looking all directory named "tests". If there are files have "phpt" test scripts by looking extracting all directories named "tests"
extension, "run-tests.php" takes test php code from the file and from the source root and any subdirectories below. If there are files,
executes it. which have a "phpt" extension, "run-tests.php" looks at the sections
in these files, determines whether it should run it, by evaluating
the 'SKIP' section. If the test is elligable for execution, the 'FILE'
section is extracted into a ".php" file, with a unique name and
executed.
When an argument is given or TESTS environment variable is set, the
GLOB is expanded by the shell and any file with extension "*.phpt" is
regarded as a testfile.


Tester can easily execute tests selectively with as follows. Tester can easily execute tests selectively with as follows.


Examples: Examples:
./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/mbstring ./sapi/cli/php run-tests.php ext/mbstring/*
./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/mbstring/020.phpt ./sapi/cli/php run-tests.php ext/mbstring/020.phpt




[Test results] [Test results]
-------------- --------------
Test results are printed to standard output. If there is a failed test, Test results are printed to standard output. If there is a failed test,
"run-tests.php" script saves the result, expected result and code the "run-tests.php" script saves the result, the expected result and the
executed to the test script directory. For example, if code executed to the test script directory. For example, if
ext/myext/tests/myext.phpt is failed to pass, following files are ext/myext/tests/myext.phpt fails to pass, the following files are created:
created:


ext/myext/tests/myext.php - actual testfile executed ext/myext/tests/myext.php - actual testfile executed
ext/myext/tests/myext.log - log of test execution (L) ext/myext/tests/myext.log - log of test execution (L)
ext/myext/tests/myext.exp - expected output (E) ext/myext/tests/myext.exp - expected output (E)
ext/myext/tests/myext.out - output from test script (O) ext/myext/tests/myext.out - output from test script (O)
ext/myext/tests/myext.diff - diff of .out and .exp (D) ext/myext/tests/myext.diff - diff of .out and .exp (D)


Tester can verify these files, if failed test is actually a bug or not. Failed tests are always bugs. Either the test is bugged or not considering
factors applying to the tester's environment, or there is a bug in PHP.
If this is a known bug, we strive to provide bugnumbers, in either the
test name or the file name. You can check the status of such a bug, by
going to: http://bugs.php.net/12345 where 12345 is the bugnumber.
For clarity and automated processing, bugnumbers are prefixed by a hash
sign '#' in testnames and/or testcases are named bug12345.phpt.


NOTE: The files generated by tests can be selected by setting the NOTE: The files generated by tests can be selected by setting the
environment variable TEST_PHP_LOG_FORMAT. For each file you want to be environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
Expand All @@ -86,6 +94,21 @@ The php file will be generated always.
NOTE: You can set environment variable TEST_PHP_DETAILED to enable NOTE: You can set environment variable TEST_PHP_DETAILED to enable
detailed test information. detailed test information.


[Automated testing]
If you like to keep up to speed, with latest developments and quality
assurance, setting the environment variable NO_INTERACTION to 1, will not
prompt the tester for any user input.

Normally, the exit status of "make test" is zero, regardless of the results
of independant tests. Set the environment variable REPORT_EXIT_STATUS to 1,
to make "make test' set the exit statis ("$?") to non-zero, when a test has
failed.

NOTE: the exit status of run-tests.php will be 1 when
REPORT_EXIT_STATUS is set. The result of "make test" may be higher
than that. At present, gmake 3.79.1 returns 2, so it is
advised to test for non-zero, rather then a specific value.



[Creating new test files] [Creating new test files]
------------------------- -------------------------
Expand Down

0 comments on commit 8bfb6c5

Please sign in to comment.