Skip to content
A test tool for the PHP ecosystem designed for convenience, thoroughness and speed. - moved to http://git.php.net/?p=pftt2.git
Java Groovy Other

README.mdown

PFTT - PHP Full Test Tool

PFTT is a test tool for the PHP ecosystem designed for convenience, thoroughness and speed. It supports PHPUnit tests for applications and frameworks, WebDriver based UI Testing for User Interfaces and PHPT tests for the PHP Core interpreter. It can test multiple scenarios including running PHPT and PhpUnit tests on real web servers like Apache to actually test what will happen in production.

INSTALL

  1. Download binary from http://131.107.220.66/PFTT-Results/PFTT/current/ and run it.
  2. The Install Wizard will install PFTT to your local system. a. If you have installed PFTT before, you MUST delete c:\php-sdk\PFTT\current so that it does NOT exist before running the installer b. If you set the environment variable PHP_SDK, PFTT will use that place of C:\php-sdk (including for c:\php-sdk\PFTT\current) i. You can create a batch script named \PFTT\current\bin\internal\set_php_sdk.cmd where you can SET PHP_SDK, which will be used whenever you run PFTT instead of the default PHP_SDK value
  3. If you want to test with Apache, currently you’ll need to manually decompress/copy it a. Look in c:\php-sdk\PFTT\current\cache\dep\Apache (Windows) b. Each of the .7z files is a different Apache build (built by apachelounge.com) c. You’ll need to decompress them to c:<Apache Build> so you’ll have i. C:\Apache244-VC11-OpenSSL1.0.1e-x64\bin\httpd.exe ii. C:\Apache244-VC11-OpenSSL1.0.1e-x86\bin\httpd.exe iii. C:\Apache244-VC9-OpenSSL0.9.8y-x86\bin\httpd.exe iv. C:\ Apache224-VC9-OpenSSL0.9.8y-x86\bin\httpd.exe d. PFTT will assume they’re there and fail if they aren’t

USAGE

Windows:

Click on PFTT Shell in \php-sdk\PFTT\current\bin.
Add it to your taskbar or start menu for next time.

Linux:

> PFTT_HOME/bin/set_env

Using the PFTT Shell (customized bash shell on Linux or Windows command prompt)

List releases (by build, branch, both, or all)

> release_list 5.3
> release_list NTS
> rl nts 5.3
> rl

Download specific release (test-pack and build)

> pftt rg MASTER TS rc165a0b

Run all PHPTs from the test-pack

> core_all C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567

Run all PHPTs from the test-pack on Apache

> core_all -c apache C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567

Run all PHPTs from the test-pack with Opcache on CLI and Apache

> core_all -c apache,cli,opcache C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567

Run all PHPTs from the test-pack with Opcache and with no code cache on CLI and Apache and PHP's builtin web server

> core_all -c apache,cli,builtin_web,opcache,no_code_cache C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567

Run all Symfony's PhpUnit tests on Apache with Opcache and with no code cache

> app_all -c symfony,apache,opcache,no_code_cache C:\php-sdk\php-5.4.7-win32-vc9-x86

Runs PHPTs only from list in SKIP_list_file (can add comments to lists with a ; or #)

> core_list C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567 SKIP_list_file

Runs PHPTs that contain 021 (name fragment) in their name

> core_named C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567 021

Opens a GUI to efficiently review large amounts of tests

> ca -gui C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567 pftt -gui -config config_file core_list C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567 pftt -gui -config config_file core_named C:\php-sdk\php-5.4.7-win32-vc9-x86 C:\php-sdk\php-test-pack-5.4-r1234567

Shows help info (additional commands, etc...)

> pftt -help

Adding a PHPUnit Application

  1. Create a .groovy file in /conf/app
  2. Extend PhpUnitSourceTestPack
  3. implement #getSourceRoot
  4. find all the phpunit.dist.xml files in the test-pack
  5. implement #openAfterInstall
  6. (required) create corresponding PhpUnitDists by calling PhpUnitSourceTestPack#addPhpUnitDist and provide all the information from the phpunit.dist.xml file. the Javadoc on the PhpUnitDist methods explains which method matches which XML tag.
  7. provide files and directories to include to PhpUnitSourceTestPack (optional)
  8. check if any tests are non-thread-safe (NTS) and if so, add their file names or partial file names to the list returned by PhpUnitSourceTestPack#getNonThreadSafeTestFileNames To speed test running, making testing more convenient and thus done more frequently and thoroughly, test running is threaded, so multiple tests are run at the same time except for NTS tests.
  9. you may provide some additional info to PhpUnitSourceTestPack (optional; mainly, its just doing steps 5 and 6)

Crash debugging

To re-run tests in exactly the same order, use -thread_count 1 console option with app_list or core_list commands.

For crashes that happen every time

  1. Create a list of the tests that are crashing or might be crashing
  2. Run with -debug_all and core_list

This will run all the tests in a Debugger (WinDebug for Windows and GDB for Linux). Do 1 first to reduce the number of tests to run since they'll run much slower in a debugger.

When testing with Apache, this will run the Debugger on Apache to help trace Apache related problems.

Stress Testing

For tests that crash occasionally

run tests 100 times each, 32 threads, no thread safety = so multiple instances of a test may be run at the same time (depending on size of list/speed of test)

> cl –run_test_times_all 100 –no_thread_safety –thread_count 32 –c apache,opcache php-5.5 php-5.5.0beta4-nts-Win32-VC11-x64 php-test-pack-5.5.0beta4 list_of_crashing_tests.txt

Stress, running listed tests 1000 times and all other tests once each

> ca -run_test_times_list 1000 wincache_crash.txt -ignore_output -randomize_order -no_thread_safety -thread_count 192 -c wincacheu_file_and_user,opcache,cli -debug_pack php-debug-pack-5.5-nts-windows-vc11-x86-rdb61685 php-5.5-nts-windows-vc11-x86-rdb61685 php-test-pack-5.5-ts-windows-vc11-x86-rdb61685

On Windows, with the Debugging Tools installed, you can use Time-Travel-Tracing (TTT) to trace bugs

> ca -debugger ttt -run_test_times_list 1000 wincache_crash.txt -ignore_output -randomize_order -no_thread_safety -thread_count 192 -c wincacheu_file_and_user,opcache,cli -debug_pack php-debug-pack-5.5-nts-windows-vc11-x86-rdb61685 php-5.5-nts-windows-vc11-x86-rdb61685 php-test-pack-5.5-ts-windows-vc11-x86-rdb61685

These options may be useful with core_list or core_all or core_named:

-no_restart_all - will not restart any web server. this allows the web server to be stressed more and for longer amounts of time (this will invalidate any functional testing results!). -run_test_times_all - runs each test N times in a row/consecutively -run_test_times_list - runs tests in that list N times. if used with -run_test_times_all, tests not in list can be run different number of times from tests in list (ex: run listed tests 5 times, run all other tests 2 times). -debug_list - runs tests in list in Debugger -run_group_times_all - runs all groups of tests N times (in same order every time, unless -randomize used) -run_group_times_list - like grun_group_times_All and run_test_times_list (but for groups of tests) -no_thread_safety - runs tests in any thread, regardless of thread-safety. This can increase load/stress, but may lead to false FAILS/ERRORs, especially in file or database tests.

Crash Reporting

When a PHP (or Apache) process crashes, PFTT will try running it a second time to confirm that that specific test really caused a crash before reporting the test as CRASH.

Cleaning up an interrupted Test Run (Windows)

Use taskkill /im:php.exe /f /t or taskkill /im:httpd.exe /f /t to kill off remaining php or apache processes

Check net use and use net use z: /delete to take care of remaining remote file systems

Windows PHP Extensions

WinCacheU

Run WinCacheU, with file-cache and user-cache, only file-cache, only user-cache and with opcache and with no code cache (all on CLI)

> ca -c wincacheu_file_and_user,wincacheu_only_user,wincacheu_only_file,opcache,no_code_cache,cli php-5.5.6RC1-nts-Win32-VC11-x86 php-test-pack-5.5.6RC1
MS SQL Server

Edit conf/internal_example/mssql11 and save it to conf/internal/mssql11 (also optionally mssql10). Then you can run this:

> ca -c mssql11,mssql10,cli php-5.5.6RC1-nts-Win32-VC11-x86 php_sqlsvr-test-pack-2013-09-03

Note: The MSSQL extension and its test-pack are not in PHP Core. They are currently in Microsoft's Codeplex.

Unattended Automation

Be sure to use -auto console option to signal that its being run automatically. This will supress blocking warnings and change some other settings to make sure that it runs without you having to click any buttons, etc...

On Windows, be sure to run PFTT under Admistrator privileges or you'll get a bunch of UAC popups that will block automation.

License

BSD-3 Clause. See LICENSE.txt

API Documentation

Something went wrong with that request. Please try again.