Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Fetching contributors…

Cannot retrieve contributors at this time

732 lines (570 sloc) 27.433 kb
Revision history for Perl extension Test::Harness
This is the revision history for the previous version of Test::Harness
up to 2.64. The current version of test harness is a complete rewrite of
this code.
* prove's --perl=/path/to/file wasn't taking a value.
* prove's version number was not getting incremented. From now on,
prove's $VERSION will match Test::Harness's $VERSION, and I added
a test to make sure this is the case.
* Added test straps overload via HARNESS_STRAP_OVERLOAD environment
variable. prove now takes a --strap=class parameter. Thanks,
Adam Kennedy.
2.63_01 Fri Jun 30 16:59:50 CDT 2006
* Failed tests used to say "NOK x", and now say "NOK x/y".
Thanks to Will Coleda.
* Added the Test::Harness::Results object, so we have a well-defined
object, and not just a hash that we pass around. Thanks to YAPC::NA
2006 Hackathon!
2.62 Thu Jun 8 14:11:57 CDT 2006
* Restored the behavior of dying if any subtests failed. This is a
pretty crucial bug that I should have fixed long ago. Not having this
means that CPANPLUS will install modules even if their tests fail. :-(
2.60 Wed May 24 14:48:44 CDT 2006
* Fixed the headers in the summary failure table.
2.58 Sat May 13 22:53:53 CDT 2006
No changes. Released to the world with a non-beta number.
2.57_06 Sun Apr 23 00:55:43 CDT 2006
* Anything that displays a percentage of tests passed has been
removed. Output at the end of failing runs is now different.
* Fixed the TODO-passing patch from 2.57_05.
* The unnecessary display of percentages of tests passing and failing
have been removed. Tests are not a percentage game.
* Caches the results of _default_inc(), which is expensive because
of shelling out to get the pathnames. Benchmarking was showing that
15% of Test::Harness's time was spent in this function. For test
suites with many test files, this can be significant. With this
speedup, the "make test" for the Perl core speeds up 2.5%.
Thanks to Nicholas Clark for finding this.
* Fixed HARNESS_PERL_SWITCHES typo. Thanks, Andreas Koenig.
* Added docs on HARNESS_TIMER and --timer. Thanks, Mike O'Regan.
2.57_05 Wed Apr 19 00:31:10 CDT 2006
* Now shows details of the tests that unexpectedly pass, instead of
just giving a number. Thanks, demerphq!
* Fixed globbing to work under Perls before 5.6.0. Before Perl 5.6.0,
prove just uses the internal glob() function.
2.57_04 Mon Apr 17 13:35:10 CDT 2006
* prove's globbing is now done with File::Glob::bsd_glob().
Otherwise, "prove c:\program files\svk\t\*" fails because glob()
considers it to be two patterns, splitting on whitespace. Thanks to
Audrey Tang.
* Added information about other TAP implementations in other languages.
2.57_03 Dec 31 2005
* Internal functions _run_all_tests() and _show_results() no longer
exist. You shouldn't have been using them anyway since they're
prepended with underscores.
* Added the ability to send test output to a filehandle of
one's choosing. Two internal functions are now exposed:
execute_tests() and get_results() (formerly _run_all_tests() and
_show_results()). This should allow CPANPLUS to work properly
with Module::Build. Thanks to Ken Williams.
* Hid the documentation for the private methods in Test::Harness::Straps.
2.57_02 Fri Dec 30 23:51:17 CST 2005
* prove's --ext option has been removed. I'm betting that nobody used it.
* prove can now take -w and -W switches, analogous to those in perl.
This means that "prove -wlb t/*.t" is exactly the same as "make test".
Thanks to Rob Kinyon.
* Started a Test::Harness::Util module for code that may be reused
by other Harness-using modules.
* The t/prove*.t tests now use $^X to call prove. Thanks to Yves Orton.
* Test::Harness::Straps no longer uses Win32::GetShortPathName().
Thanks to Gisle Aas.
2.57_01 Mon Dec 26 01:39:07 CST 2005
* Removed code and docs mentioning HARNESS_IGNORE_EXITCODE, which
is not used anywhere.
* If we have hi-res timings, then they're shown in integer
milliseconds, rather than fractional seconds.
* Added the --perl switch to prove.
* Added links to CPAN support sites.
2.56 Wed Sep 28 16:04:00 CDT 2005
* Incorporate bleadperl patch to fix Test::Harness on VMS.
2.54 Wed Sep 28 09:52:19 CDT 2005
* Test counts were wrong, so wouldn't install on Perls < 5.8.0.
2.53_02 Thu Aug 25 21:37:01 CDT 2005
* File order in prove is now sorted within the directory. It's not
the sorting that's important as much as the deterministic results.
Thanks to Adam Kennedy and Casey West for pointing this out,
independently of each other, with 12 hours of the other.
* Fix calls to podusage() to not use the DATA typeglob. Thanks sungo.
2.53_01 Sun Jul 10 10:45:27 CDT 2005
* If we go over 100,000 tests, it used to print out a warning for
every test over 100,000. Now, we stop after the first. Thanks to
Sebastien Aperghis-Tramoni.
2.52 Sun Jun 26 23:05:19 CDT 2005
No changes
* The Test::Harness timer is now off by default. Set HARNESS_TIMER
true if you want it. Added --timer flag to prove.
* Call CORE::time() to figure out if we should print when we're
printing once per second. Otherwise, we're using Time::HiRes'
version of it. Thanks, Nicholas Clark.
2.50 Tue Jun 21 14:32:12 CDT 2005
* Added some includes in t/strap-analyze.t to make Cygwin happy.
2.49_02 Tue Jun 21 09:54:44 CDT 2005
* Added some includes in t/test_harness.t to make Cygwin happy.
2.49_01 Fri Jun 10 15:37:31 CDT 2005
* Now shows elapsed time in 1000ths of a second if Time::HiRes
is available.
* Test::Harness::Iterator didn't have a 1; at the end. Thanks to
Steve Peters for finding it.
2.48 Fri Apr 22 22:41:46 CDT 2005
Released after weeks of non-complaint.
2.47_03 Wed Mar 2 16:52:55 CST 2005
* Test::Harness now requires Perl 5.005_03 or above.
* Fixed incorrect "confused by tests in wrong order" error in 2.47_02.
2.47_02 Tue Mar 1 23:15:47 CST 2005
* Test directives for skip tests used to be anything that matches
/^skip/i, like the word "skipped", but now it must match
* T::H now sets environment variable HARNESS_VERSION, in case a test
program wants to know what version of T::H it's running under.
2.47_01 Mon Feb 21 01:14:13 CST 2005
* Fixed a problem submitted by Craig Berry:
Several of the Test::Harness tests now fail on VMS with the
following warning:
Can't find string terminator "]" anywhere before EOF at -e line 1.
The problem is that when a command is piped to the shell and that
command has a newline character embedded in it, the part after
the newline is invisible to the shell. The patch below corrects
that by escaping the newline so it is not subject to variable
interpolation until it gets to the child's Perl one-liner.
* Test::Harness::Straps now has diagnostic gathering without changing
how tests are run. It also adds these messages by default.
Note that the new method, _is_diagnostic(), is for internal
use only. It may change soon. Thanks to chromatic.
* Expanded Test::Harness::TAP.pod, and added examples.
* Fixed a crucial documentation typo in Test::Harness::Straps.
2.46 Thu Jan 20 11:50:59 CST 2005
2.45_02 Fri Dec 31 14:57:33 CST 2004
* Turns off buffering on both STDERR and STDOUT, so that the two
output handles don't get out of sync with each other. Thanks to
David Wheeler.
* No longer requires, or supports, the HARNESS_OK_SLOW environment
variable. Test counts are only updated once per second, which
used to require having HARNESS_OK_SLOW set.
2.45_01 Fri Dec 17 22:39:17 CST 2004
* Test::Harness now requires Perl 5.004_05.
* We no longer try to print a stack if a coredump is detected.
* Reverted Test::Harness::Iterator::next()'s use of readline, since
it fails under Perl 5.5.4.
* We no longer try to print a stack if a coredump is detected.
This means that the external problems we've had with
now disappear. This resolves a number of problems that various
Linux distros have, and closes a couple of RT tickets like #2729
and #7716.
* Added Test::Harness->strap() method to access the internal strap.
* Obfuscated the email address. The damage is already
done, but at least we'll have it hidden going forward.
2.44 Tue Nov 30 18:38:17 CST 2004
* De-anonymized the callbacks and handlers in Test::Harness, mostly
so I can profile better.
* Checks _is_header() only if _is_line() fails first. No point
in checking every line of the input for something that can only
occur once.
* Inline the _detailize() function, which was getting called once
per line of input. Reduced execution time about 5-7%.
* Removed unnecessary temporary variables in Test::Harness::Straps
and in Test::Harness::Iterator.
2.43_02 Thu Nov 25 00:20:36 CST 2004
* Added more debug output if $Test::Harness::Debug is on.
* Test::Harness now removes default paths from the paths that it
sets in PERL5LIB. This fixes RT #5649. Thanks, Schwern.
* Test::Harness::Straps' constructor no longer will work as an
object method. You can't say $strap->new any more, but that's
OK because you never really wanted to anyway.
* Added workaround for local $ENV{} bug on Cygwin to
t/prove-switches.t. See the following RT tickets for details.
2.42 Wed Apr 28 22:13:11 CDT 2004
* prove -v now sets TEST_VERBOSE in case your tests rely on them.
* prove globs the command line, since Win32's shell doesn't.
* Cross-platform test fixes on t/prove-globbing.t
2.40 Tue Dec 30 20:38:59 CST 2003
* Test::Harness::Straps should now properly quote on VMS.
* prove now takes a -l option to add lib/ to @INC. Now when you're
building a module, you don't have to do a make before you run
the prove. Thanks to David Wheeler for the idea.
* Internal functions corestatus() and canonfailed() prepended with
underscores, to indicate such.
* Gratuitous text-only changes in Test::Harness::Iterator.
* All tests now do their use_ok() in a BEGIN block. Some of the
use_ok() calls were too much of a hassle to put into a BEGIN block,
so I changed them to regular use calls.
2.38 Mon Nov 24 22:36:18 CST 2003
Released. See changes below.
2.37_03 Tue Nov 18 23:51:38 CST 2003
* prove -V now shows the Perl version being used.
* Now there's a HARNESS_DEBUG flag that shows diagnostics as the
harness runs the tests. This is different from HARNESS_VERBOSE,
which shows test output, but not information about the harness
* Added _command_line() to the Strap API.
* Bad interaction with Module::Build: The strap was only checking
$ENV{HARNESS_PERL_SWITCHES} for definedness, but not emptiness.
It now also strips any leading or trailing whitesapce from the
* Test::Harness and prove only quote those parms that actually need
to be quoted: Have some whitespace and aren't already quoted.
2.36 Fri Nov 14 09:24:44 CST 2003
* t/prove-includes.t properly ignores PROVE_SWITCHES that you may
already have set.
2.35_02 Thu Nov 13 09:57:36 CST 2003
* prove's --blib now works just like the blib pragma.
2.35_01 Wed Nov 12 23:08:45 CST 2003
* Fixed taint-handling and path preservation under MacOS. Thanks to
Schwern for the patch and the tests.
* Preserves case of -t or -T in the shebang line of the test.
* Added -t to prove analogous to Perl's -t. Removed the --taint
* prove can take default options from the PROVE_SWITCHES variable.
* Added HARNESS_PERL to allow you to specify the Perl interpreter
to run the tests as.
* prove's --perl switch sets the HARNESS_PERL on the fly for you.
* Quotes the switches and filename in the subprogram. This helps
with filenames with spaces that are subject to shell mangling.
2.34 Sat Nov 8 22:09:15 CST 2003
* Allowed prove to run on Perl versions < 5.6.0.
* Command-line switches to prove may now be stacked.
* Added check for proper Pod::Usage version.
* "make clean" does a better job of cleaning up after itself.
2.32 Fri Nov 7 09:41:21 CST 2003
Test::Harness now includes a powerful development tool to help
programmers work with automated tests. The prove utility runs
test files against the harness, like a "make test", but with many
* prove is designed as a development tool
Perl users typically run the test harness through a makefile via
"make test". That's fine for module distributions, but it's
suboptimal for a test/code/debug development cycle.
* prove is granular
prove lets your run against only the files you want to check.
Running "prove t/live/ t/master.t" checks every *.t in t/live, plus
* prove has an easy verbose mode
To get full test program output from "make test", you must set
"HARNESS_VERBOSE" in the environment. prove has a "-v" option.
* prove can run under taint mode
prove's "-T" runs your tests under "perl -T".
* prove can shuffle tests
You can use prove's "--shuffle" option to try to excite problems
that don't show up when tests are run in the same order every time.
* Not everything is a module
More and more users are using Perl's testing tools outside the
context of a module distribution, and may not even use a makefile at
Prove requires Pod::Usage, which is standard after Perl 5.004.
I'm very excited about prove, and hope that developers will begin
adopting it to their coding cycles. I welcome your comments at
There are also some minor bug fixes in Test::Harness itself, listed
below in the 2.31_* notes.
2.31_05 Thu Nov 6 14:56:22 CST 2003
- If a MacPerl script had a shebang with -T, the -T wouldn't get
passed as a switch.
- Removed the -T on three *.t files, which didn't need them, and
which were causing problems.
- Conditionally installs bin/prove, depending on whether Pod::Usage
is available, which prove needs.
- Removed old leftover code from Makefile.PL.
2.31_04 Mon Nov 3 23:36:06 CST 2003
Minor tweaks here and there, almost ready to release.
2.31_03 Mon Nov 3 08:50:36 CST 2003
- prove is almost feature-complete. Removed the handling of
--exclude for excluding certain tests. It may go back in the
- prove -d is now debug. Dry is prove -D.
2.31_02 Fri Oct 31 23:46:03 CST 2003
- Added many more switches to prove: -d for dry run, and -b for
- T:H:Straps now recognizes MSWin32 in $^0.
- RT#3811: Could do regex matching on garbage in _is_test().
Fixed by Yves Orton
- RT#3827: Strips backslashes from and normalizes @INC entries
for Win32. Fixed by Yves Orton.
- Added $self->{_is_macos} to the T:H:Strap object.
- t/test-harness.t sorts its test results, rather than relying on
internal key order.
- Added "prove" script to run a test or set of tests through the
harness. Thanks to Curtis Poe for the foundation.
- Fixed POD problem in Test::Harness::Assert
2.30 Thu Aug 14 20:04:00 CDT 2003
No functional changes in this version. It's only to make some doc
tweaks, and bump up the version number in T:H:Straps.
- Changed Schwern to Andy as the maintainer.
- Incorporated the TODO file into proper.
- Cleaned up formatting in Test::Harness::Straps.
2.29 Wed Jul 17 14:08:00 CDT 2003
- Released as 2.29.
2.28_91 Sun Jul 13 00:10:00 CDT 2003
- Added support for HARNESS_OK_SLOW. This will make a significant
speedup for slower connections.
- Folded in some changes from bleadperl that spiff up the
failure reports.
- Added some isa_ok() checks to the tests.
- All Test::Harness* modules are used by use_ok()
- Fixed the prototype for the canonfailed() function, not that
it matters since it's never called without parens.
2.28_90 Sat Jul 05 20:21:00 CDT 2003
- Now, when you run a test harnessed, the numbers don't fly by one
at a time, one update per second. This significantly speeds
up the run time for running thousands of tests. *COUGH*
Regexp::Common *COUGH*
2.28 Thu Apr 24 14:39:00 CDT 2003
- No functional changes.
2.27_05 Mon Apr 21 15:55:00 CDT 2003
- No functional changes.
- Fixed circular depency in the test suite. Thanks, Rob Brown.
2.27_04 Sat Apr 12 21:42:00 CDT 2003
- Added test for $Test::Harness::Switches patch below.
2.27_03 Thu Apr 03 10:47:00 CDT 2003
- Fixed straps not respecting $Test::Harness::Switches. Thanks
to Miyagawa for the patch.
- Added t/pod.t to test POD validity.
2.27_02 Mon Mar 24 13:17:00 CDT 2003
2.27_01 Sun Mar 23 19:46:00 CDT 2003
- Handed over to Andy Lester for further maintenance.
- Fixed when the path to perl contains spaces on Windows
* Stas Bekman noticed that tests with no output at all were
interpreted as passing
- MacPerl test tweak for busted exit codes (bleadperl 17345)
- Abigail and Nick Clark both hit the 100000 "huge test that will
suck up all your memory" limit with legit tests. Made the check
smarter to allow large, planned tests to work.
- Partial fix of stats display when a test fails only because there's
too many tests.
- Made and WCOREDUMP anti-vommit protection more robust in
cases where loads but WCOREDUMP() pukes when run.
- Added a LICENSE.
- Ilya noticed the per test skip reason was accumlating between tests.
2.26 Wed Jun 19 16:58:02 EDT 2002
- Workaround for MacPerl's lack of a working putenv. It will never
see the PERL5LIB environment variable (perl@16942).
2.25 Sun Jun 16 03:00:33 EDT 2002
- $Strap is now a global to allow Test::Harness::Straps
- Little spelling nit in a diagnostic.
- Chris Richmond noted that the runtests() docs were wrong. It will
die, not return false, when any tests fail. This is silly, but
historically necessary for 'make test'. Docs corrected.
- MacPerl test fixes from Pudge. (mutation of bleadperl@16989)
- Undef warning introduced in 2.24 on skipped tests with no reasons
* Test::Harness now depends on File::Spec
2.24 Wed May 29 19:02:18 EDT 2002
* Nikola Knezevic found a bug when tests are completely skipped
but no reason is given it was considered a failure.
* Made Test::Harness::Straps->analyze_file & Test::Harness a bit
more graceful when the test doesn't exist.
2.23 Wed May 22 12:59:47 EDT 2002
- reason for all skip wasn't being displayed. Broken in 2.20.
- Changed the wait status tests to conform with POSIX standards.
- Quieted some SYSTEM$ABORT noise leaking out from dying test tests
on VMS.
2.22 Fri May 17 19:01:35 EDT 2002
- Fixed parsing of #!/usr/bin/perl-current to not see a -t.
(RT #574)
- Fixed exit codes on MPE/iX
2.21 Mon May 6 00:43:22 EDT 2002
- removed a bunch of dead code left over after 2.20's gutting.
- The fix for the $^X "bug" added in 2.02 has been removed. It
caused more trouble than the old bug (I'd never seen a problem
before anyway)
- 2.20 broke $verbose
2.20 Sat May 4 22:31:20 EDT 2002
* An almost complete conversion of the Test::Harness test parsing
to use Test::Harness::Straps.
2.04 Tue Apr 30 00:54:49 EDT 2002
* Changing the output format of skips
- Taking into account VMS's special exit codes in the tests.
2.03 Thu Apr 25 01:01:34 EDT 2002
* $^X fix made safer.
- Noise from loading to analyze core files supressed
- MJD found a situation where a test could run Test::Harness
out of memory. Protecting against that specific case.
- Made the 1..M docs a bit clearer.
- Fixed TODO tests so Test::Harness does not display a NOK for
- Test::Harness::Straps->analyze_file() docs were not clear as to
its effects
2.02 Thu Mar 14 18:06:04 EST 2002
* Ken Williams fixed the long standing $^X bug.
* Fixed a bug where Test::Harness::Straps was considering a test that
is ok but died as passing.
- Added the exit and wait codes of the test to the
analyze_file() results.
2.01 Thu Dec 27 18:54:36 EST 2001
* Added 'passing' to the results to tell you if the test passed
* Added Test::Harness::Straps example (examples/mini_harness.plx)
* Header-at-end tests were being interpreted as failing sometimes
- The 'skip_all' results from analyze* was not being set
- analyze_fh() and analyze_file() now work more efficiently, reading
line-by-line instead of slurping as before.
2.00 Sun Dec 23 19:13:57 EST 2001
- Fixed a warning on VMS.
- Removed a little unnecessary code from analyze_file()
- Made sure filehandles are getting closed
- analyze() now considers "not \nok" to be a failure (VMSism)
but Test::Harness still doesn't.
2.00_05 Mon Dec 17 22:08:02 EST 2001
* Wasn't filtering @INC properly when a test is run with -T, caused the
command line to be too long on VMS. VMS should be 100% now.
- Little bug in the skip 'various reasons' logic.
- Minor POD nit in 5.004_04
- Little speling mistak
2.00_04 Sun Dec 16 00:33:32 EST 2001
* Major Test::Harness::Straps doc bug.
2.00_03 Sat Dec 15 23:52:17 EST 2001
* First release candidate
* 'summary' is now 'details'
* Test #1 is now element 0 on the details array. It works out better
that way.
* analyze_file() is more portable, but no longer taint clean
* analyze_file() properly preserves @INC and handles -T switches
- minor mistake in the test header line parsing
1.26 Mon Nov 12 15:44:01 EST 2001
* An excuse to upload a new version to CPAN to get Test::Harness
back on the index.
2.00_00 Sat Sep 29 00:12:03 EDT 2001
* Partial gutting of the internals
* Added Test::Harness::Straps
1.25 Tue Aug 7 08:51:09 EDT 2001
* Fixed a bug with tests failing if they're all skipped
reported by Stas Bekman.
- Fixed a very minor warning in 5.004_04
- Fixed displaying filenames not from @ARGV
- Merging with bleadperl
- minor fixes to the filename in the report
- '[no reason given]' skip reason
1.24 Tue Aug 7 08:51:09 EDT 2001
- Added internal information about number of todo tests
1.23 Tue Jul 31 15:06:47 EDT 2001
- Merged in Ilya's "various reasons" patch
* Fixed "not ok 23 - some name # TODO" style tests
1.22 Mon Jun 25 02:00:02 EDT 2001
* Fixed bug with failing tests using header at end.
- Documented how Test::Harness deals with garbage input
- Turned on test counter mismatch warning
1.21 Wed May 23 19:22:53 BST 2001
* No longer considered unstable. Merging back with the perl core.
- Fixed minor nit about the report summary
- Added docs on the meaning of the failure report
- Minor POD nits fixed mirroring perl change 9176
- TODO and SEE ALSO expanded
1.20 Wed Mar 14 23:09:20 GMT 2001 by Michael G Schwern *UNSTABLE*
* Fixed and tested with 5.004!
- Added EXAMPLE docs
- Added TODO docs
- Now uneffected by -l, $\ or $,
1.19 Sat Mar 10 00:43:29 GMT 2001 by Michael G Schwern *UNSTABLE*
- More internal reworking
* Removed use of experimental /(?>...)/ feature for backwards compat
* Removed use of open(my $fh, $file) for backwards compatibility
* Removed use of Tie::StdHandle in tests for backwards compat
* Added dire warning that this is unstable.
- Added some tests from the old CPAN release
1.18 Mon Mar 5 17:35:11 GMT 2001 by Michael G Schwern
* Under new management!
* Test::Harness is now being concurrently shipped on CPAN as well
as in the core.
- Switched "our" for "use vars" and moved the minimum version back
to 5.004. This may be optimistic.
*** Missing version history to be extracted from Perl changes ***
1.07 Fri Feb 23 1996 by Andreas Koenig
- Gisle sent me a documentation patch that showed me, that the
unless(/^#/) is unnessessary. Applied the patch and deleted the block
checking for "comment" lines. -- All lines are comment lines that do
not match /^1\.\.([0-9]+)/ or /^(not\s+)?ok\b/.
- Ilyaz request to print "ok (empty test case)" whenever we say 1..0
- Harness now doesn't abort anymore if we received confused test output,
just warns instead.
1.05 Wed Jan 31 1996 by Andreas Koenig
- More updates on docu and introduced the liberality that the script
output may omit the test numbers.
1.03 Mon January 28 1996 by Andreas Koenig
- Added the statistics for subtests. Updated the documentation.
1.02 by Andreas Koenig
- This version reports a list of the tests that failed accompanied by
some trivial statistics. The older (unnumbered) version stopped
processing after the first failed test.
- Additionally it reports the exit status if there is one.
Jump to Line
Something went wrong with that request. Please try again.