Write module documentation in POD format.

_get_all_tap_files() is not needed in
lib/TAP/Harness/Archive/MultipleHarnesses.pm, as it is inherited; remove it.

lib/TAP/Harness/Archive/MultipleHarnesses.pm

@@ -120,8 +120,6 @@ sub runtests {
 
 1;
 
-__END__
-
 =head1 NAME
 
 TAP::Harness::Archive::MultipleHarnesses - Create an archive of multiple
@@ -142,144 +140,97 @@ CPAN.  It provides its own C<runtests()> method for the case where you need to
 create an archive of test results generated by running multiple harnesses
 sequentially.
 
-=head2 The Simplest Case
-
-Under normal circumstances, you supply a list of tests to a single test harness and
-run them all in the same environment.  This code is typical:
-
-    $harness->runtests(@files);
-
-=head2 Several Harnesses in Sequence
-
-Now suppose that you need to run B<several> test harnesses in sequence.  If
-all you want to do is to print their results and summaries to the console
-without any attempt to report an B<overall> summary, you can simply wrap these
-multiples harnesses in a Perl program and then -- to simplify things even
-farther -- slap a F<make> target on it.
-
-For example, in the Parrot project we call a series of harnesses with C<make
-fulltest>, which is defined as:
-
-    fulltest :
-        -make testb \
-        testf \
-        testr \
-        src_tests \
-        run_tests \
-        buildtools_tests \
-        perl_tests \
-        library_tests \
-        codetest \
-        benchmark_tests \
-        manifest_tests \
-        examples_tests \
-        distro_tests \
-        headerizer_tests
-
-Each of these individual F<make> targets is a Perl program like:
-
-    manifest_tests :
-       	$(PERL) t/harness $(MANIFEST_TEST_FILES)
-
-These individual targets do not necessarily run in the same environment.  For
-example, C<testb>, C<testf> and C<testr> run exactly the same set of tests as
-each other, but they run them in B<different> environments -- specifically,
-with different Parrot runcores.
-
-Each of these targets prints out its own summary.  No overall summary is kept.
-
-=head2 Single Harness with Multiple Subharnesses
-
-Now suppose that you wanted to run multiple harnesses but, for summary
-purposes, have them aggregated into a single overall harness.  It is possible
-to do this with a judicious mixture of methods from the Perl 5 core packages
-TAP::Harness::Formatter and TAP::Harness::Aggregator in conjunction with
-the non-core TAP::Harness::Archive.
-
-    my @targets = (
-        {
-            label   => 'headerizer',
-            rule    => sub { # subroutine to set environmental variables },
-            tests   => ( # list of headerizer tests ),
-        },
-        {
-            label   => 'testb',
-            rule    => sub { # subroutine to set different environmental variables },
-            tests   => ( # list of tests to be run with 'b' runcore ),
-        },
-    );
-    # ...
-
-    my ($formatter, $aggregator);
-    $formatter   = TAP::Formatter::Console->new( {
-        verbosity  => $ENV{HARNESS_VERBOSE},
-        jobs       => $ENV{TEST_JOBS} || 1,
-        directives => 1,
-        timer      => $ENV{HARNESS_TIMER} || 0,
-    } );
-    $aggregator = TAP::Parser::Aggregator->new;
-
-    $aggregator->start();
-    foreach my $set (@targets) {
-        # rewrite environment
-        &{$set->{rule}} if defined $set->{rule};
-        print STDERR "$set->{label}: running with: $ENV{TEST_PROG_ARGS}\n";
-        my $harness = TAP::Harness->new( { formatter => $formatter } );
-        $harness->aggregate_tests($aggregator, @{$set->{tests}});
-    }
-    $aggregator->stop();
-    $formatter->summary($aggregator);
-    # We'll treat both FAIL and NOTESTS as failures.
-    my $exit_value = ($aggregator->get_status() eq 'PASS') ? 0 : 1;
-    exit($exit_value);
+For a discussion of use cases for this functionality, see the documentation
+for TAP::Harness::ReportByDescription.
 
-This code enables us to write rules for different environmental variables for
-different individual harnesses, but keep track of overall results and print a
-single, overall summary.
+    perldoc TAP::Harness::ReportByDescription
 
-=head2 Archive of Single Harness with Multiple Subharnesses
+=head1 METHODS
 
-Now suppose that we want to make an archive of the above and send it off to a
-smoke server.  Here things get tricky, so here's where we have to overwrite
-C<TAP::Harness::Archive::runtests()>.
+=head2 C<new()>
 
-=head1 METHODS
+Inherited from Test::Harness::Archive.
 
 =head2 C<runtests()>
 
-Does B<not> take the same arguments as C<TAP::Harness::runtests()> or
-C<TAP::Harness::Archive::runtests()>.  Instead, it takes a reference to an
-array of hash references, where each hash has, at minimum, a C<tests> element
-that holds a list of tests to be run and a C<rule> element which is a
-reference to a subroutine that is executed before the corresponding list of
-tests is run.
+Replaces C<Test::Harness::Archive::runtests()>.  B<Note that its interface is
+different from other packages' C<runtests()> interface:  It takes a reference
+to an array of hash references rather than a simple array.>
 
-=cut
+Each hash reference holds information on how a particular set of tests is to
+be run.  The various sets are run and placed into the archive in the order in
+which they appear in the array.
+
+Each hash reference needs three elements:
+
+=over 4
+
+=item * C<tests>
+
+A list of tests to be run (typically expressed as a list of file glob
+patterns).
+
+=item * C<rule>
 
-sub _get_all_tap_files {
-    my ($self, $dir, $meta) = @_;
-    $dir ||= $self->{__archive_tempdir};
-    my @files;
-    my %x_files;
-    if($meta && $meta->{extra_files}) {
-        %x_files = map { $_ => 1 } @{$meta->{extra_files}};
+A reference to a subroutine which will be run before a given set of tests is
+executed.   The purpose of this subroutine is to set up the environmental
+variables as needed for a particular subharness.
+
+=item * C<label>
+
+A string describing a particular subharness which will be combined with a
+particular test file's name to form the description of the test both in STDOUT
+and in the test archive.
+
+=back
+
+=head2 C<summary()>
+
+Inherited from Test::Harness::Archive.
+
+=head1 EXAMPLE
+
+Adapted (simplified) from Parrot's C<t/fullharness>.
+
+    use Parrot::Harness::Smoke qw( collect_test_environment_data );
+    use TAP::Harness::Archive::MultipleHarnesses;
+
+    sub set_runcore_target {
+        my ($target)  = @_;
+        return {
+            label   => "test$target",
+            rule    => sub { set_runcore_environmental_args($target) },
+            tests   => [
+                        map { [ $_, "test${alt}__$_", ] } 
+                            @Parrot::Harness::TestSets::runcore_test_files
+                       ],
+        };
     }
+    my @targets = map { set_runcore_target($_) } ( qw| b f r | );
+    my %env_data = collect_test_environment_data();
+
+    my $archive = TAP::Harness::Archive::MultipleHarnesses->new( {
+        verbosity        => $ENV{HARNESS_VERBOSE},
+        archive          => 'parrot_test_run.tar.gz',
+        merge            => 1,
+        jobs             => $ENV{TEST_JOBS} || 1,
+        extra_properties => \%env_data,
+        extra_files      => [ 'myconfig', 'config_lib.pir' ],
+    } );
+    my $overall_aggregator = $archive->runtests(\@targets);
+    $archive->summary($overall_aggregator);
 
-    File::Find::find(
-        {
-            no_chdir => 1,
-            wanted   => sub {
-                return if /^\./;
-                return if -d;
-                my $rel_name = File::Spec->abs2rel($_, $dir);
-                return if $rel_name eq 'meta.yml';
-                push(@files, $rel_name) unless $x_files{$rel_name};
-            },
-        },
-        $dir
-    );
-    return @files;
-}
+=head1 AUTHOR
 
+This code was derived from Michael Peters' Test::Harness::Archive distribution
+on CPAN, as well as examples in the documentation for TAP::Harness,
+TAP::Parser, TAP::Parser::Aggregator and other CPAN modules.  Documentation
+and code assemblage by James E Keenan.
+
+=head1 LICENSE
+
+This is free software and is released under the same terms as Perl itself.
+
+=cut
 
-1;    # End of TAP::Harness::Archive::MultipleHarnesses
+# vim:ts=4:sw=4:et:sta

lib/TAP/Harness/ReportByDescription.pm

@@ -5,7 +5,128 @@ use base 'TAP::Harness';
 use File::Path;
 use File::Spec;
 use IO::Handle;
-our $VERSION = '0.01';
+use vars qw( $VERSION );
+$VERSION = '0.01';
+
+=head1 NAME
+
+TAP::Harness::ReportByDescription - Report TAP output by test file description rather than test file name
+
+=head1 VERSION
+
+0.01
+
+=head1 SYNOPSIS
+
+    use TAP::Harness::ReportByDescription;
+    my $harness = TAP::Harness::ReportByDescription->new();
+    $harness->aggregate_tests($aggregator, @tests);
+
+=head1 DESCRIPTION
+
+This package subclasses TAP::Harness for the purpose of enabling a user to
+report the TAP output for a test file by a user-provided description rather
+than by the name of the test file itself.
+
+Why would you want to do this?  Three reasons come to mind.
+
+=over 4
+
+=item * One Master Summary Rather Than Summaries for Individual Subharnesses
+
+Suppose that you had a F<make> testing target that is in essence nothing more
+than a sequential run of several smaller testing targets, each of which is a
+separate invocation of a test harness.
+
+    make fulltest : test_prep \
+        compilers \
+        src \
+        oo \
+        codingstd \
+        examples
+
+Other things being equal, you would get a B<Summary> at the end of each of the
+five targets or subharnesses.  Under some circumstances, you might prefer to
+get a single master Summary at the end of the overall program.
+
+=item * Multiple Runs of Same Tests in Different Environments
+
+Suppose that you had a set of tests that you wanted to run several times, each
+time in a slightly different environment.  You could write a program which
+executes multiple runs, writing a summary after each run and then modifying
+the environment for the next run.
+
+    perl t/harness --gc-debug --runcore=bounds
+    perl t/harness --gc-debug --runcore=fast
+    perl t/harness --gc-debug --run-pbc
+
+As the TAP output flowed by, you would see three instances of each test:
+
+    t/pmc/arrayiterator.t ............................ ok
+    # ...
+    t/pmc/arrayiterator.t ............................ ok
+    # ...
+    t/pmc/arrayiterator.t ............................ ok
+
+... but you would not be able to tell from the test file's report itself which
+harness it was a part of.  Under certain circumstances it would be nice to be
+able to differentiate among the different runs:
+
+    bounds__t/pmc/arrayiterator.t .................... ok
+    # ...
+    fast__t/pmc/arrayiterator.t ...................... ok
+    # ...
+    pbc__t/pmc/arrayiterator.t ....................... ok
+
+Here you're providing a B<description> of each run of each test which provides
+an observer with more information.
+
+=item * Preparation of a Test Harness Archive
+
+The ability to provide a specific description for a different run of the same
+test becomes crucial when preparing a test harness archive.  Currently, CPAN
+distribution Test::Harness::Archive stores the TAP for a particular test file
+in a file with the name of the test file itself.  If you do multiple runs of
+the same file in different environments, a later run of a test will overwrite
+the TAP file from an earlier run.  You would therefore only be able to include
+the TAP from the last subharness in an archive.  That would impede you from
+sharing the full results of testing via a smoke-test aggregator such as
+Smolder.
+
+=back
+
+In short, we need (a) a way to run multiple harnesses as if they were one, (b)
+run the same tests through multiple harnesses and be able to quickly identify
+which harness we were running it through, and (c) store multiple versions of a
+file's TAP output in a test harness archive.
+
+Need (a) can actually be fulfilled with existing TAP::Parser::Aggregator
+functionality.  Let's build on that to meet needs (b) and (c).  To do that we
+need one package to subclass TAP::Harness and one to subclass
+TAP::Harness::Archive.  TAP::Harness::ReportByDescription and
+TAP::Harness::Archive::MultiplesHarnesses are these packages.
+
+=head1 METHODS
+
+=head2 C<new()>
+
+Inherited from TAP::Harness.
+
+=head2 C<aggregate_tests()>
+
+Replicated, along with methods called internally from this method, from
+TAP::Harness.  The only change occurs in an internal method
+C<_get_parser_args()>, which now assigns the individual test's B<filename> to
+one variable and a user-provided B<description> to a second variable.
+
+    my $test_prog = $job->filename;
+    my $spool_prog = $job->description;
+
+It is the latter variable which will appear on the console and in a test
+archive.  Since this occurs within an internal method, the user need make no
+change in how C<aggregate_tests()> is called.
+
+=cut
 
 sub aggregate_tests {
     my ( $self, $aggregate, @tests ) = @_;
@@ -189,3 +310,20 @@ sub _open_spool {
 
 1;
 
+=head1 EXAMPLE
+
+See C<TAP::Harness::Archive::MultipleHarnesses::runtests()>.
+
+=head1 AUTHOR
+
+99% of the code in this module comes from TAP::Harness, written by Andy
+Armstrong and generations of Perl QA hackers.  Documentation and the one small
+code tweak needed were written by James E Keenan.
+
+=head1 LICENSE
+
+This is free software and is released under the same terms as Perl itself.
+
+=cut
+
+# vim:ts=4:sw=4:et:sta