Missing documentation for 'rules' and 'scheduler' in Test::Harness App::Prove

Here's a list of places where documentation "scheduling" and "rules" in TAP::Harness / App::Prove is known to be missing. It can be serve as a To-Do list to remedy the situation.

prove

• prove has a --rules flag, but is not documented. --rules can repeated multiple times and appears that it expects values like this: ** par=GLOB - indicates that the following file path, with glob-style wildcards, is able to run in parallel ** seq=GLOB - indicates that the following file path, with glob-style wildcards, must be run serially.
• App::Prove mentions 'rules' without further documentation. However, it's not clear that more is necessary here, if prove is properly documented.

TAP::Parser::Scheduler

• new() should document that it takes 2 arguments
  • 'tests' - I think it's an arrayref of 2 element arrays, where the first element is the test name, and the second element is the test description.
  • 'rules' - a hashref of "rule" data, defaulting to { par => '**' }.
    • An explanation of the data structure should be linked to, if not included.
    • The data structure can be recursively defined.
• The as_string() should include a simple of the output generated.
• Should link to TAP::Parser::Scheduler::Spinner and TAP::Parser::Scheduler::Job
• When the docs say "next job", I think they mean a TAP::Parser::Scheduler::Job object. This should be clarified.

"rules" data structure

The data structure can be recursively defined.

From the source code:

 # SCHEDULER-DATA ::= JOB
 #                ||  ARRAY OF ARRAY OF SCHEDULER-DATA
 #
 # The nested arrays are the key to scheduling. The outer array contains
 # a list of things that may be executed in parallel. Whenever an
 # eligible job is sought any element of the outer array that is ready to
 # execute can be selected. The inner arrays represent sequential
 # execution. They can only proceed when the first job is ready to run.

Examples:

    # From TAP::Harness
    $harness->rules(
        {   par => [
                { seq => '../ext/DB_File/t/*' },
                { seq => '../ext/IO_Compress_Zlib/t/*' },
                { seq => '../lib/CPANPLUS/*' },
                { seq => '../lib/ExtUtils/t/*' },
                '*'
            ]
        }
    );

    # From t/scheduler.t
    my $incomplete_rules = { par => [ { seq => [ '*A', '*D' ] } ] };

    # From t/scheduler.t
    my $rules = {
        par => [
            { seq => 'A*' },
            { par => 'B*' },
            { seq => [ 'C1', 'C2' ] },
            {   par => [
                    { seq => [ 'C3', 'C4', 'C5' ] },
                    { seq => [ 'C6', 'C7', 'C8' ] }
                ]
            },
            {   seq => [
                    { par => ['D*'] },
                    { par => ['E*'] }
                ]
            },
        ]
    };

glob rules

• ** is any number of characters, including /, within a pathname
• • is zero or more characters within a filename/directory name
• ? is exactly one character within a filename/directory name
• {foo,bar,baz} is any of foo, bar or baz.
• \ is an escape character

TAP::Parser::Scheduler::Job

• new() contains a third argument, "context", which is not document
• The "$name" argument to new() should be clarified to be a file name.
• The docs refer to every method as a "Class Method". This appears to be a mistake. Only new() appears to be a class method.
• The "filename", "description", and "context" methods are all undocumented. They appear to be "Attributes".

TAP::Parser::Scheduler::Spinner

• Only new() is a class method

TAP::Harness

• The 'rules' data structure is shown but not documented. ** Do rules specify tests to run, or do they only specify "rules" about how to handle tests if the happen to be encountered?

Examples of mixing parallel and serial testing with prove

These examples are fittingly made to run on the Test-Harness test suite.

Run in serial by default, with some exceptions in parallel (2x):

prove -l --rules='par=t/s*.t' -j2 t/*.t

This works because any unmatched tests are run in serial at the end.

Run in parallel (2x) by default, with exceptions in serial:

prove --merge  --rules='seq=t/p*.t' --rules='par=**' -j2 t/*.t

The 'seq' rule must comes first, because of the "first-match-wins" rule. The parallel rule for a default match is necessary because specifying any rules overrides the default of "all parallel", so you have restore it.

Q & A

• Q: What if a particular test has no rule?
  • A: It still gets run. A source code comments explains: "If any tests are left add them as a sequential block at the end of the run." Example:
    • prove --rules='seq=t/p*.t' --rules='par=t/p*.t' -j2 t/regression.t
• Q: What if a particular test has conflicting rules, telling it both to run in parallel and in serial?
  • A: The first matching rule "wins"
    • This gets run in serial: prove --rules='seq=t/p*.t' --rules='par=t/p*.t' -j2 t/p*.t
    • This runs in parallel: prove --rules='par=t/p*.t' --rules='seq=t/p*.t' -j2 t/p*.t
• Q: What does the default rule set look like?
  • A: By default, all tests are eligible for running in parallel.
• Q: What does the default rule set look like with -j2 (Does jobs affect rules)?
  • A: There is no change.
• Q: How does the GLOB matching syntax work?

Tip: Viewing and understanding generated rule plans.

If you are uncertain what "rule plan" you've ended up for your tests, you can add this diagnostic just before the call to "return" in TAP::Parser::Scheduler::new():

warn $self->as_string;

The plan format can be confusing to read, but here's how I would explain it:

• All the "seq" entries below a "par" entry are eligible to be run in parallel.
• All file entries below a 'seq' entry must be run in sequence.