Documentation for the 'rules' system and the 'Scheduler' #5

Merged
merged 5 commits into from Apr 2, 2013

2 participants

@markstos

This is a follow-up to the pull request to my previous pull request, in which I created a patch to add sequence-only exceptions for parallel test runs, only to discover that there was an undocumented option to prove called '--rules' which already has this feature.
--rules is handled by TAP::Parser::Scheduler, and much of the parts of the code for --rules and the Scheduler were not documented since they were introduced in 2008.

Despite being difficult to discover by most people, the rules system has been well exercised over time, as it is used by t/harness in the Perl distribution, which uses a more advanced syntax for it.

The ability to mark some tests as a not parallel-ready will be a very welcome feature for users with large test suites. Indeed, it turns out it was key to speeding up the Perl testing process. With this feature available, it can be a fairly simple process to convert a complex test suite to parallel testing:

  1. Start with tests 100% passing as a baseline (or at least inventory those which are failing)
  2. Try a fully parallel test run (or 5 or 10)
  3. Note which tests fail when run in parallel
  4. Add these exceptions to your .proverc using the --rules syntax.

Profit!

There are a couple of discussion points going forward:

  1. In the little documentation that was pre-existing, the rule system was marked as "experimental" in 2008. Is it still experimental because it's been largely undocumented and undiscovered, or is it considered stable now before because it's 4 years old? I would inclined to continue to mark it as experimental until more people can discover it and provide feedback. I left it marked as such in the documentation.

  2. Now that I figured out how to use it via prove, I think the syntax for prove could use particular scrutiny to see if that's what we want to keep, or if we want to simply it further. Here's my concern. The primary use-case I see for having "rules" options in prove is to make some tests as not-parallel-ready. Yet, here's the cumbersome, non-memorable recipe for that:

    # All tests are allowed to run in parallel, except those starting with "p"
    --rules='seq=t/p.t' --rules='par=*'

That's a pretty gross syntax for what could be a common usage. On one hand, people may copy/paste this syntax into their .proverc file and forget. On the other, I think we can do better. My initial inclination is to leave the "--rules" flag exposed for advanced uses, but support a simpler shorthand, for example, this could mean the same as the above:

# Tests starting with "p" are not parallel-ready, they must be run in sequence. 
--not-parallel='t/p*.t' 

There would be some details to work out there, like how the "simple" and advanced syntaxes would interact, but it seems worth considering.

@omega omega commented on the diff Aug 28, 2012
bin/prove
@@ -264,6 +265,61 @@ The C<--state> switch may be used more than once.
$ prove -b --state=hot --state=all,save
+=head2 --rules
+
+The C<--rules> option is used to control which tests are run sequentially and
+which are run in parallel, if the C<--jobs> option is specified. The option may
+be specified multiple times, and the order matters.
+
+The most practical use is likely to specify that some tests are not
+"parallel-ready". Since mentioning a file with --rules doens't cause it to
+selected to run as a test, you can "set and forget" some rules preferences in
+your .proverc file. Then you'll be able to take maximum advantage of the
+performance benefits of parallel testing, while some exceptions are still run
+in parallel.
@omega
omega added a line comment Aug 28, 2012

did you mean "sequentially" here? Right now it says parallel twice in my mind :)

@markstos
markstos added a line comment Aug 28, 2012

Correct. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@schwern
Perl Toolchain Gang member

I don't know anything about the rules system, but having docs is better than not having docs. Even if they're wrong, it's a good place to revise from.

@schwern schwern merged commit 99633c2 into Perl-Toolchain-Gang:master Apr 2, 2013
@markstos markstos deleted the markstos:scheduler-rules-docs branch Dec 15, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment