New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation for the 'rules' system and the 'Scheduler' #5

Merged
merged 5 commits into from Apr 2, 2013

Conversation

Projects
None yet
3 participants
@markstos
Copy link
Contributor

markstos commented Aug 20, 2012

This is a follow-up to the pull request to my previous pull request, in which I created a patch to add sequence-only exceptions for parallel test runs, only to discover that there was an undocumented option to prove called '--rules' which already has this feature.
--rules is handled by TAP::Parser::Scheduler, and much of the parts of the code for --rules and the Scheduler were not documented since they were introduced in 2008.

Despite being difficult to discover by most people, the rules system has been well exercised over time, as it is used by t/harness in the Perl distribution, which uses a more advanced syntax for it.

The ability to mark some tests as a not parallel-ready will be a very welcome feature for users with large test suites. Indeed, it turns out it was key to speeding up the Perl testing process. With this feature available, it can be a fairly simple process to convert a complex test suite to parallel testing:

  1. Start with tests 100% passing as a baseline (or at least inventory those which are failing)
  2. Try a fully parallel test run (or 5 or 10)
  3. Note which tests fail when run in parallel
  4. Add these exceptions to your .proverc using the --rules syntax.

Profit!

There are a couple of discussion points going forward:

  1. In the little documentation that was pre-existing, the rule system was marked as "experimental" in 2008. Is it still experimental because it's been largely undocumented and undiscovered, or is it considered stable now before because it's 4 years old? I would inclined to continue to mark it as experimental until more people can discover it and provide feedback. I left it marked as such in the documentation.

  2. Now that I figured out how to use it via prove, I think the syntax for prove could use particular scrutiny to see if that's what we want to keep, or if we want to simply it further. Here's my concern. The primary use-case I see for having "rules" options in prove is to make some tests as not-parallel-ready. Yet, here's the cumbersome, non-memorable recipe for that:

    All tests are allowed to run in parallel, except those starting with "p"

    --rules='seq=t/p_.t' --rules='par=_*'

That's a pretty gross syntax for what could be a common usage. On one hand, people may copy/paste this syntax into their .proverc file and forget. On the other, I think we can do better. My initial inclination is to leave the "--rules" flag exposed for advanced uses, but support a simpler shorthand, for example, this could mean the same as the above:

# Tests starting with "p" are not parallel-ready, they must be run in sequence. 
--not-parallel='t/p*.t' 

There would be some details to work out there, like how the "simple" and advanced syntaxes would interact, but it seems worth considering.

selected to run as a test, you can "set and forget" some rules preferences in
your .proverc file. Then you'll be able to take maximum advantage of the
performance benefits of parallel testing, while some exceptions are still run
in parallel.

This comment has been minimized.

@omega

omega Aug 28, 2012

Contributor

did you mean "sequentially" here? Right now it says parallel twice in my mind :)

This comment has been minimized.

@markstos

markstos Aug 28, 2012

Contributor

Correct. Thank you!

@schwern

This comment has been minimized.

Copy link
Member

schwern commented Apr 2, 2013

I don't know anything about the rules system, but having docs is better than not having docs. Even if they're wrong, it's a good place to revise from.

schwern added a commit that referenced this pull request Apr 2, 2013

Merge pull request #5 from markstos/scheduler-rules-docs
Documentation for the 'rules' system and the 'Scheduler'

@schwern schwern merged commit 99633c2 into Perl-Toolchain-Gang:master Apr 2, 2013

@markstos markstos deleted the markstos:scheduler-rules-docs branch Dec 15, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment