Add --default-time-limit

[epdenouden]

Implementation for #2085

This is a convenience feature for people managing large(r) and potentially slow legacy test collections. Classic example: tests for database integration that have huge @dataprovider sets or when using PHPUnit to test webservices/APIs.

Note: I tried running around with my laptop, however I did not manage to validate the timer at relativistic speeds.

[codecov-io]

Codecov Report

Merging #3224 into master will decrease coverage by 0.07%.
The diff coverage is 54.54%.

@@             Coverage Diff              @@
##             master    #3224      +/-   ##
============================================
- Coverage     82.69%   82.61%   -0.08%     
- Complexity     3517     3528      +11     
============================================
  Files           143      143              
  Lines          9294     9315      +21     
============================================
+ Hits           7686     7696      +10     
- Misses         1608     1619      +11

Impacted Files	Coverage Δ	Complexity Δ
src/Util/Configuration.php	97.48% <100%> (+0.01%)	179 <0> (+1)	⬆️
src/TextUI/Command.php	71.45% <33.33%> (-0.2%)	206 <0> (+1)
src/TextUI/TestRunner.php	66.4% <44.44%> (-0.32%)	281 <0> (+6)
src/Framework/TestResult.php	75.49% <50%> (-0.45%)	168 <1> (+3)
src/Util/Test.php	93.73% <0%> (-0.2%)	220% <0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3570ac1...1439026. Read the comment docs.

[reinholdfuereder]

I am not perfectly happy with that help text: (a) I personally think something like "Timeout in seconds ..." is more common? (b) Is "unsized tests" also understandable for others? How short/compact must the help text be: is "... for tests without either @small, @medium or @large" too long?
(Sorry, if this is too picky)

[epdenouden]

First of all: feel free to be as picky as you want, it was your idea after all! There is no deadline pressure from management, so we can take all the time we need to get it from 'good enough' to 'exactly what I needed'. :-)

Mentioning the three sizes of tests might not fit. I do agree that using timeout would be nice. The config flag is already called *-time-limit and adding timeout will make it extra clear what it does. I'll commit an improved help text, let me know what you think.

[reinholdfuereder]

Thanks, I like this!

[epdenouden]

@sebastianbergmann Getting the diff coverage above the threshold would require php-invoker or adding a fake Invoker. For such a small feature in the TestRunner, are you okay with it as-is?

[karmakoder]

@epdenouden @sebastianbergmann @reinholdfuereder any update here? This change makes a lot of sense for our legacy phpunit suites and a merge will be much appreciated. Even fallback of all non-annotated tests to @Small is something that will do a huge favor.

[epdenouden]

@karmakoder No news from my side. I agree it would be nice to see it merged.

[bc-garybottom]

@sebastianbergmann huge +1 here as well, working with our php code base for functional and unit tests, particularly for UI issues having this functionality would be huge. Running in remote CI/CD SaaS solutions we have the need to gracefully timeout more and more to achieve automation autonomy without chasing down red herrings in our tests or CI/CD tool. We're currently using phpunit 6.0.13 for our test suite, for reference, which works great except having the ability to skip tests that are idle to rerun them later with this functionality.

[epdenouden]

@bc-garybottom This pull request is for version 7. If it gets merged I'd be happy to backport it to 6.x for you if @sebastianbergmann would accept a PR.

[sebastianbergmann]

Sorry, but no new features for PHPUnit 6.

[karmakoder]

@epdenouden I tried using your changes for this PR on my local copy of phpunit version 6.0.13 to create the intended behavior but somehow I dont see the tests timing/erroring out after defaultTimeLimit value that I defined in my config file. Any idea what I may be missing here?
I did see errorMsg like the following when I was missing php-invoker package:
Error: Package phpunit/php-invoker is required for enforcing time limits
but dont see expected msg like
Execution aborted after 1 second
Thx in advance.

[karmakoder]

NM. Figured what was wrong with my local setup. All good.

[epdenouden]

@karmakoder No problem. Let me know if you run into bugs or edge cases I didn't think of yet.

[reinholdfuereder]

@karmakoder Ad "Any update here?": Not really

• there is still my small pending review request (also mentioned/outlined in Configurable default for --enforce-time-limit #2085 (comment))
• and actually based on your experience mentioned in Add --default-time-limit #3224 (comment) one suggestion: what about replacing the writeMessage with a more defensive failing of the test?

[epdenouden]

@reinholdfuereder Apologies for not engaging with your recent comment, missed that completely! Let me give that a think.

[epdenouden]

@sebastianbergmann if you have time to look at the PR during the Code Sprint, that'd be lovely 🔬

[sebastianbergmann]

New attributes should be private.

[epdenouden]

Fixing

[sebastianbergmann]

For consistency, I would rather not add an option like --default-time-limit to set the default time limit on the CLI. The timeouts for @small, @medium, and @large can also only be configured in the configuration file.

[epdenouden]

Yes, that is true. I asked @reinholdfuereder what the exact use case would be and this time limit is meant to be for tests that are not marked with a @size: #2085 (comment)

I know a bunch of older test collections with legacy would benefit from this.

[localheinz]

Not sure about the trailing=, what do you think?

[epdenouden]

The = is required for --key=value, right? I have not found a clean way of doing --key[=value].

[sebastianbergmann]

The = means that the option requires a value.

[localheinz]

Thank you for clarifying, @sebastianbergmann!

[epdenouden]

@sebastianbergmann Thanks for merging this!

[karmakoder]

@epdenouden After doing several rounds of testing, am learning that this default-time-limit doesnt always take effect for our tests. I basically backported your changes in to phpunit-6.0.13 but the behavior of tests getting marked as failed doesnt seem to be working 100% of the time. Any pointers on what could be the reason? Unfortunately, we cannot upgrade to latest phpunit version as several of our dependencies don't yet support higher versions of php and phpunit. If it helps, we can connect offline to discuss this. Thx in advance.