Add functionality to test dependencies by changing test running order #3092
Conversation
…r, with and without dependency management.
… making it easy to reproduce results.
Codecov Report
@@ Coverage Diff @@
## test-reorder #3092 +/- ##
===============================================
Coverage 81.22% 81.22%
Complexity 2947 2947
===============================================
Files 110 110
Lines 7707 7707
===============================================
Hits 6260 6260
Misses 1447 1447
Continue to review full report at Codecov.
|
tests/Util/ConfigurationTest.php
Outdated
| $this->assertFalse($_ENV['foo']); | ||
| $this->assertEquals(1, \getenv('foo')); | ||
| $this->assertEquals($fooValue, $_ENV['foo']); | ||
| // $this->assertEquals(false, \getenv('foo')); |
There was a problem hiding this comment.
Is this commented out intentionally?
There was a problem hiding this comment.
Thanks for catching that one. I did not yet know what to do with this one, $_ENV and getenv()/putenv() are not really equivalent. Setting $_ENV does not change the UNIX environment variables, but putenv() does.
Uncommenting this one makes the test fail. Do you have a preference?
sebastianbergmann
added
the
type/enhancement
src/Framework/TestCase.php
Outdated
| /** | ||
| * @param string $order | ||
| */ | ||
| public function setTestRunningOrder($order): void |
There was a problem hiding this comment.
Please use scalar type declarations when possible (like here). Also please do not use @param annotations etc. when they provide no additional information.
src/Framework/TestSuiteIterator.php
Outdated
|
|
||
| /** | ||
| * Reorder Tests within a TestCase in such a way as to resolve as many dependencies as possible. | ||
| * The algorithm will leave the tests in original running order when it can. |
There was a problem hiding this comment.
What about cross-class dependencies? They are possible, though I rarely see them "in the wild".
There was a problem hiding this comment.
I am aware of those, yes, although I have not encountered them. They are the reason I made this change: epdenouden@36697f0
Currently the business logic isn't smart enough yet to also sort the TestSuites based on development. The first work in the form of TestSuite::getDependencies() is there.
|
Thank you, @epdenouden, for working on this. I will look into this more deeply as soon as I can. I hope we can get this into PHPUnit 7.2. |
|
@epdenouden I have moved the current state of this to a new branch: https://github.com/sebastianbergmann/phpunit/tree/test-reorder. Please rebase this pull request onto that branch and target that branch for future changes. Thanks! |
|
I am wondering whether it would make sense to move the reorder logic from Pseudo Code: <?php
class DependencyResolver extends ArrayIterator
{
public function __construct(TestSuiteIterator $tests)
{
$tests = iterator_to_array($tests);
// reorder $tests
parent::__construct($tests);
}
} |
|
@lstrojny This is probably interesting to you, too, considering "clever and smart". |
|
@sebastianbergmann Thanks for the quick replies! I will rebase and continue making the basic version clean and stable for 7.2. Good ideas for making this cleaner and more extensible. I am up for doing more coding for that. Re: the clever and fast logic: this project is actually the groundwork for something that we call the Breakfast extension, it automatically sorts the tests to run skipped, failed, etc first, but keeps their dependencies working. It also measures test and suite timings to make better sorting predictions. |
|
The "breakfast" thing sounds interesting. Lets get this sorted first, then have a look at that. Maybe it can also be integrated into the "core", maybe we can put it into an extension that uses the new, yet-to-be-added hook. |
Inspiration
Are your unit tests truly independent units?
In the past I have often been bitten by test suites that do not test what they claim to test. A common reason is hidden dependencies in the runtime context. Worst case this has caused applications to fail when deployed in production environments.
Changing the order of unit tests turns out to be a simple and effective way of bringing these assumptions out in the open.
Thanks to @Crazybus and @geogdog for providing real-world scenarios and feedback during development.
Summary of functionality
The test ordering functionality can be configured using command line parameters:
--reverse-orderrun tests in reverse order--random-orderrun tests in random order--random-order-seed=<number>fix the random order, allowing random runs to be reproduced--ignore-dependenciesdisable the bundled dependency resolverDesign decisions
Low impact, high performance
The feature will only wake up when there is work to do. The iterative dependency resolver is designed to do as little work as possible, using few iterations and minimal memory. It uses
@dependsannotations to keep as many dependent tests as possible in the required order.No unnecessary extra output is generated. Only the randomizer outputs the random seed using the default results printer.
Clean abstraction
The reordering is built into the main test runner and only changes the order of the tests of each test suite. Other parts of PHPUnit are not affected, with the exception of a bit of configuration handling.
Changes to the built-in testsuite
MultiDependencyTestand updated existing tests to match--reverse-order,--random-orderand--ignore-dependencies@dependsfor testGlobalsBackupPre() to PHPUnit's own testsuiteFuture work
phpunit.xmlin addition to the command line flags