Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functionality to test dependencies by changing test running order #3092

Closed
wants to merge 86 commits into from

Conversation

@epdenouden
Copy link
Collaborator

@epdenouden epdenouden commented Apr 12, 2018

Note: updated documentation
There is a wiki page with more documentation with examples while the official docs are updated.

Inspiration

Are your unit tests truly independent units?

In the past I have often been bitten by test suites that do not test what they claim to test. A common reason is hidden dependencies in the runtime context. Worst case this has caused applications to fail when deployed in production environments.

Changing the order of unit tests turns out to be a simple and effective way of bringing these assumptions out in the open.

Thanks to @Crazybus and @geogdog for providing real-world scenarios and feedback during development.

Summary of functionality

The test ordering functionality can be configured using command line parameters:

  • --reverse-order run tests in reverse order
  • --random-order run tests in random order
  • --random-order-seed=<number> fix the random order, allowing random runs to be reproduced
  • --ignore-dependencies disable the bundled dependency resolver

Design decisions

Low impact, high performance

The feature will only wake up when there is work to do. The iterative dependency resolver is designed to do as little work as possible, using few iterations and minimal memory. It uses @depends annotations to keep as many dependent tests as possible in the required order.

No unnecessary extra output is generated. Only the randomizer outputs the random seed using the default results printer.

Clean abstraction

The reordering is built into the main test runner and only changes the order of the tests of each test suite. Other parts of PHPUnit are not affected, with the exception of a bit of configuration handling.

Changes to the built-in testsuite

  • added a fourth MultiDependencyTest and updated existing tests to match
  • added new TextUI tests for --reverse-order, --random-order and --ignore-dependencies
  • added @depends for testGlobalsBackupPre() to PHPUnit's own testsuite

Future work

  • update the main PHPUnit documentation to reflect the changes in (optional) order and dependency handling
  • add configuration options for phpunit.xml in addition to the command line flags
  • add options and/or annotations for finer grained reordering config (e.g. "never reorder this suite")
@codecov-io
Copy link

@codecov-io codecov-io commented Apr 12, 2018

Codecov Report

Merging #3092 into test-reorder will not change coverage.
The diff coverage is 96.57%.

Impacted file tree graph

@@               Coverage Diff               @@
##             test-reorder    #3092   +/-   ##
===============================================
  Coverage           81.22%   81.22%           
  Complexity           2947     2947           
===============================================
  Files                 110      110           
  Lines                7707     7707           
===============================================
  Hits                 6260     6260           
  Misses               1447     1447
Impacted Files Coverage Δ Complexity Δ
src/Framework/TestSuiteIterator.php 100% <ø> (ø) 9 <0> (ø) ⬇️
src/Util/TestDox/NamePrettifier.php 97.61% <100%> (ø) 22 <0> (ø) ⬇️
src/Util/TestDox/CliTestDoxPrinter.php 88.09% <100%> (ø) 28 <0> (ø) ⬇️
src/Framework/DataProviderTestSuite.php 100% <100%> (ø) 3 <2> (ø) ⬇️
src/TextUI/Command.php 68.02% <100%> (ø) 192 <0> (ø) ⬇️
src/Framework/TestCase.php 67.86% <100%> (ø) 307 <1> (ø) ⬇️
src/TextUI/TestRunner.php 62.47% <100%> (ø) 255 <0> (ø) ⬇️
src/Util/TestDox/RunningTestResult.php 94.64% <94.64%> (ø) 24 <24> (ø) ⬇️
src/Runner/TestSuiteSorter.php 95.74% <95.74%> (ø) 23 <23> (ø) ⬇️
src/Util/TestDox/BootstrappingTestResult.php 96.29% <96.29%> (ø) 8 <8> (ø) ⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2a1231...248009b. Read the comment docs.

$this->assertFalse($_ENV['foo']);
$this->assertEquals(1, \getenv('foo'));
$this->assertEquals($fooValue, $_ENV['foo']);
// $this->assertEquals(false, \getenv('foo'));

This comment has been minimized.

@sebastianbergmann

sebastianbergmann Apr 12, 2018
Owner

Is this commented out intentionally?

This comment has been minimized.

@epdenouden

epdenouden Apr 12, 2018
Author Collaborator

Thanks for catching that one. I did not yet know what to do with this one, $_ENV and getenv()/putenv() are not really equivalent. Setting $_ENV does not change the UNIX environment variables, but putenv() does.

Uncommenting this one makes the test fail. Do you have a preference?

/**
* @param string $order
*/
public function setTestRunningOrder($order): void

This comment has been minimized.

@sebastianbergmann

sebastianbergmann Apr 12, 2018
Owner

Please use scalar type declarations when possible (like here). Also please do not use @param annotations etc. when they provide no additional information.

This comment has been minimized.

@epdenouden

epdenouden Apr 12, 2018
Author Collaborator

Fixed


/**
* Reorder Tests within a TestCase in such a way as to resolve as many dependencies as possible.
* The algorithm will leave the tests in original running order when it can.

This comment has been minimized.

@sebastianbergmann

sebastianbergmann Apr 12, 2018
Owner

What about cross-class dependencies? They are possible, though I rarely see them "in the wild".

This comment has been minimized.

@epdenouden

epdenouden Apr 12, 2018
Author Collaborator

I am aware of those, yes, although I have not encountered them. They are the reason I made this change: epdenouden@36697f0

Currently the business logic isn't smart enough yet to also sort the TestSuites based on development. The first work in the form of TestSuite::getDependencies() is there.

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Apr 12, 2018

Thank you, @epdenouden, for working on this. I will look into this more deeply as soon as I can. I hope we can get this into PHPUnit 7.2.

@sebastianbergmann sebastianbergmann added this to the PHPUnit 7.2 milestone Apr 12, 2018
@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Apr 12, 2018

@epdenouden I have moved the current state of this to a new branch: https://github.com/sebastianbergmann/phpunit/tree/test-reorder.

Please rebase this pull request onto that branch and target that branch for future changes. Thanks!

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Apr 13, 2018

I am wondering whether it would make sense to move the reorder logic from TestSuiteIterator into separate classes, one for each type of reorder (random, dependencies, ...).

Pseudo Code:

<?php
class DependencyResolver extends ArrayIterator
{
    public function __construct(TestSuiteIterator $tests)
    {
        $tests = iterator_to_array($tests);

        // reorder $tests

        parent::__construct($tests);
    }
}
@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Apr 13, 2018

I am wondering whether it would make sense to add an extension hook for test reordering. This would make @padraic and everyone interested in #2660 happy.

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Apr 13, 2018

@lstrojny This is probably interesting to you, too, considering "clever and smart".

@epdenouden
Copy link
Collaborator Author

@epdenouden epdenouden commented Apr 13, 2018

@sebastianbergmann Thanks for the quick replies! I will rebase and continue making the basic version clean and stable for 7.2. Good ideas for making this cleaner and more extensible. I am up for doing more coding for that.

Re: the clever and fast logic: this project is actually the groundwork for something that we call the Breakfast extension, it automatically sorts the tests to run skipped, failed, etc first, but keeps their dependencies working. It also measures test and suite timings to make better sorting predictions.

@sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented Apr 13, 2018

The "breakfast" thing sounds interesting. Lets get this sorted first, then have a look at that. Maybe it can also be integrated into the "core", maybe we can put it into an extension that uses the new, yet-to-be-added hook.

@keradus
Copy link
Contributor

@keradus keradus commented May 13, 2018

Thanks for accepting the request 👍
no need to treat this one as prio, no rush on it ;)

@epdenouden
Copy link
Collaborator Author

@epdenouden epdenouden commented May 14, 2018

Indeed thanks for merging it in. I will continue on the followup feature for rerunning failures first now

@epdenouden
Copy link
Collaborator Author

@epdenouden epdenouden commented May 14, 2018

@keradus having reordering as a default would be fun, but it would spook a lot of people that run phpunit as "any 7.x" version and suddenly things are breaking without them chainging anything. Testing frameworks are important for creating trust that is not so great :)

@keradus
Copy link
Contributor

@keradus keradus commented May 14, 2018

Indeed thanks for merging it in. I will continue on the followup feature for rerunning failures first now

Sounds like https://github.com/fiunchinho/phpunit-randomizer is obsolete, now it's time for
https://github.com/lstrojny/phpunit-clever-and-smart ? :)

having reordering as a default ...

That's why I suggested to have at least config option, thus one don't need to remember about option switch every single time.
On the other hand, if I have tests that are always passing, and then we would detect that assertSame doesnt work for string for some weird reason, I do expect assertSame to be fixed and yes, my tests would start failing. It is not that my test was working and suddenly it stopped. It's that my tests were not able to detect real issue in my application, and after update of testing framework they are. And it's super cool to know that my src has bugs.

@epdenouden epdenouden deleted the epdenouden:new-order branch May 16, 2018
@Dxtan Dxtan mentioned this pull request Nov 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants