Breakfast: rerun defects first #3147

epdenouden · 2018-05-28T15:06:38Z

Use case

Rerun defective unit tests first to speed up the red-green-refactor cycle!

Test results and timings are shared between tests runs using a cache. This allows for sorting defective tests to the front of the execution order using an additional sorter. This new sort option is compatible with existing test ordering options.

This project is a followup to my previous work on reordering test execution. The code is available in my Breakfast development branch.

Summary of changes

the TestSuiteSorter gains the ability to sort suites and tests by priority of defects and execution time
added a ResultCacheExtension to gather result and timing information during test runs
added a TestResultCache for sharing of result state between runs
added caching of run results using CLI flag --cache-result and configuration attribute cacheResult='true'
added sorting defects to run as quick as possible with CLI flag --order-by=defects and configuration attribute executionOrder='defects'
ability to specifiy the cache filename using CLI flag --cache-result-file and configuration attribute cacheResultFile
more extensive testing of sorting scenarios

How to test

To activate storing the results of a run use --cache-result:
```
phpunit --cache-result
```
For this example we switch the cache on for every run by adding the attribute cacheResult="true" to the <phpunit> element in phpunit.xml.
```
<phpunit cacheResult="true"> <-- no other changes required --></phpunit>
```

Break TestCaseTest by running it backwards:

phpunit --order-by=reverse tests/Framework/TestCaseTest.php

Look at the cache file. By default this is .phpunit.result.cache in the PHPUnit working directory.

Break TestCaseTest again with the sorting feature turning on:

phpunit --order-by=defects tests/Framework/TestCaseTest.php

The skipped tests are still there but are run immediately now, but still fail. This is easily avoided by asking the sorter to keep dependencies in mind:
```
phpunit --order-by=depends,defects tests/Framework/TestCaseTest.php
```

Design considerations

Like the New Order feature the aim is for a robust system with sane defaults that silently does its work and is easily maintained by other developers:

Require as little new configuration as possible. Just add cacheResult="true" to your configuration and it silently does its work.
Add no extra output as PHPUnit often lives in automated pipelines and dev scripts.
Provide robust and maintainable code. This feature also provides a lot of new test coverage for functionality from Add functionality to test dependencies by changing test running order #3092.
Implemented a simplistic cache on purpose. Obviously people might want more complex caches based on users, groups, configuration, git hash, etc etc. Let's see what people need.

codecov-io · 2018-05-28T17:37:27Z

Codecov Report

Merging #3147 into master will increase coverage by 0.59%.
The diff coverage is 98.24%.

@@             Coverage Diff             @@
##             master   #3147      +/-   ##
===========================================
+ Coverage      81.5%   82.1%   +0.59%     
- Complexity     3412    3506      +94     
===========================================
  Files           137     140       +3     
  Lines          9014    9218     +204     
===========================================
+ Hits           7347    7568     +221     
+ Misses         1667    1650      -17

Impacted Files	Coverage Δ	Complexity Δ
src/Util/Configuration.php	`97.11% <100%> (+0.68%)`	`179 <0> (+7)`	⬆️
src/Runner/ResultCacheExtension.php	`100% <100%> (ø)`	`13 <13> (?)`
src/Util/NullTestResultCache.php	`100% <100%> (ø)`	`4 <4> (?)`
src/Runner/TestSuiteSorter.php	`100% <100%> (+6.12%)`	`43 <28> (+14)`	⬆️
src/TextUI/Command.php	`71.5% <100%> (+3.22%)`	`204 <7> (+11)`	⬆️
src/Util/TestResultCache.php	`100% <100%> (ø)`	`25 <25> (?)`
src/Framework/TestResult.php	`72.75% <100%> (+0.65%)`	`163 <1> (+5)`	⬆️
src/TextUI/TestRunner.php	`63.96% <84.61%> (+1.56%)`	`274 <0> (+15)`	⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1bde207...7dc6729. Read the comment docs.

sebastianbergmann · 2018-05-29T05:27:46Z

This branch cannot be rebased due to conflicts.
Why is the ResultCacheListener needed, or rather why is it needed to be known to the user? As this pull request is about adding functionality to the "core" of PHPUnit, there is no need to expose the usage of an extension point like that. We can simply attach the listener when we need it.
I think we should add a CLI switch and configuration option to record test results instead.
I would prefer not to use the old, bloated TestListener interface for this but rather the new hook interfaces (introduced in PHPUnit 7.1 with Towards a better plugin infrastructure #3002), if possible.

epdenouden · 2018-05-29T09:21:52Z

Thanks for the quick review, again!

Branch is fixed. The culprit was a backported TestSuiteSorterTest. Doing a git rebase instead of merge even suggested just removing a conflicting commit altogether.
I extended the public TestListener based on the example in the manual. I would also prefer to keep this internal. Will refactor into an extension that can be switched on with a command line flag, also makes testing cleaner.

keradus · 2018-05-29T10:25:43Z

src/Util/TestResultCache.php

+    /**
+     * @var string
+     */
+    public const DEFAULT_RESULT_CACHE_FILENAME = 'result_cache.json';


if I have my project, eg ~/php-cs-fixer folder, and I would run phpunit with cache enabled, it would create ~/php-cs-fixer/result_cache.json ? if so, please let us make it phpunit.cache.json or sth like this

I have no opinion on the filename of the default value. As a developer I want the following:

Simple usage that just works in a basic code, test, repeat cycle. A file in the working directory works well.

PHPUnit is part of automation: use the environment variable to do your devops and have it store the cache(s) for users/configs/build-unique-ID/git-hashes/etc however en whereever you want.

The default implementation of TestResultCache is there so the simplest use cases work out-of-the-box. The current cache is just a simple MVP. I would prefer NOT to use an external dependency like SQLite like PHPUnit Clever and Smart did, but I am starting to see why they went that way.

well, here I'm even not talking about using SQLite or anything like that. (while Indeed, i can see value of it - yet in caching of PHP CS Fixer we don't use it as well - too fancy for our simple purpose).

what I'm saying here @epdenouden is that if I will run phpunit in my project, using cache and out-of-the-box approach (so I don't configure anything extra), phpunit will leave result_cache.json file there. if phpunit will leave sth, it shall indicate it's his leftover, putting it to the name part.
like

-result_cache +phpunit_cache

@keradus quick reply: that was a good suggestion, thx! Have pushed a change for that.

I mixed two related topics regarding the caching, making my comment confusion. Adding in persistence cleanly has been more of a chore than I expected. Will spend more time tonight on a cleaner approach.

new name looks shiny, thanks for the change!
then, I will try to review changes tomorrow ;)

keradus · 2018-05-29T10:29:25Z

src/Util/TestResultCache.php

+            return;
+        }
+
+        $this->defects = $cacheData['defects'];


we have no clue that decoded data contain such key...
and even if - what is the content of it.

it's optimistic approach here.

cache file could be treated as-is, one is not supposed to modify it manually.
maybe serialize and then unserialize with unserialize(..., ['allowed_classes' => CacheData::class]) ?

Yes, the earliest versions of Breakfast were just that: patches to the code that quickly (un)serialized a results file to store the state between runs. I'll try some other options.

keradus · 2018-05-29T10:30:18Z

src/Util/TestResultCache.php

+
+    private function formatJSONOutputStruct(): string
+    {
+        return \json_encode([


json_encode can fail as well (eg on non utf-8 input)

keradus · 2018-05-29T10:31:02Z

src/Util/TestResultCache.php

+    /**
+     * @var array
+     */
+    private $times = [];


what is the content of those collections? array of.... ints? is key meaningful or numeric?
how single defect looks like ?

Thanks, all good points regarding the results cache. Started refactoring that part to make it more robust and cleaner.

keradus · 2018-05-29T10:32:22Z

phpunit.xsd

@@ -227,6 +233,7 @@
    <xs:attribute name="printerClass" type="xs:string" default="PHPUnit\TextUI\ResultPrinter"/>
    <xs:attribute name="printerFile" type="xs:anyURI"/>
    <xs:attribute name="processIsolation" type="xs:boolean" default="false"/>
+    <xs:attribute name="stopOnDefect" type="xs:boolean" default="false"/>


would be nice to document somewhere what is a defect

Thank you. I have added a description to the help message.

keradus · 2018-05-29T10:33:15Z

src/Framework/ResultCacheListener.php

+
+    private function updateRunState(\PHPUnit\Framework\Test $test, string $state): void
+    {
+        $testName                  = $this->fullTestName($test);


too many spaces

keradus · 2018-05-29T10:34:27Z

src/Runner/TestSuiteSorter.php

+    ];
+
+    /**
+     * @var array


Added description

keradus · 2018-05-29T10:35:11Z

src/Runner/TestSuiteSorter.php

+    /**
+     * @var array
+     */
+    private $defectSortWeight = [


Fixed, was a precalculated array before.

keradus · 2018-05-29T10:36:38Z

src/Runner/TestSuiteSorter.php

        }
    }

-    private function sort(TestSuite $suite, int $order, bool $resolveDependencies): void
+    public function setCache(TestResultCache $cache): void


I believe this property shall not be exposed

You're right, the method is only used in the test. Will clean it up :)

keradus · 2018-05-29T10:41:09Z

tests/Runner/TestSuiteSorterEmptyTestCaseTest.php

+use PHPUnit\Framework\TestCase;
+use PHPUnit\Framework\TestSuite;
+
+class TestSuiteSorterEmptyTestCaseTest extends TestCase


there is no TestSuiteSorterEmptyTestCase class, how we can have test for it ?

test here shall be part of TestSuiteSorterTest test class, it's one concrete, edge scenario to be tested, not completely other class

Refactored the fixtures in TestSuiteSorterTest and added this test in. Cleaner, thanks for the comment!

It started off as a quick test where I didn't want the fixtures from TestSuiteSorterTest. The name is indeed confusing, it was supposed to mean "TestSuiteSorter (tested with a) EmptyTestCase".

keradus · 2018-05-30T06:36:30Z

tests/Runner/TestSuiteSorterTest.php

        $sorter = new TestSuiteSorter();
-        $sorter->setCache($cache);
+        $_sorter = new \ReflectionClass($sorter);


_? leading _?

what about $sorterReflection ?

also, having a need to access class' internals in test indicate bad class structure. maybe Cache shall be injected to Sorter via constructor ?

The $_var as a temporary version of $var as I quickly needed one, like the $_suite in TestRunner. Will pick something else for now, then look at the use of Reflection.

I can imagine $sorter = new TestSuiteSorter($cache) but to me that feels like juggling the hot potato of persisting the cache between runs. When added to the constructor it becomes the problem of the TestRunner which creates the TestSuiteSorter.

Thank you for your detailed comments! I will spend more time experimenting with the code and structure later today.

keradus · 2018-05-30T06:38:14Z

src/Runner/TestSuiteSorter.php

@@ -51,7 +51,7 @@
    ];

    /**
-     * @var array
+     * @var array[string]int Associative array of (string => DEFECT_SORT_WEIGHT) elements


array[string]int is not proper type according to phpdoc
array<string, int>

Thanks! Where did you find this syntax? I saw this recommended a few times, but have found no definitive answer in the manual https://docs.phpdoc.org/guides/types.html

I found some discussions regarding proposals but most of them seem to have gone to sleep, for example: phpDocumentor/phpDocumentor#650

psr-5 always-proposal, sadly. but it's widely suported by IDEs and SCAs

keradus · 2018-05-30T14:59:13Z

src/Util/TestResultCache.php

            return;
        }

-        $this->loadFromJSON($json);
+        $cache = \unserialize($cacheData);


please provide 2nd argument with allowed class deserialization.
now, you are exposing injection point for security hole (for PHPUnit it's not big deal, but still...)

Yes thx :) had added that in the meantime. Since you're here, is it very picky about namespacing?

self::class takes FQCN into consideration, so all good ;)

keradus · 2018-05-30T19:36:31Z

tests/Util/TestResultCacheTest.php

    public function testCacheFilenameViaEnv()
    {
        $_ENV['PHPUNIT_RESULT_CACHE'] = '/some/cache';
        $cache                        = new TestResultCache;

        $this->assertEquals('/some/cache', $cache->getResultCacheFilename());
+        unset($_ENV['PHPUNIT_RESULT_CACHE']);


if assertEquals would fail,
this line would never be executed

keradus · 2018-05-30T19:37:19Z

src/Framework/ResultCacheListener.php

+    private $runState = [];
+
+    /**
+     * @var TestResultCache


new ResultCacheListener()

boom ! this property is null, not TestResultCache ;)

added some phpdoc ...which I then even had to sort 🤓

keradus · 2018-06-13T08:24:36Z

src/TextUI/Command.php

-  --defects-first-order       Run previously unsuccessful tests first
-  --random-order              Run tests in random order
+  --order-by=<order>          Run tests in 'default', 'random' or 'reverse' order
+  --order-by=defects          Run previously failed tests first


not sure about this docs here. it's single option, it shall have single doc entry, in which we explain what is defects

also, looks like depends is missing

Good points, yes. Wanted to give it a quick try using it myself for a while first. If this is something that @sebastianbergmann also likes I will make it production-ready.

It barely fits on one line now, but everybody reads the manual... right? :)

keradus · 2018-06-13T08:25:25Z

src/TextUI/Command.php

+
+    private function handleOrderByOption(string $value): void
+    {
+        foreach (\explode(',', $value) as $order) {


if I would put as value foobar, it shall fail. now, it's ignored

Yes. I didn't see other --options throw nice error messages yesterday. Will have a look at how to do this nicely. Silent failures are a big no-no here, I have typed defecst waaaaay too often

Added a simple error message with failure exit code.

epdenouden · 2018-06-13T15:50:19Z

@keradus The --order-by=foobar will now exit with an error message:

./phpunit --order-by=defects,foobar,depends

PHPUnit 7.3-g0a4a8c258 by Sebastian Bergmann and contributors.

unrecognized order-by option -- foobar

The help message is now:

  --order-by=<order>          Run tests in order: default|reverse|random|defects|depends

keradus · 2018-06-13T19:10:01Z

src/TextUI/Command.php

@@ -1347,6 +1346,8 @@ private function handleOrderByOption(string $value): void
                    $this->arguments['resolveDependencies'] = true;

                    break;
+                default:
+                    $this->exitWithErrorMessage("unrecognized order-by option -- $order");


why double -- ?

Fixed. Was a leftover from where I found this in GetOpt.

keradus · 2018-06-13T19:15:04Z

phpunit.xsd

@@ -171,6 +171,12 @@
      <xs:enumeration value="random"/>
    </xs:restriction>
  </xs:simpleType>
+  <xs:simpleType name="executionOrderDefectsType">


this shall be removed now, and executionOrderType shall be extended to contain defects and depends

Removed. I have added a list of the common scenarios that make sense. Couldn't find a nice way to define the list in the XSD to help the tooling with validation and autocomplete.

keradus · 2018-06-13T19:16:17Z

src/Framework/TestResult.php

@@ -1011,6 +1016,14 @@ public function stopOnSkipped(bool $flag): void
        $this->stopOnSkipped = $flag;
    }

+    /**
+     * Enables or disables the stopping for any defect: skipped, error, failure, warning, incomplete or risky.


I believe this description is not up to date now

keradus · 2018-06-18T14:51:43Z

tests/Runner/TestSuiteSorterTest.php

+                    'testFour'  => ['state' => BaseTestRunner::STATUS_FAILURE, 'time' => 1],
+                    'testFive'  => ['state' => BaseTestRunner::STATUS_FAILURE, 'time' => 1],
+                ],
+                ['testFive', 'testOne', 'testTwo', 'testThree', 'testFour']],


👎
testFour and testFive failed, so I want to have them as soon as possible, with respect to their dependencies.

I would expect that order, instead:
['testFive', 'testThree', 'testFour', 'testOne', 'testTwo']
or
['testThree', 'testFour', 'testFive', 'testOne', 'testTwo']

as in the end, we claimed to run defects first, not last (vide testFour)

testThree has dependencies on testOne and testTwo

testFour has a dependency on testThree

what happens:

after sorting for defects the order is: 4 (failed), 5 (failed), 1, 2, 3

running it like this would skip 4 because it needs to run 3 first

same story for 3: it needs both 1 and 2

the dependency resolver fixes this quietly for us: 5 can stay up front (=defects FIRST) but 4 goes behind 3 which goes behind 1,2 (=defects ASAP ;-)

ha, ok, your first 2 points were not part of description (while dependencies of testThree for TestFour was)

then, all good 👍

Good but not good enough :) I will document the use of MultiDependencyTest in the dataproviders to avoid some of the confusion.

keradus · 2018-06-19T09:22:53Z

src/TextUI/TestRunner.php

+        $arguments['processIsolation']                               = $arguments['processIsolation'] ?? false;
+        $arguments['processUncoveredFilesFromWhitelist']             = $arguments['processUncoveredFilesFromWhitelist'] ?? false;
+        $arguments['randomOrderSeed']                                = $arguments['randomOrderSeed'] ?? \time();
+        $arguments['registerMockObjectsFromTestArgumentsRecursively']= $arguments['registerMockObjectsFromTestArgumentsRecursively'] ?? false;


a space is missing before =
after solving that, alignment will be back to original one again

I tried something like that in the past, let's see if your suggestion works nicer.

keradus

looks damn good piece of work here !

epdenouden · 2018-06-21T14:17:11Z

@sebastianbergmann Hi Sebastian! @keradus and I think this pull request is done. For new ideas and improvements I will open new PRs in the near future. :)

keradus

double checked after squash, looks same as my local copy before squashing

sebastianbergmann · 2018-06-24T21:09:41Z

Thanks!

epdenouden · 2018-06-24T21:15:38Z

🎁 thanks for the merge! :)

keradus · 2018-06-24T22:35:31Z

🎆

Jis sugeneruojamas, kad būtų greitesnis testavimo-koregavimo ciklas. Įjungtas standatiškai nuo PHPUnit 8 Dokumentacija: * sebastianbergmann/phpunit#3147 * https://phpunit.de/announcements/phpunit-8.html

epdenouden force-pushed the breakfast branch 3 times, most recently from 4f40a19 to 476c1ea Compare May 28, 2018 22:49

sebastianbergmann added the type/enhancement A new idea that should be implemented label May 29, 2018

sebastianbergmann added this to the PHPUnit 7.3 milestone May 29, 2018

epdenouden force-pushed the breakfast branch from 0b63afb to 15848af Compare May 29, 2018 08:40

epdenouden mentioned this pull request May 29, 2018

Default Testsuite in XML, redux #2876

Closed

keradus reviewed May 29, 2018

View reviewed changes

epdenouden force-pushed the breakfast branch 2 times, most recently from 59d043a to b46e416 Compare May 29, 2018 16:36

keradus reviewed May 30, 2018

View reviewed changes

epdenouden force-pushed the breakfast branch 3 times, most recently from 0139690 to cbeb934 Compare May 30, 2018 14:55

keradus reviewed May 30, 2018

View reviewed changes

keradus reviewed Jun 13, 2018

View reviewed changes

keradus reviewed Jun 18, 2018

View reviewed changes

keradus reviewed Jun 19, 2018

View reviewed changes

keradus approved these changes Jun 19, 2018

View reviewed changes

epdenouden force-pushed the breakfast branch from d13647d to 6b2f472 Compare June 19, 2018 10:37

epdenouden force-pushed the breakfast branch from 16e0235 to 5b16423 Compare June 21, 2018 14:48

keradus approved these changes Jun 21, 2018

View reviewed changes

epdenouden force-pushed the breakfast branch 2 times, most recently from 771abc2 to 5b16423 Compare June 22, 2018 18:19

Breakfast: rerun defects first

7dc6729

epdenouden force-pushed the breakfast branch from 5b16423 to 7dc6729 Compare June 22, 2018 18:24

sebastianbergmann closed this Jun 24, 2018

epdenouden deleted the breakfast branch July 11, 2018 14:29

epdenouden mentioned this pull request Aug 3, 2018

result cache file created by default #3237

Closed

reinholdfuereder mentioned this pull request Aug 7, 2018

Implement "previous failures first" mode #46

Closed

sebastianbergmann mentioned this pull request Sep 8, 2018

Reorder tests based on execution time #3284

Closed

sebastianbergmann mentioned this pull request Nov 2, 2018

"Buffered" TestDox printer(s) #3380

Closed

epdenouden mentioned this pull request Dec 19, 2018

À tout de suite: fast self-test suite #3453

Closed

38 tasks

gerardroche mentioned this pull request Oct 17, 2019

Rerun failed tests gerardroche/sublime-phpunit#29

Closed

Breakfast: rerun defects first #3147

Breakfast: rerun defects first #3147

Conversation

epdenouden commented May 28, 2018 • edited

Use case

Summary of changes

How to test

Design considerations

codecov-io commented May 28, 2018 • edited

Codecov Report

sebastianbergmann commented May 29, 2018

epdenouden commented May 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keradus May 30, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keradus May 29, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keradus May 29, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epdenouden May 30, 2018 • edited

Choose a reason for hiding this comment

keradus May 30, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keradus May 30, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epdenouden commented Jun 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keradus Jun 18, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keradus left a comment

Choose a reason for hiding this comment

epdenouden commented Jun 21, 2018 • edited

keradus left a comment

Choose a reason for hiding this comment

sebastianbergmann commented Jun 24, 2018

epdenouden commented Jun 24, 2018

epdenouden commented May 28, 2018 •

edited

codecov-io commented May 28, 2018 •

edited

keradus May 30, 2018 •

edited

keradus May 29, 2018 •

edited

keradus May 29, 2018 •

edited

epdenouden May 30, 2018 •

edited

keradus May 30, 2018 •

edited

keradus May 30, 2018 •

edited

keradus Jun 18, 2018 •

edited

epdenouden commented Jun 21, 2018 •

edited