All-day breakfast at the dataprovider buffet #3401
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rovider rows do not mark the parent test failed.
Codecov Report
@@ Coverage Diff @@
## master #3401 +/- ##
============================================
+ Coverage 83.05% 83.11% +0.06%
+ Complexity 3570 3567 -3
============================================
Files 143 143
Lines 9503 9502 -1
============================================
+ Hits 7893 7898 +5
+ Misses 1610 1604 -6
Continue to review full report at Codecov.
|
epdenouden
changed the title
|
Merged into |
|
@sebastianbergmann Thanks! This PR fixes #3246 and #3396, they can be closed, too |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Thanks to @reinholdfuereder some remaining issues with the reordering of
@dataprovider-fed tests have been resolved. As described in more detail #3396 and its predecessor ticket #3246, the main problem with execution reordering has been understanding the hierarchy of the run queue.The fix
Lack of a consistent naming scheme for each individual combination of (test, data) reared its ugly head again. This has now been solved cleanly and I've updated all the tests to match the new scheme, so all previous functionality remains under guard. I have added a regression end-to-end test to keep an eye on the dataproviders.
Impact
TestSuite, in addition to the (already existing) functionality to sort failing dataprovider rows to the front of their parent testNameSpace\TestClass::testMethod,NameSpace\TestClass::testMethod with data set $dataor a PHPT filenameTestSuiteSorternow uses normalized FQCN+dataname everywhereResultCacheListenerkeeps the FQCN+dataname of tests but strips the dump of the dataAdditional notes
I have opted to not add refinements to the
TestResultCacheat this time, to keep the changes atomic. If I learned anything from #3246 is that debugging the sorters is still very time consuming. Such optimizations add complexity and would only be noticeable by maintainers of large projects and even then it's more of a nice-to-have.As always let me know if you have any remarks or wishes.