Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved dependency resolver and tests #3144

Merged
merged 3 commits into from May 26, 2018
Merged

Improved dependency resolver and tests #3144

merged 3 commits into from May 26, 2018

Conversation

epdenouden
Copy link
Contributor

@epdenouden epdenouden commented May 26, 2018

Summary

Changes to test execution order can result in unexpected defects and create uncertainty, which is the exact opposite of what we want. The dependency resolver is improved to be more hands-off and leave more tests in original order.

Why: backwards compatibility

During development of the test execution reordering feature #3092 backwards compatibility was discussed. Automatic dependency resolution will be optional in 7.x and default for 8.x.

How: improved dependency manager

The current resolver has a speed optimization that always sorts all independent tests to the front before looking at tests with dependencies. While this produces a technically correct execution order, it produces 'noise' by moving tests around more than necessary. The simplified resolver returns a more natural execution order:

# Original order after noisy resolver after quiet resolver
1 A_independent A_independent A_independent
2 B_dep_on_A C_independent B_dep_on_A
3 C_independent E_independent C_independent
4 D_dep_on_C B_dep_on_A D_dep_on_C
5 E_independent D_dep_on_C E_independent

Changed functionality and impact

  • no noticeable difference in performance or resource footprint
  • no difference in configuration and input
  • output is functionally unchanged, however the order might be slightly different; this only shows up using --debug or relying on an exact sequence of tests when logging

Software design and testing

  • make dependency resolution optional again: 8fa2aff
  • the TestSuiteSorter dependency resolver now is simpler and checks all tests at least once
  • all interfaces between components stay the same
  • add isolation tests for the TestSuiteSorter
  • keep the current TextUI end-to-end tests

@sebastianbergmann sebastianbergmann merged commit 66a8521 into sebastianbergmann:master May 26, 2018
@epdenouden epdenouden deleted the test-reorder-quiet-resolve branch May 26, 2018 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants