Better support test reproducibility #2721
Conversation
ktbarrett
force-pushed
the
random-seed-reproducibility
branch
from
621f79d
to
e72bba5
Compare
Codecov Report
@@ Coverage Diff @@
## master #2721 +/- ##
==========================================
- Coverage 76.47% 74.19% -2.28%
==========================================
Files 45 45
Lines 7846 7848 +2
Branches 633 1249 +616
==========================================
- Hits 6000 5823 -177
+ Misses 1662 1633 -29
- Partials 184 392 +208
Continue to review full report at Codecov.
|
| # seed random number generator based on test module, name, and RANDOM_SEED | ||
| hasher = hashlib.sha1() | ||
| hasher.update(test.__qualname__.encode()) | ||
| hasher.update(test.__module__.encode()) |
There was a problem hiding this comment.
As I commented, it's not unusual where I work for people to collect failing tests from many different test files (MODULE files) into one big regression MODULE file. Users will often call this file "The Kitchen Sink" or something creative like that. :-) If the MODULE name is used here then users would not be able to do that i.e. if they had a hundred failing tests they could potentially be forced to have a hundred separate MODULE files as well... rather than one. Not a deal breaker but I thought I should mention it.
There was a problem hiding this comment.
__module__ will be the name of the module they were defined in, not the module they were discovered in. So importing tests from other modules into a "kitchen sink" module will continue to work.
|
Needs newsfragment. |
Co-authored-by: Colin Marquardt <cmarqu42@gmail.com>
Closes #2672. Pinging @walt3k.
Tests are now reproducible if:
RANDOM_SEEDis used