Fix random seed on tests - #2844
Merged
Merged
Conversation
Codecov ReportPatch and project coverage have no change.
Additional details and impacted files@@ Coverage Diff @@
## develop #2844 +/- ##
========================================
Coverage 99.69% 99.69%
========================================
Files 273 273
Lines 19065 19065
========================================
Hits 19006 19006
Misses 59 59 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
jsbrittain
marked this pull request as ready for review
March 31, 2023 19:25
valentinsulzer
approved these changes
Apr 5, 2023
valentinsulzer
left a comment
Member
There was a problem hiding this comment.
Looks good to me. Will need changelog update (moving to new section) and merging the changes from the casadi 3.6 update
8 tasks
Member
|
Changelog updates are in the wrong month here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Some tests in the unit and integration suites call functions that make use of random variables, or include random variables in their calls (specifically numpy.random). This can lead to tests sometimes failing due to reasons that may be unrelated to recent code changes, as outlined (and demonstrated) in issue #2833.
Fixes #2833
Type of change
Fixing the numpy random seed within the test routines produces consistent behavior. However, since stochastic elements are present in the PyBaMM codebase in relation to both the Casadi and Idaklu solvers, extensive dependencies on these functions within the test suite are likely to be widespread and opaque. In order to fully rectify the issue (whilst maintaining the same simplicity in test set-up moving forward) a metaclass solution is proposed.
The proposed solution wraps all methods that inherit from
TestCasewith a wrapper function that fixes the random seed to a hash code derived from the method's name. This permits hashes to be changed in problematic circumstances by a hash modifier, such as appending a trailing underscore (or other modifier) to the method name.Specifically, the
TestCaseclass, which is described as the 'Custom TestCase class for pybamm' is given a custom metaclass. However, it was also noted that this test class is not in widespread use across the testbase, and henceunittest.TestCasehas now been replaced withTestCaseas imported fromtestsacross all test scripts.Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.
Key checklist:
$ pre-commit run(see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)$ python run-tests.py --all$ python run-tests.py --doctestYou can run unit and doctests together at once, using
$ python run-tests.py --quick.Further checks: