Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe fix history - result reconstruction mismatch #902

Merged
merged 18 commits into from
Aug 1, 2022
Merged

Conversation

yannikschaelte
Copy link
Member

@yannikschaelte yannikschaelte commented Jul 30, 2022

Should fix #743.

  • Essentially @PaulJonasJost was right in Fix hdf5 result mismatch #880: At the end in finalize, one should consult the history, which may have more entries than saved in the ..._min attributes of OptimizerHistory.
  • However, only overwrite if history has a better value (hardly possible) or has additional values, as ..._min may contain values not saved in history, e.g. when trace_record_grad is False.

Further

  • Set some checks to np.array_equal instead of np.allclose, as in-memory changes to array entries should not be possible.
  • Remaining np.allclose checks are I think only required because CsvHistory has an information loss, which MemoryHistory and Hdf5History don't have. If possible, one could further adjust the tolerances atol, rtol to the accuracy of the storage (e.g. 16 digits instead of numpy's default 1e-5, 1e-8?)
  • And did some tidying up.

@yannikschaelte yannikschaelte changed the title Maybe fix history - result reconstruction mismatch [WIP] Maybe fix history - result reconstruction mismatch Jul 30, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jul 30, 2022

Codecov Report

Merging #902 (93874ed) into develop (6529d4d) will decrease coverage by 1.79%.
The diff coverage is 93.42%.

@@             Coverage Diff             @@
##           develop     #902      +/-   ##
===========================================
- Coverage    88.33%   86.53%   -1.80%     
===========================================
  Files          107      107              
  Lines         7585     7613      +28     
===========================================
- Hits          6700     6588     -112     
- Misses         885     1025     +140     
Impacted Files Coverage Δ
pypesto/objective/__init__.py 100.00% <ø> (ø)
pypesto/optimize/util.py 81.03% <62.50%> (ø)
pypesto/optimize/load.py 96.34% <85.71%> (+0.09%) ⬆️
pypesto/objective/history.py 95.26% <94.02%> (-0.75%) ⬇️
pypesto/C.py 100.00% <100.00%> (ø)
pypesto/objective/base.py 87.62% <100.00%> (+0.36%) ⬆️
pypesto/objective/util.py 93.10% <100.00%> (+3.44%) ⬆️
pypesto/util.py 97.22% <100.00%> (+0.92%) ⬆️
pypesto/result/predict.py 56.92% <0.00%> (-29.24%) ⬇️
pypesto/petab/importer.py 62.56% <0.00%> (-25.67%) ⬇️
... and 10 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

@yannikschaelte yannikschaelte changed the title [WIP] Maybe fix history - result reconstruction mismatch Maybe fix history - result reconstruction mismatch Jul 31, 2022
@yannikschaelte yannikschaelte self-assigned this Jul 31, 2022
Copy link
Contributor

@FFroehlich FFroehlich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 CHI2 and SCHI2 are currently missing in all of the updates, but looking at #901, I would rather completely remove them than update the code.

pypesto/objective/history.py Show resolved Hide resolved
pypesto/objective/history.py Outdated Show resolved Hide resolved
pypesto/objective/history.py Outdated Show resolved Hide resolved
pypesto/objective/history.py Outdated Show resolved Hide resolved
pypesto/objective/history.py Outdated Show resolved Hide resolved
pypesto/objective/history.py Show resolved Hide resolved
pypesto/objective/util.py Outdated Show resolved Hide resolved
pypesto/objective/util.py Outdated Show resolved Hide resolved
pypesto/util.py Outdated Show resolved Hide resolved
Copy link
Member

@dweindl dweindl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Hopefully this resolves those issues 🤞

pypesto/objective/history.py Show resolved Hide resolved
# in what is available.

# check if a useful history exists
# TODO Y This can be solved prettier
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do it? :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will leave for future refactoring ^^

pypesto/objective/history.py Show resolved Hide resolved
pypesto/optimize/load.py Outdated Show resolved Hide resolved
for a, b in zip(partial_trace, [full_trace[i] for i in arr]):
if var != 'schi2':
assert np.all(a == b) or np.isnan(a) and np.isnan(b)
assert np.all(a == b) or np.isnan(a) and np.isnan(b), var
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert_array_equal provides more helpful output than ==, but fine to leave, as those were there before.

yannikschaelte and others added 5 commits August 1, 2022 09:21
Co-authored-by: Fabian Fröhlich <fabian_froehlich@hms.harvard.edu>
Co-authored-by: Fabian Fröhlich <fabian_froehlich@hms.harvard.edu>
@yannikschaelte
Copy link
Member Author

+1 CHI2 and SCHI2 are currently missing in all of the updates, but looking at #901, I would rather completely remove them than update the code.

👍 will make a separate pr removing chi2, schi2

@yannikschaelte yannikschaelte merged commit cdb6d28 into develop Aug 1, 2022
@yannikschaelte yannikschaelte deleted the fixes_ys branch August 1, 2022 14:06
@PaulJonasJost PaulJonasJost mentioned this pull request Sep 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants