Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Keep staterror from affecting multiple samples #1965

Merged
merged 10 commits into from
Aug 28, 2022

Conversation

lukasheinrich
Copy link
Contributor

@lukasheinrich lukasheinrich commented Aug 27, 2022

Pull Request Description

This PR is supposed to fix the issue w/ staterrors which erroneously affected multiple samples.

  • Would be great to get add. test coverage and cross-checks w/ real world models

@alexander-held @matthewfeickert @kratsg

Checklist Before Requesting Reviewer

  • Tests are passing
  • "WIP" removed from the title of the pull request
  • Selected an Assignee for the PR to be responsible for the log summary

Before Merging

For the PR Assignees:

  • Summarize commit messages into a comprehensive review of the PR
* Fix staterror bug introduced after v0.6.3 where staterror erroneously affected
  multiple samples.
   - Amends PR #1639 which introduced regression.
   - c.f. Issue #1720
* Remove staterror_combined.__staterror_uncrt as it is unused by the codebase.
* Add test to verify that staterror does not affect more samples than shapesys equivalently.

Co-authored-by: Giordon Stark <kratsg@gmail.com>

@lukasheinrich lukasheinrich changed the title fix staterror fix: fix staterror Aug 27, 2022
@matthewfeickert matthewfeickert marked this pull request as draft August 27, 2022 06:53
@matthewfeickert matthewfeickert added the fix A bug fix label Aug 27, 2022
@matthewfeickert matthewfeickert added this to In progress in v0.7.0 via automation Aug 27, 2022
@matthewfeickert
Copy link
Member

Would be great to get add. test coverage and cross-checks w/ real world models

👍 From the list in Issue #1945, we should be able to check against Issue #1720.

@kratsg kratsg self-assigned this Aug 27, 2022
@codecov
Copy link

codecov bot commented Aug 27, 2022

Codecov Report

Merging #1965 (3bacc51) into master (d4e1fc2) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1965      +/-   ##
==========================================
- Coverage   98.24%   98.24%   -0.01%     
==========================================
  Files          68       68              
  Lines        4380     4377       -3     
  Branches      728      726       -2     
==========================================
- Hits         4303     4300       -3     
  Misses         45       45              
  Partials       32       32              
Flag Coverage Δ
contrib 26.57% <ø> (+0.01%) ⬆️
doctest 60.56% <ø> (+0.01%) ⬆️
unittests-3.10 96.13% <ø> (-0.01%) ⬇️
unittests-3.7 96.12% <ø> (-0.01%) ⬇️
unittests-3.8 96.16% <ø> (-0.01%) ⬇️
unittests-3.9 96.18% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/pyhf/modifiers/staterror.py 98.03% <ø> (-0.06%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Member

@alexander-held alexander-held left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix! Works as intended in the setups I have tested and I did not spot any problems with it.

@matthewfeickert matthewfeickert added the tests pytest label Aug 27, 2022
@matthewfeickert matthewfeickert marked this pull request as ready for review August 28, 2022 06:31
Copy link
Member

@matthewfeickert matthewfeickert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lukasheinrich for this PR and to @kratsg for adding the very nice test! I just have some questions that hopefully have very easy answers that I'm just missing as it is late for me.

(@kratsg As I was the one who originally moved this PR to draft I moved it to ready for review. So we were waiting for me and not Lukas I think.)

src/pyhf/modifiers/staterror.py Show resolved Hide resolved
tests/test_scripts.py Show resolved Hide resolved
Copy link
Member

@matthewfeickert matthewfeickert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 👍 Thanks @lukasheinrich and @kratsg!

Late tonight after I'm done packing I'll try to get Issue #1907 resolved so that we can do a v0.7.0rc2 correctly and then I'll send out a request email for testing and being drafting up the v0.7.0 release notes in PR #1705 so that we can release v0.7.0 early this week.

@matthewfeickert matthewfeickert changed the title fix: fix staterror fix: Keep staterror from affecting multiple samples Aug 28, 2022
@matthewfeickert matthewfeickert merged commit afbc4f3 into master Aug 28, 2022
@matthewfeickert matthewfeickert deleted the staterror_fix branch August 28, 2022 20:12
v0.7.0 automation moved this from In progress to Done Aug 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix A bug fix tests pytest
Projects
Development

Successfully merging this pull request may close these issues.

staterror regressions blocking v0.7.0 Large cls value difference between v0.6.3 and master
4 participants