Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add autoscan for upper limit using TOMS Algorithm 748 #1274

Merged
merged 94 commits into from
Sep 21, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
258786d
Autoscan for upper limit using TOMS748
Jan 21, 2021
5755d20
Fix CodeFactor issues
Jan 21, 2021
0639eb0
Add a test for upperlimit_auto
Jan 21, 2021
b2bb37f
Fix typo in upperlimit_auto test
Jan 21, 2021
ba7859e
Fix test_upperlimit_auto definition
Jan 21, 2021
abcd6ed
Adjust docstring for pyhf.infer.upperlimit_auto
beojan Jan 21, 2021
f9f1cfd
upperlimit_auto: Warn if rtol is not provided
Jan 22, 2021
66ab5c7
upperlimit_auto: Fix == None -> is None
Jan 22, 2021
1b4e5b6
Merge branch 'master' into autoscan
beojan Jan 29, 2021
f5a7033
Test upperlimit_auto against upperlimit
Jan 29, 2021
68e7da2
Use absolute tolerance in upperlimit_auto test
Jan 29, 2021
522db2f
Add upperlimit_auto to docs
Jan 29, 2021
8bf5b1c
Add option to find upper limit using toys and qmu
Jan 29, 2021
913d499
Merge branch 'master' into autoscan
beojan Jan 30, 2021
459fb38
Merge branch 'master' into autoscan
beojan Feb 19, 2021
bbc5b71
Merge branch 'master' into autoscan
beojan Apr 15, 2021
14932f9
Merge branch 'master' into autoscan
beojan May 28, 2021
4ec32e1
Add scan="auto" option to upperlimit
May 28, 2021
8eeab01
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 28, 2021
032e4ea
Merge branch 'master' into autoscan
beojan Jul 26, 2021
c4aa486
Factor out upperlimit_fixedscan and auto-extend auto bracket if needed
Jul 26, 2021
94ff516
Fix code quality issue
Jul 26, 2021
531d319
Merge branch 'master' into autoscan
matthewfeickert Aug 17, 2021
609deff
Merge branch 'master' into autoscan
kratsg Mar 30, 2022
836cf91
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 30, 2022
21f5737
Merge branch 'master' into autoscan
matthewfeickert Apr 6, 2022
a4b2923
Merge branch 'master' into autoscan
matthewfeickert Jun 23, 2022
d592fc6
Merge branch 'master' into autoscan
matthewfeickert Sep 5, 2022
2692c41
Merge branch 'master' into autoscan
kratsg Sep 8, 2022
a57871b
Merge branch 'master' into autoscan
matthewfeickert Sep 9, 2022
98fbf98
isort
matthewfeickert Sep 9, 2022
dc344ed
formatting
matthewfeickert Sep 9, 2022
bacd453
Use fstrings for rtol warning
matthewfeickert Sep 9, 2022
2e2cd1e
Use level arg over hardcoded 0.05
matthewfeickert Sep 9, 2022
d40372b
add in lru_cache import
matthewfeickert Sep 9, 2022
1756118
Merge branch 'master' into autoscan
kratsg Sep 9, 2022
e9bb955
Rename and add to public API
matthewfeickert Sep 9, 2022
10658b8
Add in hypotest_kwargs
matthewfeickert Sep 9, 2022
db8615b
Add FIXME notice for later. FIX BEFORE MERGE
matthewfeickert Sep 9, 2022
73562d9
Update public API repr
matthewfeickert Sep 9, 2022
5ea1a86
Merge branch 'master' into autoscan
matthewfeickert Sep 9, 2022
a9d886c
Indent docstrings correctly
matthewfeickert Sep 10, 2022
0e8f4df
Use uncorrelated_background API for docstring example
matthewfeickert Sep 10, 2022
b013c44
Use rtol in docstring example to avoid warning
matthewfeickert Sep 10, 2022
46e3c4d
drop lru_cache
kratsg Sep 15, 2022
c20560b
fix kwargs for autoscan
kratsg Sep 15, 2022
e2d6c25
Merge branch 'master' into autoscan
kratsg Sep 15, 2022
1dd7e94
docs: Seperate docstrings semantically
matthewfeickert Sep 16, 2022
ef8581e
Apply sourcery suggestion for simplification
matthewfeickert Sep 16, 2022
931e124
Rename to cached for clarity
matthewfeickert Sep 16, 2022
b15f1dd
Add test for auto through upperlimit API
matthewfeickert Sep 16, 2022
54035e7
Use None instead of auto to simplify API
matthewfeickert Sep 16, 2022
dbbbea1
Avoid function level globals
matthewfeickert Sep 16, 2022
19a2ee1
Use np.asarray to avoid copy
matthewfeickert Sep 16, 2022
8fa2f03
Use lower and upper to match scipy terms
matthewfeickert Sep 16, 2022
b88bbfc
Split warning for readability
matthewfeickert Sep 16, 2022
fc6badd
Add test for rtol warning
matthewfeickert Sep 16, 2022
bf5f10d
Remove tmpdir fixture as not needed for these tests given no writing …
matthewfeickert Sep 16, 2022
61554f4
Add check for return_results
matthewfeickert Sep 16, 2022
4efc5de
More verbose
matthewfeickert Sep 16, 2022
939f005
fix: Correct concatenate lists instead of adding float to all list el…
matthewfeickert Sep 16, 2022
ddcb198
Test bounds expansion
matthewfeickert Sep 16, 2022
805e849
Merge branch 'master' into autoscan
matthewfeickert Sep 16, 2022
6b29851
Add upperlimit_fixed_scan to API docs
matthewfeickert Sep 16, 2022
d954472
Add return_results test
matthewfeickert Sep 17, 2022
56da154
move to test_upperlimit_with_kwargs
matthewfeickert Sep 17, 2022
89853a1
Move the pop out before evaluation to make everything very clean and …
matthewfeickert Sep 17, 2022
332eef7
Note what scan
matthewfeickert Sep 17, 2022
924bbb5
Rename to auto_scan
matthewfeickert Sep 17, 2022
c9fd89a
docs: fix link
matthewfeickert Sep 17, 2022
2d99028
Provide better coverage and use np.allclose
matthewfeickert Sep 17, 2022
5535af1
docs: Add Beojan Stanislaus to contributor list
matthewfeickert Sep 17, 2022
63636c3
change auto_scan to toms748_scan
matthewfeickert Sep 19, 2022
f6f7325
rename fixed_scan to linear_grid_scan
matthewfeickert Sep 19, 2022
f7e7ad4
Make intervals module and change API to upper_limit
matthewfeickert Sep 19, 2022
2246e4b
feat: Add internal API to warn of deprecation and future removal
matthewfeickert Sep 19, 2022
a034af3
Note deprecated API
matthewfeickert Sep 19, 2022
00f59b2
Seperate into condifence intervals section
matthewfeickert Sep 20, 2022
d6b6533
fix: Use function scope to avoid altering hypotest_args fixture
matthewfeickert Sep 20, 2022
d5916d6
Make test name explicit
matthewfeickert Sep 20, 2022
72c57b0
Use deprecated Sphinx note
matthewfeickert Sep 20, 2022
9835d25
Add versionadded directives
matthewfeickert Sep 20, 2022
10a2040
Merge remote-tracking branch 'origin/master' into autoscan
matthewfeickert Sep 20, 2022
4d91104
Update lower bound on scipy as toms748 added in scipy v1.2.0
matthewfeickert Sep 20, 2022
c40d982
back off relative tolerance
matthewfeickert Sep 20, 2022
d5fc4d1
Explicitly set as UserWarning
matthewfeickert Sep 20, 2022
42a7523
Get rid of UserWarning and just set defaults
matthewfeickert Sep 20, 2022
21d54f1
docs: Correct relative tolerance statement
matthewfeickert Sep 20, 2022
d7baca6
DEBUG: IF THIS WORKS BUG IN tensor/manager.py
matthewfeickert Sep 20, 2022
716e043
docs: Add FIXME
matthewfeickert Sep 21, 2022
798eab7
use tighter relative tolerance to force convergence
matthewfeickert Sep 21, 2022
9fd1968
docs: Remove statement on default
matthewfeickert Sep 21, 2022
0bf8a86
remove hack to fail now as this will be fixed in PR 2017
matthewfeickert Sep 21, 2022
36a9911
Merge branch 'master' into autoscan
matthewfeickert Sep 21, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,19 @@ Fits and Tests
mle.fit
mle.fixed_poi_fit
hypotest
intervals.upperlimit
utils.all_pois_floating

Confidence Intervals
~~~~~~~~~~~~~~~~~~~~

.. autosummary::
:toctree: _generated/
:nosignatures:

intervals.upper_limits.upper_limit
intervals.upper_limits.toms748_scan
intervals.upper_limits.linear_grid_scan
intervals.upperlimit

Schema
------
Expand Down
1 change: 1 addition & 0 deletions docs/contributors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,4 @@ Contributors include:
- Aryan Roy
- Jerry Ling
- Nathan Simpson
- Beojan Stanislaus
6 changes: 5 additions & 1 deletion docs/examples/notebooks/ShapeFactor.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,11 @@
}
],
"source": [
"obs_limit, exp_limits, (poi_tests, tests) = pyhf.infer.intervals.upperlimit(\n",
"(\n",
" obs_limit,\n",
" exp_limits,\n",
" (poi_tests, tests),\n",
") = pyhf.infer.intervals.upper_limits.upper_limit(\n",
" data, pdf, np.linspace(0, 5, 61), level=0.05, return_results=True\n",
")"
]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2295,7 +2295,11 @@
"outputs": [],
"source": [
"mu_tests = np.linspace(0, 1, 16)\n",
"obs_limit, exp_limits, (poi_tests, tests) = pyhf.infer.intervals.upperlimit(\n",
"(\n",
" obs_limit,\n",
" exp_limits,\n",
" (poi_tests, tests),\n",
") = pyhf.infer.intervals.upper_limits.upper_limit(\n",
" data, pdf, mu_tests, level=0.05, return_results=True\n",
")"
]
Expand Down
6 changes: 5 additions & 1 deletion docs/examples/notebooks/multiBinPois.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,11 @@
"init_pars = model.config.suggested_init()\n",
"par_bounds = model.config.suggested_bounds()\n",
"\n",
"obs_limit, exp_limits, (poi_tests, tests) = pyhf.infer.intervals.upperlimit(\n",
"(\n",
" obs_limit,\n",
" exp_limits,\n",
" (poi_tests, tests),\n",
") = pyhf.infer.intervals.upper_limits.upper_limit(\n",
" data, model, np.linspace(0, 5, 61), level=0.05, return_results=True\n",
")"
]
Expand Down
2 changes: 1 addition & 1 deletion setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ package_dir =
include_package_data = True
python_requires = >=3.7
install_requires =
scipy>=1.1.0 # requires numpy, which is required by pyhf and tensorflow
scipy>=1.2.0 # requires numpy, which is required by pyhf and tensorflow
click>=8.0.0 # for console scripts
tqdm>=4.56.0 # for readxml
jsonschema>=4.15.0 # for utils
Expand Down
75 changes: 0 additions & 75 deletions src/pyhf/infer/intervals.py

This file was deleted.

30 changes: 30 additions & 0 deletions src/pyhf/infer/intervals/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
"""Interval estimation"""
import pyhf.infer.intervals.upper_limits

__all__ = ["upper_limits.upper_limit"]


def __dir__():
return __all__


def upperlimit(
data, model, scan=None, level=0.05, return_results=False, **hypotest_kwargs
):
"""
.. deprecated:: 0.7.0
Use :func:`~pyhf.infer.intervals.upper_limits.upper_limit` instead.
.. warning:: :func:`~pyhf.infer.intervals.upperlimit` will be removed in
``pyhf`` ``v0.9.0``.
"""
from pyhf.exceptions import _deprecated_api_warning
kratsg marked this conversation as resolved.
Show resolved Hide resolved

_deprecated_api_warning(
"pyhf.infer.intervals.upperlimit",
"pyhf.infer.intervals.upper_limits.upper_limit",
"0.7.0",
"0.9.0",
)
return pyhf.infer.intervals.upper_limits.upper_limit(
data, model, scan, level, return_results, **hypotest_kwargs
)
Loading