Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: nonmonotonicity in betaincinv, btdtri, stats.beta.ppf #12796

Closed
josef-pkt opened this issue Aug 31, 2020 · 2 comments · Fixed by #17697
Closed

BUG: nonmonotonicity in betaincinv, btdtri, stats.beta.ppf #12796

josef-pkt opened this issue Aug 31, 2020 · 2 comments · Fixed by #17697
Labels
defect A clear bug or issue that prevents SciPy from being installed or used as expected scipy.special
Milestone
1.12.0

Comments

@josef-pkt
Copy link
Member

betaincinv and related function have a nonmonotonicity which looks incorrect

statsmodels issue statsmodels/statsmodels#7008 is for scipy.stats.beta.isf
but that uses btdri which is equivalent to betaincinv


count_ = np.arange(1, 20)
q, a, b = 1-alpha_2, count_ + 1, nobs - count_
q, a, b
(0.999995,
 array([ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
        19, 20]),
 array([99999, 99998, 99997, 99996, 99995, 99994, 99993, 99992, 99991,
        99990, 99989, 99988, 99987, 99986, 99985, 99984, 99983, 99982,
        99981]))

special.betaincinv(a, b, q)
array([0.00014976, 0.00017332, 0.03499907, 0.03633139, 0.03530332,
       0.00025256, 0.00027046, 0.00028788, 0.00030491, 0.00032161,
       0.00033802, 0.00035417, 0.00037009, 0.0003858 , 0.00040134,
       0.0004167 , 0.00043192, 0.00044699, 0.00046193])

inv = special.btdtri(a, b * 1., 1 - alpha_2)
inv
array([0.00014976, 0.00017332, 0.03499907, 0.03633139, 0.03530332,
       0.00025256, 0.00027046, 0.00028788, 0.00030491, 0.00032161,
       0.00033802, 0.00035417, 0.00037009, 0.0003858 , 0.00040134,
       0.0004167 , 0.00043192, 0.00044699, 0.00046193])

# inverse check

special.betainc(a, b, inv)
array([0.999995, 0.999995, 1.      , 1.      , 1.      , 0.999995,
       0.999995, 0.999995, 0.999995, 0.999995, 0.999995, 0.999995,
       0.999995, 0.999995, 0.999995, 0.999995, 0.999995, 0.999995,
       0.999995])

```​
@josef-pkt josef-pkt added defect A clear bug or issue that prevents SciPy from being installed or used as expected scipy.special labels Aug 31, 2020
@cassiersg
Copy link

For information to anyone who wants to solve (or work around) this issue: the implementation of Boost seems to be correct: https://www.boost.org/doc/libs/1_58_0/libs/math/doc/html/math_toolkit/sf_beta/ibeta_inv_function.html.

@mdhaber
Copy link
Contributor

mdhaber commented Dec 11, 2020

@mckib2 Boostinator : )

@mckib2 mckib2 mentioned this issue Dec 12, 2020
Closed
mckib2 added a commit to mckib2/scipy that referenced this issue Dec 20, 2020
@mckib2
TEST: compare fix for issue scipy#12796 to R
865e8f8
mckib2 added a commit to mckib2/scipy that referenced this issue Jan 2, 2021
@mckib2
TEST: compare fix for issue scipy#12796 to R
4954b18
@mckib2 mckib2 mentioned this issue Jan 2, 2021
Merged
WarrenWeckesser added a commit to WarrenWeckesser/scipy that referenced this issue Jan 2, 2023
@WarrenWeckesser
ENH: special: Improvements for the incomplete beta functions.
f600896 
* Use boost's ibeta as the backend for special.betainc
* Use boost's ibeta_inv as the backend for special.betaincinv
* Add special.betaincc and special.betainccinv as wrappers of
  boost's ibetac and ibetac_inv, resp.
* Make special.btdtr and special.btdtri aliases of
  special.betainc and special.betaincinv, resp.
* Assorted tests added.

Closes scipygh-12796.
WarrenWeckesser added a commit to WarrenWeckesser/scipy that referenced this issue Jan 2, 2023
@WarrenWeckesser
ENH: special: Improvements for the incomplete beta functions.
33c0ed4 
* Use boost's ibeta as the backend for special.betainc
* Use boost's ibeta_inv as the backend for special.betaincinv
* Add special.betaincc and special.betainccinv as wrappers of
  boost's ibetac and ibetac_inv, resp.
* Make special.btdtr and special.btdtri aliases of
  special.betainc and special.betaincinv, resp.
* Add new tests.
* Move all tests of betainc, betaincinv, betaincc and betainccinv
  into a separate class.
* Move all tests of beta and betaln into a separate class.

Closes scipygh-12796.
WarrenWeckesser added a commit to WarrenWeckesser/scipy that referenced this issue Jan 2, 2023
@WarrenWeckesser
ENH: special: Improvements for the incomplete beta functions.
ee11719 
* Use boost's ibeta as the backend for special.betainc
* Use boost's ibeta_inv as the backend for special.betaincinv
* Add special.betaincc and special.betainccinv as wrappers of
  boost's ibetac and ibetac_inv, resp.
* Make special.btdtr and special.btdtri aliases of
  special.betainc and special.betaincinv, resp.
* Add new tests.
* Move all tests of betainc, betaincinv, betaincc and betainccinv
  into a separate class.
* Move all tests of beta and betaln into a separate class.

Closes scipygh-12796.
@WarrenWeckesser WarrenWeckesser mentioned this issue Jan 2, 2023
Merged
WarrenWeckesser added a commit to WarrenWeckesser/scipy that referenced this issue Jan 2, 2023
@WarrenWeckesser
ENH: special: Improvements for the incomplete beta functions.
1ca4d1e 
* Use boost's ibeta as the backend for special.betainc
* Use boost's ibeta_inv as the backend for special.betaincinv
* Add special.betaincc and special.betainccinv as wrappers of
  boost's ibetac and ibetac_inv, resp.
* Make special.btdtr and special.btdtri aliases of
  special.betainc and special.betaincinv, resp.
* Add new tests.
* Move all tests of betainc, betaincinv, betaincc and betainccinv
  into a separate class.
* Move all tests of beta and betaln into a separate class.

Closes scipygh-12796.
steppi pushed a commit that referenced this issue Aug 31, 2023
@WarrenWeckesser @lorentzenchr
ENH: special: Improvements for the incomplete beta functions. (#17697)
0419998 
* ENH: special: Improvements for the incomplete beta functions.

* Use boost's ibeta as the backend for special.betainc
* Use boost's ibeta_inv as the backend for special.betaincinv
* Add special.betaincc and special.betainccinv as wrappers of
  boost's ibetac and ibetac_inv, resp.
* Make special.btdtr and special.btdtri aliases of
  special.betainc and special.betaincinv, resp.
* Add new tests.
* Move all tests of betainc, betaincinv, betaincc and betainccinv
  into a separate class.
* Move all tests of beta and betaln into a separate class.

Closes gh-12796.

* TST: special: Loosen a couple tolerances.

* MAINT: special: Fix the following issue reported by the linter:

scipy/special/_add_newdocs.py:11:10: UP006 [*] Use `dict` instead of `Dict` for type annotations

* Add regression test for the boost issue (currenty fails)

* Override a couple Boost policies; update a test.

* fix whitespace

* fix a comment

* Slightly looser tolerance for a hyp1f1 test

* Reformat code in comment.

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>

* Restore cephes-based btdtr and btdri, and deprecate them.

* Tweak docstrings of betainc and betaincinv.

* DOC: Add Sphinx 'deprecated' directive in btdtr and btdtri docstrings.

* DOC: add scipy.stats.beta to 'See Also' for betainc and betaincc.

* Add a utility to modify deprecated ufunc docstrings, and use it for btdtr and btdtri.

* DOC: Explain what 'deprecate_ufunc()' does in more detail.

* TST (temporary): Define a local variable in a failing test, so it will show up in the error log.

* Filter deprecation warnings only for the deprecated functions.

* * Use custom deprecation wrappers instead of the deprecate_ufunc utility.
* In the btdtr and btdtri docstrings, move the `deprecated` directive
  from the notes to just before the paramers.

* Tell mypy to ignore some checks.

* Undo a couple trivial changes that ended up being committed.

* Update SciPy version for deprecation of btdtr and btdtri

* Use pytest.deprecated_call() context when testing deprecated functions.

* Two updates:

* Add recommended alternative to the deprecation messages of
  btdtr and btdtri.
* In special/__init__.py, change name depr_msg to _depr_msg.

---------

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>
@j-bowhay j-bowhay added this to the 1.12.0 milestone Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
defect A clear bug or issue that prevents SciPy from being installed or used as expected scipy.special
Projects
None yet
Milestone
1.12.0
Development

Successfully merging a pull request may close this issue.

ENH: special: Improvements for the incomplete beta functions. WarrenWeckesser/scipy
4 participants
@josef-pkt @mdhaber @cassiersg @j-bowhay