BUG: stats.nct.pdf inconsistent behavior when compared to MATLAB and R

[Martin2Constant]

Describe your issue.

Hello,

When computing stats.nctpdf with large t and nc values, the function returns 'inf'.
This behavior is inconsistent with both MATLAB and R. This "bug" appeared with version scipy 1.10 (behavior was consistent before).
I'm not sure whether it's truly a bug or whether MATLAB and R are both wrong in not returning inf.

The following R code (equivalent to the Python code) returns 0.0043403 instead of 'inf' (R version 4.3.1):

t <- 8.45316107575123
N <- 25
df <- N-1
safe_int <- .9999
safe_range <- t / sqrt(N) + c(-1, 1) * qt(1 - (1 - safe_int) / 2, df) / sqrt(N)

dt(t, df, ncp=safe_range[[2]]*sqrt(N))

Same value is returned in MATLAB (R2023b):

t = 8.45316107575123;
N = 25;
df = N-1;
safe_int = .9999;
safe_range = t / sqrt(N) + [-1, 1] * tinv(1 - (1 - safe_int) / 2, df) / sqrt(N);

nctpdf(t, df, safe_range(2)*sqrt(N))

Reproducing Code Example

import numpy as np
from scipy import stats

t = 8.45316107575123
N = 25
df = N-1
safe_int = .9999
safe_range = t / np.sqrt(N) + np.array([-1, 1]) * stats.t.ppf(1 - (1 - safe_int) / 2, df=df) / np.sqrt(N)

# SciPy 1.9.3 returns 0.0043403; SciPy 1.10 and above return Inf
stats.nct.pdf(t, df, nc=safe_range[1]*np.sqrt(N))

Error message

N/A

SciPy/NumPy/Python version and system information

1.11.3 1.26.0 sys.version_info(major=3, minor=10, micro=12, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/constama/miniforge3/Library/include
    lib directory: C:/Users/constama/miniforge3/Library/lib
    name: blas
    openblas configuration: unknown
    pc file directory: C:\bld\scipy-split_1696467770591\_h_env\Library\lib\pkgconfig
    version: 3.9.0
  lapack:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/constama/miniforge3/Library/include
    lib directory: C:/Users/constama/miniforge3/Library/lib
    name: lapack
    openblas configuration: unknown
    pc file directory: C:\bld\scipy-split_1696467770591\_h_env\Library\lib\pkgconfig
    version: 3.9.0
  pybind11:
    detection method: pkgconfig
    include directory: C:/Users/constama/miniforge3/Library/include
    name: pybind11
    version: 2.11.1
Compilers:
  c:
    commands: clang-cl
    linker: lld-link
    name: clang-cl
    version: 17.0.0
  c++:
    commands: clang-cl
    linker: lld-link
    name: clang-cl
    version: 17.0.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 0.29.36
  fortran:
    commands: flang-new
    linker: lld-link
    name: flang
    version: 17.0.0
  pythran:
    include directory: ..\..\_h_env\lib\site-packages\pythran
    version: 0.14.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
Python Information:
  path: C:\bld\scipy-split_1696467770591\_h_env\python.exe
  version: '3.10'

[WarrenWeckesser]

I'm not sure whether it's truly a bug or whether MATLAB and R are both wrong in not returning inf.

Thanks for reporting the issue. It is a bug, and it is still present in the development version of SciPy. Here's a simplified example, with integer parameter values so it is easy to compare to the result computed by Wolfram Alpha:

In [28]: import scipy

In [29]: scipy.__version__
Out[29]: '1.12.0.dev0+1813.0a3dce2'

In [30]: from scipy.stats import nct

In [31]: nct.pdf(8, 24, 13)
Out[31]: inf

With Wolfram Alpha, PDF[NoncentralStudentTDistribution[24, 13], 8] gives 0.0017644351379120543496495442757378120219461588078650315935117814288005136461742269261929088489060060675999625476400913578446784114357283664002733506219393887469961463578572949425885674236825307738127452 (after clicking on "More digits" a couple times).

[josef-pkt]

too much boost ?

>>> from scipy import stats
>>> stats.nct.pdf(8, 24, 13)
0.0017644351379121064
>>> stats.nct.pdf(8, 24, 13) - .0017644351379120543496
5.204170427930421e-17

>>> import scipy
>>> scipy.__version__
'1.7.3'

[mdhaber]

Boost isn't used by nct.pdf currently.
https://github.com/scipy/scipy/blob/main/scipy/stats/_continuous_distns.py#L7631-L7709

[WarrenWeckesser]

The development version of boost/math gives the correct result in this case. noncentralt_pdf.cpp takes x, df and nc from the command line and evaluates the PDF using boost's non_central_t distribution:

$ ./noncentralt_pdf 8 24 13
x  = 8
df = 24
nc = 13
p  =  1.7644351379120544e-03

As noted in the comment in the SciPy code, the boost implementation has issues in the left tail.

[mdhaber]

@WarrenWeckesser are you willing to open an issue with Boost about the left tail of the pdf? It would be nice if we could switch to Boost's implementation after all rather than stitching ours and theirs together. Here's a demonstration in SciPy (using Boost for the PDF):

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
dist = stats.nct(8, 8.5)
x = np.linspace(-3, 5)
plt.semilogy(x, dist.pdf(x))
plt.show()

Or maybe in the meantime just going with Boost is the lesser of two evils.

[josef-pkt]

one possible candidate for change compared to my version is
#17302 boost

(because pdf code itself is unchanged0

[WarrenWeckesser]

Thanks @josef-pkt, that is in fact the source of the problem reported here. When the b parameter of hyp1f1 is 1.5 and the third parameter (x) is greater than 50, hyp1f1 returns inf:

In [17]: import scipy

In [18]: scipy.__version__
Out[18]: '1.12.0.dev0+1814.166e1f2'

In [19]: from scipy.special import hyp1f1

In [20]: hyp1f1(13.0, [1.499999, 1.5, 1.500001], 61.0).tolist()
Out[20]: [1.3550915047757008e+39, inf, 1.3550800368458125e+39]

The problem is that the underlying boost function hypergeometric_1F1 throws an exception when it shouldn't. I reported the boost issue here: boostorg/math#1034.

I'll see if we can work around the boost bug in our wrapper of hypergeometric_1F1. This needs to be fixed regardless of whether or not we replace the current implementation of the PDF with the boost noncentral t PDF function.

The problem with the PDF calculation in the left tail is a separate issue. @mdhaber, if you already have the evidence, go ahead and file a boost issue. I probably won't get to that in the immediate future.

[mdhaber]

I ask because I don't ever work with C++. We'll see if they'll take the report in Python form.