cephes_smirnov nan-to-int conversion bug (Trac #1638)

[scipy-gitbot]

Original ticket http://projects.scipy.org/scipy/ticket/1638 on 2012-03-30 by @pv, assigned to @pv.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=653948
http://permalink.gmane.org/gmane.comp.python.scientific.devel/16262

---

The cephes_smirnov function in kolmogorov.c can go into a very long loop, due to the following line of code

      double e;
      ...
      nn = (int) (floor ((double) n * (1.0 - e)));

If e happens to be nan, the value of nn is unspecified. It is used as a loop upper bound, so it will effectively hang the process.

The cast miraculously seems to work OK (gives zero?) on x86, but produces large numbers on MIPS.

There are probably also other instances of this bug in the code. All float-to-integer casts where the operand may be NAN probably should be guarded by n = (int)x; if (x - n != 0) { mtherr(...); return NPY_NAN; } or something like that (this would also protect against integer overflow).

[scipy-gitbot]

@pv wrote on 2012-03-31

PR http://github.com/scipy/scipy/pull/187

[scipy-gitbot]

@pv wrote on 2012-05-11

Fixed in 3418836f832

[scipy-gitbot]

Milestone changed to 0.11.0 by @pv on 2012-05-11