lowess errror: RuntimeWarning: invalid value encountered in double_scalars

[lozybean]

Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

3.6.1 (default, Dec 11 2017, 10:43:19) 
[GCC 4.7.2 20121015 (Red Hat 4.7.2-5)]
CPython
Linux-2.6.32-573.7.1.el6.x86_64-x86_64-with-centos-6.7-Final
1.69

Statement

Here is a example:

import numpy as np
from Bio.Statistics.lowess import lowess
lowess(np.array([0,1,2]),np.array([ 63.87636324,  66.11763915,  77.32401866]))

and numpy will raise a warning:

RuntimeWarning: invalid value encountered in double_scalars
  beta1 = (A22 * b1 - A12 * b2) / determinant
RuntimeWarning: divide by zero encountered in double_scalars
  beta1 = (A22 * b1 - A12 * b2) / determinant

it seems like determinant is nan in some cases.

it is also discussed in statsmodels/statsmodels#1798

[peterjc]

Confirming issue also present in Biopython 1.70 and the current master branch (what would become Biopython 1.71). Also tested under older versions of Python and Numpy which does not seem to matter (as I would have expect).

@mdehoon is this something you could advise on?

[peterjc]

In the referenced issue, the resolution was to return a flag value for this kind of bad input (something they already did in some cases):

statsmodels/statsmodels@f28610e

We currently only return another numpy array....

https://github.com/biopython/biopython/blob/biopython-170/Bio/Statistics/lowess.py#L38

[OscarMaestre]

Maybe a ZeroDivisionErrorException should be raised?

[peterjc]

I wonder if NumPy had good reason to use RuntimeWarning rather than ZeroDivisionError?

I suppose you can also give a more helpful error message with the more precise exception...

[dandyrilla]

I have the same problem. I've also encountered a warning when using BioPython's example dataset. It seems to produce the message with specific f and iter values.

Script:

# Example dataset from
# https://biopython.org/DIST/docs/api/Bio.Statistics.lowess-module.html
x = np.array([4,  4,  7,  7,  8,  9, 10, 10, 10, 11, 11, 12, 12, 12,
              12, 13, 13, 13, 13, 14, 14, 14, 14, 15, 15, 15, 16, 16,
              17, 17, 17, 18, 18, 18, 18, 19, 19, 19, 20, 20, 20, 20,
              20, 22, 23, 24, 24, 24, 24, 25], np.float)
y = np.array([2, 10,  4, 22, 16, 10, 18, 26, 34, 17, 28, 14, 20, 24,
              28, 26, 34, 34, 46, 26, 36, 60, 80, 20, 26, 54, 32, 40,
              32, 40, 50, 42, 56, 76, 84, 36, 46, 68, 32, 48, 52, 56,
              64, 66, 54, 70, 92, 93, 120, 85], np.float)
yest = lowess(x, y, f=0.1, iter=1)

Warning:

/home/lab/anaconda3/envs/Sukjun/lib/python3.6/site-packages/biopython-1.73-py3.6-linux-x86_64.egg/Bio/Statistics/lowess.py:89: RuntimeWarning: invalid value encountered in double_scalars
  beta1 = (A22 * b1 - A12 * b2) / determinant
/home/lab/anaconda3/envs/Sukjun/lib/python3.6/site-packages/biopython-1.73-py3.6-linux-x86_64.egg/Bio/Statistics/lowess.py:90: RuntimeWarning: invalid value encountered in double_scalars
  beta2 = (A11 * b2 - A21 * b1) / determinant
/home/lab/anaconda3/envs/Sukjun/lib/python3.6/site-packages/numpy/lib/function_base.py:3405: RuntimeWarning: Invalid value encountered in median
  r = func(a, **kwargs)

Python: 3.6.9
BioPython: 1.73
NumPy: 1.16.4

[mdehoon]

I would suggest to use SciPy instead of Biopython for this. Also, in Biopython 1.76, Bio.Statistics is deprecated, so it may exist for much longer.

[peterjc]

Closed by #3341