Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polyfit fails with few non-NaN values #4190

Closed
sfinkens opened this issue Jul 1, 2020 · 1 comment · Fixed by #4193
Closed

Polyfit fails with few non-NaN values #4190

sfinkens opened this issue Jul 1, 2020 · 1 comment · Fixed by #4193

Comments

@sfinkens
Copy link

sfinkens commented Jul 1, 2020

What happened:
A linear DataArray.polyfit seems to fail if there are less than 3 non-NaN elements along the fitting dimension.

Traceback
TypeError: only size-1 arrays can be converted to Python scalars

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "polyfit.py", line 6, in <module>
    out = arr.polyfit(dim='x', deg=1)
  File "/home/stephan/venv/variogram/lib/python3.8/site-packages/xarray/core/dataarray.py", line 3455, in polyfit
    return self._to_temp_dataset().polyfit(
  File "/home/stephan/venv/variogram/lib/python3.8/site-packages/xarray/core/dataset.py", line 5962, in polyfit
    coeffs, residuals = duck_array_ops.least_squares(
  File "/home/stephan/venv/variogram/lib/python3.8/site-packages/xarray/core/duck_array_ops.py", line 625, in least_squares
    return nputils.least_squares(lhs, rhs, rcond=rcond, skipna=skipna)
  File "/home/stephan/venv/variogram/lib/python3.8/site-packages/xarray/core/nputils.py", line 239, in least_squares
    out[:, nan_cols] = np.apply_along_axis(
  File "<__array_function__ internals>", line 5, in apply_along_axis
  File "/home/stephan/venv/variogram/lib/python3.8/site-packages/numpy/lib/shape_base.py", line 379, in apply_along_axis
    res = asanyarray(func1d(inarr_view[ind0], *args, **kwargs))
  File "/home/stephan/venv/variogram/lib/python3.8/site-packages/xarray/core/nputils.py", line 227, in _nanpolyfit_1d
    out[:-1], out[-1], _, _ = np.linalg.lstsq(x[~mask, :], arr[~mask], rcond=rcond)
ValueError: setting an array element with a sequence.

I've played around with the degree a little bit and the error seems to occur as soon as (# of non-NaN values - degree) < 2

What you expected to happen:

The fit to succeed - I think two non-NaN values should be enough for a linear fit. I also noticed that there is no RankWarning: Polyfit may be poorly conditioned if the degree is larger than the number of non-NaN values.

Minimal Complete Verifiable Example:

import xarray as xr
import numpy as np

arr = xr.DataArray([np.nan, 1, 2], dims='x', coords={'x': [0, 1, 2]})
arr.polyfit(dim='x', deg=1)

Anything else we need to know?:

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.2 (default, Apr 27 2020, 15:53:34)
[GCC 9.3.0]
python-bits: 64
OS: Linux
OS-release: 5.4.0-39-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.3

xarray: 0.15.2.dev112+g54b9450b
pandas: 1.0.5
numpy: 1.19.0
scipy: 1.5.0
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.4.0
cftime: 1.1.3
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.19.0
distributed: 2.19.0
matplotlib: 3.2.2
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 44.0.0
pip: 20.0.2
conda: None
pytest: None
IPython: 7.16.1
sphinx: None

@dcherian
Copy link
Contributor

dcherian commented Jul 1, 2020

Thanks for the report @sfinkens

cc @aulemahal

aulemahal added a commit to aulemahal/xarray that referenced this issue Aug 17, 2020
mathause added a commit that referenced this issue Aug 20, 2020
* Fix polyfit fail on deficient rank

* Add docs and RankWarning

* Fix deficient ranks outputs | workaround dask bug | add tests

* Add a note to the doc | whats new entry

* Update xarray/core/nputils.py

Apply suggestion from review.

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

* Fix test and catch warnings

* forgot to run black

* adapt polyfit test to properly test issue #4190

* Fix syntax in doc/whats-new.rst

Co-authored-by: keewis <keewis@users.noreply.github.com>

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>
Co-authored-by: keewis <keewis@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants