Improve interp performance #4069
Conversation
|
Maybe I'll merge this in a few days. |
doc/whats-new.rst
Outdated
| @@ -34,6 +34,11 @@ Breaking changes | |||
| (:pull:`3274`) | |||
| By `Elliott Sales de Andrade <https://github.com/QuLogic>`_ | |||
|
|
|||
| Enhancements | |||
| ~~~~~~~~~~~~ | |||
| - Performance improvement of :py:meth:`DataArray.interp` and :py:func:`Dataset.interp` (:issue:`2223`) | |||
There was a problem hiding this comment.
can we add one more line describing the improvement?
xarray/tests/test_interp.py
Outdated
|
|
||
|
|
||
| @requires_scipy | ||
| def test_decompose(): |
There was a problem hiding this comment.
Can we test both linear and nearest methods?
| len(indexes_coords) > 1 | ||
| and method in ["linear", "nearest"] | ||
| and all(dest[1].ndim == 1 for dest in indexes_coords.values()) | ||
| and len(set([d[1].dims[0] for d in indexes_coords.values()])) |
There was a problem hiding this comment.
This condition is confusing me. This will not speed up this case:
da = xr.DataArray(
np.arange(6).reshape(3, 2),
dims=["x", "y"],
coords={"x": [0, 1, 2], "y": [-0.1, -0.3]},
)
x_new = xr.DataArray([0.5, 1.5, 2.5], dims=["x1"])
y_new = xr.DataArray([-0.15, -0.25, -0.35], dims=["x1"])
da.interp(x=x_new, y=y_new)
Correct? Is that intentional?
There was a problem hiding this comment.
Thanks @dcherian
It was intentional, but you are right. This case can be also improved.
da.interp(x=x_new, y=y_new)should be equivalent to
da.interp(x=x_new).interp(y=y_new)But then I'm a bit confused about which case should be improved.
For example if len(x_new) = len(y_new) = 1000000, then the original interpretation may be faster, although this is a rare use case.
Maybe we can use some heuristics, such as
len(x_new) < len(x)?
There was a problem hiding this comment.
I'm now thinking that the simpler behavior is better; for an orthogonal interpolation we interpolate sequentially and otherwise we use interpn.
Further improvement may be done in upstream.
There was a problem hiding this comment.
da = xr.DataArray( np.arange(6).reshape(3, 2), dims=["x", "y"], coords={"x": [0, 1, 2], "y": [-0.1, -0.3]}, ) x_new = xr.DataArray([0.5, 1.5, 2.5], dims=["x1"]) y_new = xr.DataArray([-0.15, -0.25, -0.35], dims=["x1"]) da.interp(x=x_new, y=y_new)
It looks that this case is not slow even with our current code.
The problem is when the final destination is a regular grid, where interpn will compute many times.
So, probably this PR should work good enough for this case.
|
I'll merge this tomorrow. |
* upstream/master: Improve interp performance (pydata#4069) Auto chunk (pydata#4064) xr.cov() and xr.corr() (pydata#4089) allow multiindex levels in plots (pydata#3938) Fix bool weights (pydata#4075) fix dangerous default arguments (pydata#4006)
isort -rc . && black . && mypy . && flake8whats-new.rstfor all changes andapi.rstfor new APINow n-dimensional interp works sequentially if possible.
It may speed up some cases.