Process error: Cannot apply_along_axis when any iteration dimensions are 0

[cehbrecht]

• daops version: 0.5.0
• Python version:
• Operating System:

Description

Error on user request in production system.

See notebook, error 21, 24.03:
https://nbviewer.jupyter.org/github/roocs/rooki/blob/master/notebooks/tests/test-c3s-cmip6-subset-errors-dkrz-2021-03-23.ipynb

What I Did

Run:

wf = ops.Subset(
        ops.Input(
            'tos', ['c3s-cmip6.ScenarioMIP.CNRM-CERFACS.CNRM-CM6-1.ssp245.r1i1p1f2.Omon.tos.gn.v20190219']
        ),
        # time="2021-01-01/2050-12-31",
        area="1,40,2,4"
)
resp = wf.orchestrate()
resp.status

Error:

Process error: Cannot apply_along_axis when any iteration dime
nsions are 0  

Traceback (most recent call last):
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/rook/director/director.py", line 156, in process
    file_uris = runner(self.inputs)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/rook/utils/subset_utils.py", line 5, in run_subset
    result = subset(**args)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/daops/ops/subset.py", line 77, in subset
    result_set = Subset(**locals()).calculate()
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/daops/ops/base.py", line 88, in calculate
    process(self.get_operation_callable(), norm_collection, **self.params),
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/daops/processor.py", line 19, in process
    result = operation(dset, **kwargs)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/clisops/ops/subset.py", line 165, in subset
    return op.process()
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/clisops/ops/base_operation.py", line 89, in process
    processed_ds = self._calculate()
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/clisops/ops/subset.py", line 63, in _calculate
    result = subset_bbox(ds, **self.params)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/clisops/core/subset.py", line 251, in func_checker
    return func(*args, **kwargs)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/clisops/core/subset.py", line 875, in subset_bbox
    da[var] = da[var].where(lon_cond & lat_cond, drop=True)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/common.py", line 1273, in where
    return ops.where_method(self, cond, other)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/ops.py", line 203, in where_method
    keep_attrs=True,
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/computation.py", line 1134, in apply_ufunc
    keep_attrs=keep_attrs,
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/computation.py", line 271, in apply_dataarray_vfunc
    result_var = func(*data_vars)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/computation.py", line 632, in apply_variable_ufunc
    for arg, core_dims in zip(args, signature.input_core_dims)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/computation.py", line 632, in <listcomp>
    for arg, core_dims in zip(args, signature.input_core_dims)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/computation.py", line 542, in broadcast_compat_data
    data = variable.data
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/variable.py", line 374, in data
    return self.values
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/variable.py", line 554, in values
    return _as_array_or_item(self._data)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/variable.py", line 287, in _as_array_or_item
    data = np.asarray(data)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/numpy/core/_asarray.py", line 102, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/indexing.py", line 693, in __array__
    self._ensure_cached()
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/indexing.py", line 690, in _ensure_cached
    self.array = NumpyIndexingAdapter(np.asarray(self.array))
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/numpy/core/_asarray.py", line 102, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/indexing.py", line 663, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/numpy/core/_asarray.py", line 102, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/indexing.py", line 568, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 86, in __getitem__
    key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
  File "/usr/local/anaconda/envs/rook/lib/python3.7/site-packages/xarray/core/indexing.py", line 853, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)

[agstephens]

Possible related issue:

roocs/clisops#65

[ellesmith88]

@agstephens @cehbrecht

This is actually related to #61 but this time we get a different error because this is a curvilinear grid, and subset_bbox tries to mask variables after the initial subset.

The longitude bounds mean that there is no x data in the requested subset.

I think we need an extra check in subset_bbox for the curvilinear case along the lines of

if da[lat].size == 0 or da[lon].size == 0:
    raise ValueError(
        "There were no valid data points found in the requested subset. Please expand "
        "the area covered by the bounding box."
    )

as well as the check implemented for #61, which checks after the level and time subsets as well.

[ellesmith88]

The order of the latitude bounds doesn't make a difference in this case because the bounds get flipped by assign_bounds which is called here: https://github.com/roocs/clisops/blob/3e89de464db8569126b91aaa2b1ad8aa50eade2d/clisops/core/subset.py#L834-L836

[ellesmith88]

I tried to recreate this with a test dataset - still curvilinear and tos but a different dataset (/badc/cmip6/data/CMIP6/CMIP/MPI-M/MPI-ESM1-2-HR/historical/r1i1p1f1/Omon/tos/gn/v20190710/). I used one time step with the full lat/lon.

I've tried various lat/lon bounds but can't get the same issue, however it has identified another error that can occur with curvilinear grids when the lat/ lon bounds are too small:

clisops/ops/subset.py:168: in subset
    return op.process()
clisops/ops/base_operation.py:89: in process
    processed_ds = self._calculate()
clisops/ops/subset.py:64: in _calculate
    result = subset_bbox(ds, **self.params)
clisops/core/subset.py:251: in func_checker
    return func(*args, **kwargs)
clisops/core/subset.py:855: in subset_bbox
    coord=da[d], bounds=[coords.min().values, coords.max().values], dim=d
../venv/lib/python3.7/site-packages/xarray/core/common.py:46: in wrapped_func
    return self.reduce(func, dim, axis, skipna=skipna, **kwargs)
../venv/lib/python3.7/site-packages/xarray/core/dataarray.py:2518: in reduce
    var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, **kwargs)
../venv/lib/python3.7/site-packages/xarray/core/variable.py:1633: in reduce
    data = func(self.data, **kwargs)
../venv/lib/python3.7/site-packages/xarray/core/duck_array_ops.py:337: in f
    return func(values, axis=axis, **kwargs)
../venv/lib/python3.7/site-packages/xarray/core/duck_array_ops.py:56: in f
    return wrapped(*args, **kwargs)
<__array_function__ internals>:6: in amin
    ???
../venv/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2793: in amin
    keepdims=keepdims, initial=initial, where=where)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = array([], dtype=int32), ufunc = <ufunc 'minimum'>, method = 'min', axis = None, dtype = None, out = None, kwargs = {'initial': <no value>, 'keepdims': <no value>, 'where': <no value>}, passkwargs = {}

    def _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs):
        passkwargs = {k: v for k, v in kwargs.items()
                      if v is not np._NoValue}
    
        if type(obj) is not mu.ndarray:
            try:
                reduction = getattr(obj, method)
            except AttributeError:
                pass
            else:
                # This branch is needed for reductions like any which don't
                # support a dtype.
                if dtype is not None:
                    return reduction(axis=axis, dtype=dtype, out=out, **passkwargs)
                else:
                    return reduction(axis=axis, out=out, **passkwargs)
    
>       return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
E       ValueError: zero-size array to reduction operation minimum which has no identity

../venv/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: ValueError

[ellesmith88]

closed by roocs/clisops#156