Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

groupby doesn't work when a dimension is resized as part of apply #1600

Closed
shoyer opened this issue Sep 30, 2017 · 1 comment
Closed

groupby doesn't work when a dimension is resized as part of apply #1600

shoyer opened this issue Sep 30, 2017 · 1 comment

Comments

@shoyer
Copy link
Member

shoyer commented Sep 30, 2017

In [60]: da = xarray.DataArray([1, 2, 3], dims='x', coords={'y': ('x', [1, 1, 1])})

In [61]: da.groupby('y').apply(lambda x: x[:2])
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-61-4c28a4712c34> in <module>()
----> 1 da.groupby('y').apply(lambda x: x[:2])

~/dev/xarray/xarray/core/groupby.py in apply(self, func, shortcut, **kwargs)
    516         applied = (maybe_wrap_array(arr, func(arr, **kwargs))
    517                    for arr in grouped)
--> 518         return self._combine(applied, shortcut=shortcut)
    519
    520     def _combine(self, applied, shortcut=False):

~/dev/xarray/xarray/core/groupby.py in _combine(self, applied, shortcut)
    526         else:
    527             combined = concat(applied, dim)
--> 528             combined = _maybe_reorder(combined, dim, positions)
    529
    530         if isinstance(combined, type(self._obj)):

~/dev/xarray/xarray/core/groupby.py in _maybe_reorder(xarray_obj, dim, positions)
    436         return xarray_obj
    437     else:
--> 438         return xarray_obj[{dim: order}]
    439
    440

~/dev/xarray/xarray/core/dataarray.py in __getitem__(self, key)
    476         else:
    477             # orthogonal array indexing
--> 478             return self.isel(**self._item_key_to_dict(key))
    479
    480     def __setitem__(self, key, value):

~/dev/xarray/xarray/core/dataarray.py in isel(self, drop, **indexers)
    710         DataArray.sel
    711         """
--> 712         ds = self._to_temp_dataset().isel(drop=drop, **indexers)
    713         return self._from_temp_dataset(ds)
    714

~/dev/xarray/xarray/core/dataset.py in isel(self, drop, **indexers)
   1172         for name, var in iteritems(self._variables):
   1173             var_indexers = dict((k, v) for k, v in indexers if k in var.dims)
-> 1174             new_var = var.isel(**var_indexers)
   1175             if not (drop and name in var_indexers):
   1176                 variables[name] = new_var

~/dev/xarray/xarray/core/variable.py in isel(self, **indexers)
    596             if dim in indexers:
    597                 key[i] = indexers[dim]
--> 598         return self[tuple(key)]
    599
    600     def squeeze(self, dim=None):

~/dev/xarray/xarray/core/variable.py in __getitem__(self, key)
    426         dims = tuple(dim for k, dim in zip(key, self.dims)
    427                      if not isinstance(k, integer_types))
--> 428         values = self._indexable_data[key]
    429         # orthogonal indexing should ensure the dimensionality is consistent
    430         if hasattr(values, 'ndim'):

~/dev/xarray/xarray/core/indexing.py in __getitem__(self, key)
    476     def __getitem__(self, key):
    477         key = self._convert_key(key)
--> 478         return self._ensure_ndarray(self.array[key])
    479
    480     def __setitem__(self, key, value):

IndexError: index 2 is out of bounds for axis 1 with size 2

This would be useful, for example, for grouped sampling:
https://stackoverflow.com/questions/46498247/how-to-downsample-xarray-dataset-using-groupby

To fix this, we will need to update our heuristics that decide if a groupby operation is a "transform" type operation that should have the output reordered to the original order:

def _infer_concat_args(self, applied_example):
if self._group_dim in applied_example.dims:
coord = self._group
positions = self._group_indices
else:
coord = self._unique_coord
positions = None

maaleske added a commit to silmae/fpipy that referenced this issue Aug 20, 2018
@dcherian
Copy link
Contributor

Fixed by #3807

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants