Dataset.__delitem__() kills dimensions dictionary

[ms8r]

Trying to re-align slocum with xray... it seems that deleting a variable from a Dataset object erases the dimensions dictionary:

In [47]: fcst.dimensions
Out[47]: Frozen(OrderedDict([(u'lat', 9), (u'lon', 9), (u'height_above_ground4', 1), (u'time', 65)]))

In [48]: del fcst['height_above_ground4']

In [49]: fcst.dimensions
Out[49]: Frozen(OrderedDict())

Also, the removed variable still appears as a dimension in the remaining variables' coordinate systems:

In [57]: fcst.variables
Out[57]: Frozen(_VariablesDict([(u'lat', <xray.XArray (lat: 9): object>), (u'lon', <xray.XArray (lon: 9): object>), (u'time', <xray.XArray (time: 65): datetime64[ns]>), (u'u-component_of_wind_height_above_ground', <xray.XArray (time: 65, height_above_ground4: 1, lat: 9, lon: 9): float32>), (u'v-component_of_wind_height_above_ground', <xray.XArray (time: 65, height_above_ground4: 1, lat: 9, lon: 9): float32>)]))

Is this intentional? I'm using delitem as replacement for the old polyglot's Dataset.squeeze() - perhaps that's abuse?

[shoyer]

I removed Dataset.squeeze because I've never actually had a use-case for np.squeeze. But it would be very easy to bring it back if you find it genuinely useful. The equivalent in the current version of xray would be
fcst.indexed_by(height_above_ground=0), which would select out (a view) of the 0th element along the forecast dimension.

__delitem__ is intended to remove a variable from a dataset but I have not tested it very carefully. Since every dimension is a variable, I think the right behavior would be to delete every variable that uses the dimension. So the later behavior you found is definitely not right! I'll see if I can put in a fix for that.

[ms8r]

Many thanks for the clarification. np.squeeze was used in slocum to remove a dimension that had been shrunk down to one value (then via views, now via indexed_by). The idea was to get the resulting Dataset as small as possible before dumping it and sending it over a very low bandwidth email link. If a dimension with only one element (like height_above_ground in the example) can be neglected in terms of size impact, it's not worth the trouble with np.squeeze. Otherwise it would be nice to have it back. Thanks in any case.

I think the right behavior would be to delete every variable that uses the dimension.

How about deleting it from every variable that uses the dimension to be deleted, and only keep index 0 if there were multiple values along that dimension. That would seem closer to what happens in a n-dimensional coordinate system if I get rid of one axis?

[shoyer]

OK, I just added squeeze back in PR #34.

One change I made from in indexed_by from views is that it automatically squeezes out dimensions if you give an integer as the indexer. So fcst.indexed_by(height_above_ground=0) removes the height_above_ground dimension. If you wanted to keep that dimension with size 1, you would need to do fcst.indexed_by(height_above_ground=slice(0, 1)) or fcst.indexed_by(height_above_ground=[0]). This behavior is more consistent with numpy, especially because I use indexed_by under the hood for indexing with DatasetArray objects.

Your suggestion for __delitem__ is interesting but with the current implementation of squeeze (and indexed_by more generally) I'm not sure how to make it work, because we currently have to load a variable into memory as a numpy array to squeeze out a dimension. This means that deleting a variable could cause other variables to be loaded. My expectation (in general) is that deleting an item should only release resources, and do so in a fairly lightweight fashion.

[ms8r]

Many thanks! I hadn't realized the indexed_by behavior for integer indexers - that's great. In that case my __delitem__ suggestion becomes superfluous anyway, since what I described can be achieved by indexed_by as it appears.

The new polyglot aka xray looks very impressive - and the name is cool...

[shoyer]

Closing this issue since it was resolved by #34 (now merged into master).