Transpose modifies dtype of index, when a PeriodIndex

[max-sixty]

This is very peculiar & specific, but also fairly impactful for us.

If you

• Create a Dataset with a coord that is a PeriodIndex
• Transpose that coord
• Add a variable to the Dataset that needs to be reindexed

...then the type of the index changes from object to int64. This then causes other arrays added to that dataset to show up as NaNs throughout.

Here's an example. Note the dtype('O')) at the end of each output.

In [61]:
series = pd.Series(np.random.rand(10),index=pd.period_range(start='2000', periods=10,name='date'))
​
ds = xray.Dataset({'number 1':series})
ds['number 2'] = ds['number 1']
ds, ds.date.dtype
Out[61]:
(<xray.Dataset>
 Dimensions:   (date: 10)
 Coordinates:
   * date      (date) object 10957 10958 10959 10960 10961 10962 10963 10964 ...
 Data variables:
     number 1  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...
     number 2  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...,
 dtype('O'))
In [62]:

ds, ds.date.dtype
ds=ds.transpose('date')
ds, ds.date.dtype
Out[62]:
(<xray.Dataset>
 Dimensions:   (date: 10)
 Coordinates:
   * date      (date) object 10957 10958 10959 10960 10961 10962 10963 10964 ...
 Data variables:
     number 1  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...
     number 2  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...,
 dtype('O'))
In [63]:

ds
ds['number 3'] = ds['number 1']
ds, ds.date.dtype
Out[63]:
(<xray.Dataset>
 Dimensions:   (date: 10)
 Coordinates:
   * date      (date) object 10957 10958 10959 10960 10961 10962 10963 10964 ...
 Data variables:
     number 1  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...
     number 2  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...
     number 3  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...,
 dtype('O'))
In [64]:

ds
ds['number 4'] = ds['number 1'][:5]
ds, ds.date.dtype
Out[64]:
(<xray.Dataset>
 Dimensions:   (date: 10)
 Coordinates:
   * date      (date) int64 10957 10958 10959 10960 10961 10962 10963 10964 ...
 Data variables:
     number 1  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...
     number 2  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...
     number 3  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 0.6723 ...
     number 4  (date) float64 0.1133 0.5952 0.5467 0.2035 0.2022 nan nan nan ...,
 dtype('int64'))

[max-sixty]

Also peculiar - setting the coord to the original coord (saved above, but not shown here) causes the ds.date.dtype to report 'O', although not to display that in its repr, nor to cascade that for a DataArray extracted from it.

In [80]:

ds.date = date
ds, ds.date.dtype, ds['number 1'].date.dtype
Out[80]:
(<xray.Dataset>
 Dimensions:   (date: 10)
 Coordinates:
   * date      (date) int64 10957 10958 10959 10960 10961 10962 10963 10964 ...
 Data variables:
     number 1  (date) float64 0.1225 0.7202 0.5921 0.3005 0.8746 0.4186 ...
     number 2  (date) float64 0.1225 0.7202 0.5921 0.3005 0.8746 0.4186 ...
     number 3  (date) float64 0.1225 0.7202 0.5921 0.3005 0.8746 0.4186 ...
     number 4  (date) float64 0.1225 0.7202 0.5921 0.3005 0.8746 nan nan nan ...,
 dtype('O'),
 dtype('int64'))

Currently my workaround is ds=ds.assign_coords(**dict(date=date.to_index()))

[shoyer]

Hmm. One possible fix would be adjust Variable.transpose so it's a complete no-op (just returning a shallow copy) for variables with ndim < 2. Right now, it will convert everything into a numpy or dask array to do the transpose.

[max-sixty]

Yeah, that seems fairly hacky (presumably this is would still be a problem if we have nim > 2?)

From my POV I'm still thinking whether XRay can fall back to pandas indexing more extensively (as discussed here). Dims could be stored as Pandas indexes rather than dask / numpy arrays, and the (much cleaner) XRay indexing API could be a simple wrapper of the Pandas API.
And it sounds like from your comment the lazy contract could still be honored, if they're loaded when needed?

Does that make sense? What are the arguments in the other direction?

[shoyer]

The challenge is that the pandas Index has a much smaller API than numpy/dask.array. For example, it doesn't have a transpose method that accepts an axis argument. As another example, there's no equivalent of numpy.stack that would expand the contents of an Index to multiple dimensions. So we're pretty much stuck converting to numpy for these sort of situations.

I'm all for preserving indexes (including all their quirks) in situations where it's possible. And in other cases, we should fall back to numpy arrays with dtype=object holding appropriate scalars.

As for this particular issue, have you tested this after the changes in #691? I actually can't reproduce this issue on master.

[max-sixty]

That did fix it! Apologies for not testing on master.

On the broader issue, I hadn't thought of the need to expand dims / indexes to more than one dimension. I was thinking that there would be a 1D index per dim, and the result of slicing a multidimensional array would be the intersection of each dim's slice. Am I missing something more fundamental?

[shoyer]

Am I missing something more fundamental?

You're right that this won't come up in many circumstances. It's only that, like with numpy, we try to make xray operations always generalize to multi-dimensional arrays.

[max-sixty]

OK, I guess that's how I think about the distinction with Indexes - they're role is for selection in one dimension, each.

Maybe this is a more conceptual issue then...