Required changes in xarray to avoid creating indexes

[TomNicholas]

Everything mentioned in this issue pydata/xarray#8699 also needs to be done for this library to work.

EDIT: This is a lie, see comment below

[TomNicholas]

The above isn't quite true. We should distinguish here between

1. Things that are needed to support creating xarray Datasets with no indexes, which we want in order to avoid loading any data. This means Currently no way to create a Coordinates object without indexes for 1D variables pydata/xarray#8704 so we can pass indexes={} to xarray constructors. This would be useful because a concatenation problem where we know in advance the ordering of all datasets (i.e. combine_nested rather than combine_by_coords) doesn't require creating any indexes. I'll track that in this issue.

2. Things that are needed to support using the xarray backend entrypoint system to open datasets from disk as ManifestArray-backed Variables just by passing a keyword arg to open_dataset/open_mfdataset. This requires dodging some internal array wrapping that occurs in the depths of xarray's backend machinery. I'll track that in Opening via xarray backendentrypoint #35.

[TomNicholas]

1. Also requires Test concat of dimension coordinate not backed by an index #44 to be solved upstream in xarray.

EDIT: Raised pydata/xarray#8871 on xarray to track it.

[TomNicholas]

I made a branch on my fork of xarray, which contains the (unmerged upstream) PRs pydata/xarray#8711, pydata/xarray#8714 and pydata/xarray#8872. All three of these are needed for (1), i.e. to be able to create virtual datasets not backed by any indexes, and concatenate them without attempting to create any indexes.

Once those three changes are merged upstream and released I'll be able to close this issue (though #35 will still be unsolved.)

[TomNicholas]

Actually now the only unmerged PR to xarray that is needed for (1) is pydata/xarray#8872, so I've changed the VirtualiZarr build to only point to that in 08774f7.

[jsignell]

Is this now closable? 👀

[TomNicholas]

Yes I think so! But note that additional changes in xarray would be required to support #18 .