You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is not an issue, but a place for a conversation, around zarr as a library for reading via reference'd datasets and how xarray interprets it. Wasn't sure where to put this, we would discuss at the next meeting.
zarr requires regular chunking throughout. This is unlike dask-array, where chunks can be arbitrary sizes along each dimension, so the pieces themselves are ND cuboid. This limits the range of datasets that xarray(zarr(references)) can be applied to. Workarounds would be complex, involving subselection from read on-disk chunks and/or reading many chunks per virtual chunk
astro coordinates (for example) map poorly to the xarray model:
you may have a time-series composed of a repeated sequence of bandpass filters. Either each bandpass has its own time coordinate and so the filters cannot do arithmetic, you create a merged time coordinate where most values for any given filter are null, or you pretend that all the timestamps of a given sequence of filter are equal
astro (and dicom...) represents coordinates coordinates analytically instead of as materialised coordinates; multiple images of the same region of sky are typically offset in a mosaic (to build up a larger aggregate image, with overlaps), dither (to have the same sky exposed on multiple pixels to average out sensitivity variations) and jitter (natural movement of telescope). Furthermore, the instrument commonly has multiple detectors that may have gaps or overlaps.
what an astro (or radiologist) actually wants to do is, for example, "sum the intensity in this geometric area as a function of time" or "find the flux ratio between these two filters for an aperture around this point"
xarray assumes various CF conventions such as scale/offset of data, even though zarr has its own filters for dealing with that kind of thing. Other things like "unit" may also have special meaning even though they are more generic terms beyond CF.
The text was updated successfully, but these errors were encountered:
astro coordinates (for example) map poorly to the xarray model
I believe that most of these can eventually be solved via custom indexers. Under the hood I assume the raw data (not coordinates) are nd-arrays, no?
astro (and dicom...) represents coordinates coordinates analytically instead of as materialised coordinates
This is a bit similar to a projection on geospatial data. Merging discrete, irregularly sampled satellite images into a global map and then calculating statistics at each point is basically the killer feature of Google Earth Engine. Some stuff in the STAC space shows how we might do it in FOSS world.
xarray assumes various CF conventions such as scale/offset of data
This is not an issue, but a place for a conversation, around zarr as a library for reading via reference'd datasets and how xarray interprets it. Wasn't sure where to put this, we would discuss at the next meeting.
The text was updated successfully, but these errors were encountered: