Viewing .zarr files with tigger

[landmanbester]

At the moment I am creating zarr datasets containing image products (mainly because zarr plays nicely with s3 but also just because zarr is cool). This does however mean that I have to convert them to fits to view them with tigger. For example here

https://github.com/ratt-ru/pfb-clean/blob/37d328442141291981e1ef90198f7283d4439c4e/pfb/workers/grid.py#L392

It would be amazing if we could view these zarr files directly with tigger. All the information in the fits header can be specified as attributes of the zarr dataset.

[landmanbester]

As an example, if we have a zarr dataset containing multiple data variables we could select out a single one using eg.

xds = xds_from_zarr("test_I.mds.zarr", columns='MODEL')

In [6]: xds
Out[6]:
[<xarray.Dataset>
 Dimensions:  (band: 4, x: 484, y: 484)
 Dimensions without coordinates: band, x, y
 Data variables:
     MODEL    (band, x, y) float64 dask.array<chunksize=(1, 484, 484), meta=np.ndarray>
 Attributes:
     cell_rad:  5.43239476512341e-05
     dec:       0.5235987755982988
     nband:     4
     nx:        484
     ny:        484
     ra:        0.0]

Maybe we can come up with a command line syntax eg

$ tigger path/to/.zarr::COLNAME

or whatever. Happy to discuss the required format here.

[bennahugo]

Not sure I support directly supporting a non-standard internal format - could we look at making the format more conformant to the FITS standard before adding? Alternatively this could perhaps go into a helper script for tigger-convert?

…

On Thu, Jul 14, 2022 at 11:18 AM Landman Bester ***@***.***> wrote: As an example, if we have a zarr dataset containing multiple data variables we could select out a single one using eg. xds = xds_from_zarr("test_I.mds.zarr", columns='MODEL') In [6]: xds Out[6]: [<xarray.Dataset> Dimensions: (band: 4, x: 484, y: 484) Dimensions without coordinates: band, x, y Data variables: MODEL (band, x, y) float64 dask.array<chunksize=(1, 484, 484), meta=np.ndarray> Attributes: cell_rad: 5.43239476512341e-05 dec: 0.5235987755982988 nband: 4 nx: 484 ny: 484 ra: 0.0] Maybe we can come up with a command line syntax eg $ tigger path/to/.zarr::COLNAME or whatever. Happy to discuss the required format here. — Reply to this email directly, view it on GitHub <#146 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6TDCHTKSH2MGC3M3BTVT7LO7ANCNFSM53RQTLRQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- -- Benjamin Hugo PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[landmanbester]

We can in principle duplicate the full header as attributes of the dataset, not sure how much more conformant one could be. Converting to a fits file still duplicates the image data so won't really solve the problem. In fact I already have a utility for that

[landmanbester]

This https://github.com/ratt-ru/xarray-fits should make this much easier

[razman786]

Nice! Will look into this when I have a laptop again, well a working one again - taking longer than anticipated :'( 5 Apr 2024 16:04:05 Landman Bester ***@***.***>:

…

This https://github.com/ratt-ru/xarray-fits should make this much easier — Reply to this email directly, view it on GitHub[#146 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ABWJMGG474VTQTGHZRZZDNDY324OJAVCNFSM53RQTLR2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUGAYDGNRUGIZQ]. You are receiving this because you are subscribed to this thread. [Tracking image][https://github.com/notifications/beacon/ABWJMGCV7PDKZEEGPFY7IV3Y324OJA5CNFSM53RQTLR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOPGMHYRY.gif]