Description
To avoid the overhead of xarrays when working with data loaded with IMAS-python, I think it would be good to allow the conversion to and from python dictionaries to numpy masked arrays. I am suggesting masked arrays because that's the clearest way to deal with ragged data.
For the conversion from IMAS-python objects I think all the key ingredients are there:
- https://imas-python.readthedocs.io/en/stable/generated/imas.util.to_xarray.html#imas.util.to_xarray
- https://docs.xarray.dev/en/stable/generated/xarray.DataArray.to_dict.html
- Create a mask out of the fill values and convert to masked array for clarity. That's a manual step but is easy to do but potentially computationally expensive if there is a lot of data.
It would be nice though if it was possible to go directly from the netcdf tensor-layout to the python dictionary using IMAS-python. It is possible by just avoiding IMAS-python altogether by loading the netcdf file directly and then converting it.
Now the other direction from the python dictionary to an imas-compliant netcdf file is harder and I think this is were IMAS-python can help a lot by providing its own from_dict method like this https://docs.xarray.dev/en/stable/generated/xarray.DataArray.from_dict.html) method but use the existing xarray.DataArray encoding in IMAS-python for consistency checks and all the meta information from the schema.