-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Writing data to disk (Xarray) #8
Comments
Ah, looks like engine (installed libraries) issue. This defaults to netCDF3, which doesn't support complex128 np default type. Installing libraries should fix, or switching dtype to one supported before save. http://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_netcdf.html |
For scipy backend, splitting to re + im is one solution: |
Also issues with multi-level indexing here (netCDF3), so need to flatten/unstack before save, and reconstruct upon read. Now implemented in read/write wrapper functions for Xarray. Haven't tested for netCDF4 yet. |
Now having similar issues (with current write function) on AntonJr runs... need to check installed libraries and address. |
Read & write debugged (again)... now OK for default netCDF3 Xarray behaviour. Still untested for other HDF libraries. |
With newer Xarray versions (0.13+) now get issues with netCDF3 and multidim attribs (previously just gave a warning). |
Also need to wrap this for job and multijob class methods! For a quick fix, can just push tabulated data to CSV using Pandas:
|
Briefly revisited for PEMtk data IO...
Example from the Xarray docs:
|
Improved complex number handling in writeXarray() as of e3988d0 For additional work with other IO formats (Pandas > HDF5, Pickle and others), see PEMtk library, esp. phockett/PEMtk@47b3541 and phockett/PEMtk#6 |
Added more general restack() functionality, see fa8522e. Since implemented for ep.IO.readXarray(). Updated docs: https://epsproc.readthedocs.io/en/dev/dataStructures/ePSproc_dataStructures_demo_070622.html |
For more general to/from dict and to/from HDF5 functionality, see pydata/xarray#4073 TODO: wrap some of this into base code and move to h5py as preferred IO method. |
Status 29/06/22:
TODO:
Recent collected IO dev update
|
NEW BUG: currently failing for HDF5 and AFBLM data: YET ANOTHER ANNOYING ISSUE IN CHECKDIMS() routine. This needs to be fixed ASAP. e.g. Fails at nonDimStacked line 355. Specifically XR core .to_index() Tested in XR 0.19, Pandas 1.2.4 only. UPDATE:
E.g.
Options/todo:
|
Minor IO changes in b00a966 seem to have broken IO tests... or possibly revealed some underlying issues. E.g. from https://github.com/phockett/ePSproc/actions/runs/4679018663/jobs/8288471741
TODO: local testing/checks to confirm what is going on. Also check older CI runs to confirm this is a new bug. |
Xarray netCDF functionality: Currently fails with complex data-type issues.
Use HDF5 instead...?
Can also just pickle for now, but not recommended for Xarrays.
The text was updated successfully, but these errors were encountered: