Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writing class data to disk #6

Open
phockett opened this issue May 5, 2022 · 2 comments
Open

Writing class data to disk #6

phockett opened this issue May 5, 2022 · 2 comments

Comments

@phockett
Copy link
Owner

phockett commented May 5, 2022

Quick-fix is Pickle, but quite brittle to package version changes (e.g. recently broke after updating XR & Pandas versions).

Otherwise need to wrap something per-dataset/dict/dataframe/dataarray...? This is probably OK for aggregate cases, but a bit unwieldy for batch fit cases.

Basics at https://docs.xarray.dev/en/stable/user-guide/io.html

For more on this (for IO in ePSproc) see phockett/ePSproc#8

@phockett
Copy link
Owner Author

phockett commented Jun 3, 2022

Some basics in places as of 47b3541

  • Pickle for all self.data
  • Wrapped Pandas (>HDF5) and Xarray (>netCDF) routines tested for aggregate 'fits' data.
  • Basic Xarray > dict > JSON also test, but not yet wrapped to a function.

See also phockett/ePSproc#8 (comment)

Dev notes: code-share/jupyter-shared/PEMtk_dev_2022/io_dev/pemtk_dataIO_tests-dev_200522_StimpyDocker.ipynb

TODO:

  • Finish testing & docs.
  • Generalise methods, currently mainly tested with aggregate fit data. Should be OK for other data types too.
  • Finish identifying issues - currently problems with nested Xarrays and tuple coord data (Euler dim) for HDF5 or JSON serialization. May be able to avoid and/or convert such cases?
  • Custom routine with h5py? Might be best way to handle some tricky cases.
  • Fix ep.readXarray restacking, should include dim checking routines to avoid failure.

@phockett
Copy link
Owner Author

phockett commented Jun 29, 2022

Some of the above now fixed by ePSproc updates (see phockett/ePSproc#8 (comment)), see also docs: https://epsproc.readthedocs.io/en/dev/dataStructures/ePSproc_dataStructures_IO_demo_280622.html

Needs testing with and/or porting to PEMtk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant