Store with h5io

[liamhuber]

by default instead of tinybase.storage.h5io. Depends on all three of these PRs over on h5io, so like the root PR here (#160), there's no way for the tests to run on the CI.

After a bunch of upstream PRs here getting the __get/setstate__ in better shape, saving is just:

h5io.write_hdf5(
    fname=self._h5io_storage_file_path,
    data=self,
    title=self.label,
    use_state=True,
    overwrite=True,  # Don't worry about efficiency or updating yet
)

And loading is just:

inst = h5io.read_hdf5(fname=self._h5io_storage_file_path, title=self.label)
self.__setstate__(inst.__getstate__())

Which is nice and clean, but can fail if the data channels hold values that don't play nicely with h5io.

@pmrv and @jan-janssen, at some point I think I'll need a meeting with both of you together to help me work through the strengths and weaknesses of the different solutions available here. I'm slugging along steadily, but I feel a bit out of my depth.

Overall what I like about the h5io solution is that it is leveraging __get/setstate__, so it's largely enough for me to get (cloud)pickle happy with those and then I can just pass the whole objects to write_hdf5 or get them back with read_hdf5 (and use the returned object to set state). So it's extremely concise.

On the other hand tinybase.storage feels much more transparent on what it's storing where, and it is both more clear when it falls back to a pickle solution and much more powerful when it does it -- e.g. h5io.write_hdf5 borks on instances ASE's EMT because of some random and innocuous seeming method on the parent Calculator class; so I can't even store the atomistics workflows with this tool. Also, as much as it's more write, the fact tinybase.storage forces me to use something other than the state methods means that I have more control over the storage path, so nodes can be stored on their semantic path, e.g. my_wf/my_macro/some_node instead of my_wf/nodes/my_macro/nodes/some_node.

Finally, in the medium term, it will become important to resolve the versions of the stored object more vigorously, and ultimately have a framework that supports upgrading stored data to new (compatible) versions. I'm sure we can get this done, but the path forward is not yet super clear for me on this front.

TODO:

• Refactor; I just slapped this together in a rush and it's rather ugly (Not perfect, but good enough for while we're supporting two frameworks)
• Fix the docstrings so they apply to both solutions and explain the difference
• Re-run the deep-dive to make sure the example still works
• And of course updates once the dependent packages actually exist This is a wontfix unless the upstream out-of-repo PRs get finished and merged before I want to stack something on top of this

[github-actions]

👈 Launch a binder notebook on branch pyiron/pyiron_workflow/store_with_h5io

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB