Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow reloading of particular elements, and add reload: bool parameter to write functions #521

Open
4 tasks
LucaMarconato opened this issue Mar 27, 2024 · 0 comments

Comments

@LucaMarconato
Copy link
Member

With the new incremental IO PR we can write single elements or metadata into a Zarr store. However, reading is still a monolithic operation that reads all at once.

We should

  • refactor the code for reading into individual components to read single elements and metadata of single elements
  • add new APIs to reload the data, metadata and in particular transformations of a single element in a SpatialData object.
  • consider adding a parameter return_stored (as discussed here https://docs.dask.org/en/latest/generated/dask.array.store.html#dask.array.store), or maybe reload, to allow the user to call write_element() and write() and return an object that is already reloaded with Dask. write(reload=True) is equivalent to calling sdata = read_zarr(sdata.path)
  • write_element(reload=True) would introduce a new convenient functionality that has for instance been mentioned by @ArneDefauw here reimplemented incremental io #501 (comment), and could be used to simplify in particular the workaround 1 for overwriting elements (test_incremental_io_on_disk()).
@LucaMarconato LucaMarconato changed the title Allow reloading of particular elements, and add return_stored parameter to write functions Allow reloading of particular elements, and add reload: bool parameter to write functions Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant