Skip to content

Conversation

dcherian
Copy link
Contributor

@dcherian dcherian commented Jun 24, 2022

@shoyer is there a way to test this?

@shoyer
Copy link
Member

shoyer commented Jun 24, 2022

We have some tests with InaccessibleVariableDataStore for this sort of thing, but I don't know immediately how to hook that into the Zarr backend.

@shoyer
Copy link
Member

shoyer commented Jun 24, 2022

The simplest option would probably be a custom Zarr store that raises an error if you try to look at array data. This could be implemented as a subclass of an existing Zarr store (e.g., the in memory store) that raises an error in __getitem__ is the filename of requests does not start with ..

@dcherian
Copy link
Contributor Author

We have some tests with InaccessibleVariableDataStore for this sort of thing,

great tip! We don't need zarr actually, this was triggering loading of all lazy arrays.

@dcherian dcherian added the plan to merge Final call for comments label Jun 28, 2022
@andersy005
Copy link
Member

Thank you, @dcherian!

@andersy005 andersy005 merged commit 725258e into pydata:main Jun 29, 2022
@dcherian dcherian deleted the fix-lazy-array-chunks branch June 29, 2022 20:08
@rabernat
Copy link
Contributor

Awesome work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
plan to merge Final call for comments
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Accessing chunks on zarr backed xarray seems to load entire array into memory
4 participants