Skip to content

feat: Add user-accessible APIs for managing shard index caching - #137

Merged
kylebarron merged 10 commits into
mainfrom
kyle/user-accessible-shard-cache
Aug 4, 2026
Merged

feat: Add user-accessible APIs for managing shard index caching#137
kylebarron merged 10 commits into
mainfrom
kyle/user-accessible-shard-cache

Conversation

@kylebarron

@kylebarron kylebarron commented Aug 4, 2026

Copy link
Copy Markdown
Member

Change list

  • Adds two new classes: ShardCache and AsyncShardCache. These map to the upstream ArrayShardedReadableExtCache and AsyncArrayShardedReadableExtCache, which are required arguments when reading sharded data.
  • These classes hold the shard index. So if you read multiple shards from a single array, it should only have to read the shard metadata once.

Todo:

  • We need to validate that these shard caches are tied to the same array. They can't be used across multiple different Array objects I don't think. We'll improve this validation when we implement array equality.

Closes #125

@kylebarron
kylebarron marked this pull request as ready for review August 4, 2026 21:16
@kylebarron
kylebarron requested a review from d-v-b August 4, 2026 21:18
@kylebarron
kylebarron merged commit 55e5cb7 into main Aug 4, 2026
15 checks passed
@kylebarron
kylebarron deleted the kyle/user-accessible-shard-cache branch August 4, 2026 21:27
@kylebarron

Copy link
Copy Markdown
Member Author

@d-v-b I'm merging this to make it easier to iterate; but happy to make changes if you have feedback!

@d-v-b

d-v-b commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

+1 to moving fast! my two questions after a cursory read -- how are nested subchunks handled, and what's the contract between an arbitrary array-bytes codec and the array that allows the array to use the cache? I assume there's a trait in zarrs for "contains subchunks".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow user to manage subchunk read cache

2 participants