unconsolidated groups in s3 list themselves, leading to hangs

[BorisTheBrave]

Zarr version

3.1.3

Numcodecs version

0.16.3

Python Version

3.11.11

Operating System

Linux

Installation

uv

Description

Accessing a s3 store as described below, calling most operations on the group cause a hang, or other issues.

The issue is the underlying obstore lists '' as an entry of the directory, which causes the recusion in _iter_members_deep to loop.

Steps to reproduce

You'll need your own s3 bucket, with any zarr unconsolidated group in it. The bucket listed below is private.

from obstore.store import from_url
from zarr.storage import ObjectStore
import zarr


samples_path = "s3://dev-swe/adam/"
region = "eu-north-1"
obstore_store = from_url(samples_path, region=region)
store = ObjectStore(obstore_store)


g = zarr.open_group(store, mode="r")
# Hangs in definitely
print(g.tree())
# fails in _tree.py
# members = sorted([x async for x in group.members(max_depth=max_depth)])
print(g.tree(level=1))


# Here's my debugging of the root issue:
l = [x async for x in store.list_dir("/")]
assert '' not in l # fails

For clarity, this fails for me when the bucket path contains a zarr.json with the following contents. Any other directories are irrelevant.

{
  "zarr_format": 3,
  "node_type": "group"
}

Additional output

No response

[d-v-b]

The issue is the underlying obstore lists '' as an entry of the directory, which causes the recusion in _iter_members_deep to loop.

That's problematic! A fix on our side is to sanitize the output of directory listing to only contain valid zarr node names

[BorisTheBrave]

FYI I've worked around with patching:

        _original_list_dir = ObjectStore.list_dir

        def wrap_list_dir(self: ObjectStore, prefix: str) -> AsyncIterator[str]:
            return (
                item async for item in _original_list_dir(self, prefix) if item != ""
            )

        ObjectStore.list_dir = wrap_list_dir

My recommendation is you do something similar.

[d-v-b]

Glad you could fix this. I am not sure that we want to change the behavior of the storage API, because this represents some abstraction leakage -- it might be better to apply this fix in the Group API.

[BorisTheBrave]

Hi, apologies, it looks like I had an object named "/" in my store which was causing this. Deleting it fixed things.