Unable to create zarr store in unit test using moto

[christine-e-smit]

Zarr version

2.18.7

Numcodecs version

0.15.1

Python Version

3.14.2

Operating System

Mac

Installation

using conda

Description

I'm trying to update a bunch of older unit tests to use moto>5. As you may recall, there were major breaking changes to the moto library going from 4.x to 5.x. Unfortunately, I'm running into basic issues getting zarr to work with moto. I think I remember you saying that you use moto yourselves for unit testing, so I'm not sure where I'm going wrong. All the tests I've updated are getting the same error:

  File "/blah/blah/blah/lib/python3.12/site-packages/aiobotocore/endpoint.py", line 63, in convert_to_response_dict
    response_dict['body'] = await http_response.content
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object bytes can't be used in 'await' expression

Steps to reproduce

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "boto3==1.40.70",
#     "moto==5.1.19",
#     "s3fs==2026.1.0",
#     "zarr==2.18.7",
# ]
# ///
#

import s3fs
from moto import mock_aws
import boto3
import zarr

ma = mock_aws()
ma.start()
bucket_name = "test_bucket"
s3_resource = boto3.resource("s3")
s3_resource.create_bucket(
    Bucket=bucket_name,
    CreateBucketConfiguration={"LocationConstraint": "us-west-2"},
)

s3 = s3fs.S3FileSystem()
store = s3fs.S3Map(root=f"s3://{bucket_name}/my.zarr", s3=s3, check=False)
zarr.open(store, "w")
ma.stop()

Additional output

No response

[d-v-b]

here's the test code to look at for reference:

zarr-python/tests/test_store/test_fsspec.py

Line 60 in 1c05c1a

def s3_base() -> Generator[None, None, None]:

[d-v-b]

ah nevermind, this is about zarr python 2.x. I don't really know what to recommend here in that case

[christine-e-smit]

Hmmm. I've just tried directly starting the ThreadedMotoServer similar to what you do in your test and I'm still getting the same error. Maybe I'll have to go back to moto 4.x until we manage to finish transitioning to zarr v3.

[christine-e-smit]

Okay, there's more going on here. Using mock_aws directly doesn't work for zarr v3 either. I get the same error with this script:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "boto3==1.40.70",
#     "moto==5.1.19",
#     "s3fs==2026.1.0",
#     "zarr>=3",
#     "flask",
#     "flask_cors",
# ]
# ///
#

import s3fs
from moto import mock_aws
import moto.moto_server.threaded_moto_server as moto_server
import boto3
import zarr

ma = mock_aws()
ma.start()
bucket_name = "test_bucket"
s3_resource = boto3.resource("s3")
s3_resource.create_bucket(
    Bucket=bucket_name,
    CreateBucketConfiguration={"LocationConstraint": "us-west-2"},
)

s3 = s3fs.S3FileSystem(asynchronous=True)
store = zarr.storage.FsspecStore(s3, path=f"{bucket_name}/my.zarr")
zarr.open(store, mode="w")
ma.stop()

[christine-e-smit]

Hang on. I spoke too soon. I wasn't starting the server correctly following your example. I can get it to work with zarr v.2:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "boto3==1.40.70",
#     "moto==5.1.19",
#     "s3fs==2026.1.0",
#     "zarr==2.18.7",
# ]
# ///
#

import s3fs
import moto.moto_server.threaded_moto_server as moto_server
import os
import boto3
import zarr

port = 9020
ip = "127.0.0.01"
server = moto_server.ThreadedMotoServer(ip_address=ip, port=port)
server.start()
os.environ["AWS_ENDPOINT_URL"] = f"http://{ip}:{port}"
bucket_name = "test_bucket"
s3_resource = boto3.resource("s3")
s3_resource.create_bucket(
    Bucket=bucket_name,
    CreateBucketConfiguration={"LocationConstraint": "us-west-2"},
)

s3 = s3fs.S3FileSystem()
store = s3fs.S3Map(f"s3://{bucket_name}/my.zarr", s3=s3, check=False)
zarr.open(store, "w")
server.stop()

It also works for v3:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "boto3==1.40.70",
#     "moto==5.1.19",
#     "s3fs==2026.1.0",
#     "zarr>=3",
#     "flask",
#     "flask_cors",
# ]
# ///
#

import s3fs
import moto.moto_server.threaded_moto_server as moto_server
import os
import boto3
import zarr

port = 9020
ip = "127.0.0.01"
server = moto_server.ThreadedMotoServer(ip_address=ip, port=port)
server.start()
os.environ["AWS_ENDPOINT_URL"] = f"http://{ip}:{port}"
bucket_name = "test_bucket"
s3_resource = boto3.resource("s3")
s3_resource.create_bucket(
    Bucket=bucket_name,
    CreateBucketConfiguration={"LocationConstraint": "us-west-2"},
)

s3 = s3fs.S3FileSystem(asynchronous=True)
store = zarr.storage.FsspecStore(s3, path=f"{bucket_name}/my.zarr")
zarr.open(store, mode="w")
server.stop()

I do get an issue about an unclosed session:

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x1101a88c0>

But I assume that's because the zarr store wasn't closed before the server was stopped. Thank you. I think this solves my issue.