[v3] revisit runtime config

[jhamman]

This issue tracks a evaluation of the v3 runtime config

Context

The v3 branch runtime config currently looks like this:

zarr-python/src/zarr/v3/config.py

Lines 34 to 38 in 76c3450

	@dataclass(frozen=True)
	class RuntimeConfiguration:
	order: Literal["C", "F"] = "C"
	concurrency: Optional[int] = None
	asyncio_loop: Optional[AbstractEventLoop] = None

This is then attached to Array/Group classes

zarr-python/src/zarr/v3/array.py

Lines 51 to 55 in 76c3450

	@dataclass(frozen=True)
	class AsyncArray:
	metadata: ArrayMetadata
	store_path: StorePath
	runtime_configuration: RuntimeConfiguration

A few things are missing here:

1. User experience
   • as a user, I may want to set config settings and forget about them (e.g. order, concurrency)
2. Portability
   • I don't know for sure but I really doubt that putting the AsyncIO loop on the Array class is going to work when it comes to serialization

Improvements

So looking for some ideas for how to manage this better. Two ideas:

1. Xarray style set-options: https://docs.xarray.dev/en/stable/generated/xarray.set_options.html
   • Pros: allows for validation and is typed
   • Cons: a bit bespoke, doesn't support environment variables or a config file option
2. Dask style config - https://donfig.readthedocs.io/en/latest/
   • Pros: very flexible framework, support for environment variables and config files, nested namespaces, etc.
   • Cons: extra dependency (though we could vendor it), no typing or validation

what do we expect to go in the runtime config?

• Order
• Concurrency
• logging settings
• what else?

[jhamman]

I spoke with @maxrjones today about this. Our thought for now was to try using donfig and see how it goes. We can continue to evaluate the dependency vs vendoring and typing/validation as needed.

cc @djhoese

[normanrz]

Additional config options:

• Specify alternate implementations for CodecPipeline (e.g. for a rust-based codec pipeline)
• Specify alternate implementations for codecs (e.g. for GPU-based batch-aware codecs)
• Batch size in the HybridCodecPipeline

[maxrjones]

thanks @normanrz! Joe mentioned you asked about this today. I'm working on getting a minimal PR opened now and should have that submitted within the next couple hours.

[jhamman]

@normanrz - #1855 is now in the v3 branch. Should clear the way to add additional config options as needed.

[jhamman]

Thanks @maxrjones for getting this moving!

[normanrz]

I think this is a great way of dealing with configurations. Thanks!