From 91b4e0f3edcab23d01f9b6ede09ee48864575fbc Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Fri, 10 Feb 2023 03:06:55 +0530 Subject: [PATCH 1/8] Add API reference for V3 Implementation in the docs --- docs/api.rst | 1 + docs/api/v3.rst | 58 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) create mode 100644 docs/api/v3.rst diff --git a/docs/api.rst b/docs/api.rst index 2b6e7ea51..e200dd908 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -13,6 +13,7 @@ API reference api/codecs api/attrs api/sync + api/v3 Indices and tables ------------------ diff --git a/docs/api/v3.rst b/docs/api/v3.rst new file mode 100644 index 000000000..94be29347 --- /dev/null +++ b/docs/api/v3.rst @@ -0,0 +1,58 @@ +V3 Specification Implementation(``zarr._storage.v3``) +===================================================== + +This module contains the implementation of the `Zarr V3 Specification `_. + +Since Zarr Python 2.12 release, this module provides experimental infrastructure for reading and +writing the upcoming V3 spec of the Zarr format. Users wishing to prepare for the migration can set +the environment variable ``ZARR_V3_EXPERIMENTAL_API=1`` to begin experimenting, however data +written with this API should not yet be considered final. + +The new ``zarr._store.v3`` package has the necessary classes and functions for evaluating Zarr V3. +Since the design is not finalised, the classes and functions are not automatically imported into +the regular Zarr namespace. + +Code snippet for creating Zarr V3 arrays:: + + >>>import zarr + >>>z = zarr.create((10000, 10000), + chunks=(100, 100), + dtype='f8', + compressor='default', + path='path-where-you-want-zarr-v3-array', + zarr_version=3) + +Further, you can use `z.info` to see details about the array you just created:: + + >>>z.info + Name : path-where-you-want-zarr-v3-array + Type : zarr.core.Array + Data type : float64 + Shape : (10000, 10000) + Chunk shape : (100, 100) + Order : C + Read-only : False + Compressor : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0) + Store type : zarr._storage.v3.KVStoreV3 + No. bytes : 800000000 (762.9M) + No. bytes stored : 557 + Storage ratio : 1436265.7 + Chunks initialized : 0/10000 + +You can also check ``Store type`` here (which indicates Zarr V3). + +.. module:: zarr._storage.v3 + +.. autoclass:: RmdirV3 +.. autoclass:: KVStoreV3 +.. autoclass:: FSStoreV3 +.. autoclass:: MemoryStoreV3 +.. autoclass:: DirectoryStoreV3 +.. autoclass:: ZipStoreV3 +.. autoclass:: RedisStoreV3 +.. autoclass:: MongoDBStoreV3 +.. autoclass:: DBMStoreV3 +.. autoclass:: LMDBStoreV3 +.. autoclass:: SQLiteStoreV3 +.. autoclass:: LRUStoreCacheV3 +.. autoclass:: ConsolidatedMetadataStoreV3 From 09d86406ebdd7c61c927fd4eac370d3e124c59e5 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Fri, 10 Feb 2023 03:29:35 +0530 Subject: [PATCH 2/8] Minor fix --- docs/api/v3.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/api/v3.rst b/docs/api/v3.rst index 94be29347..595d20f30 100644 --- a/docs/api/v3.rst +++ b/docs/api/v3.rst @@ -14,8 +14,8 @@ the regular Zarr namespace. Code snippet for creating Zarr V3 arrays:: - >>>import zarr - >>>z = zarr.create((10000, 10000), + >>> import zarr + >>> z = zarr.create((10000, 10000), chunks=(100, 100), dtype='f8', compressor='default', @@ -24,7 +24,7 @@ Code snippet for creating Zarr V3 arrays:: Further, you can use `z.info` to see details about the array you just created:: - >>>z.info + >>> z.info Name : path-where-you-want-zarr-v3-array Type : zarr.core.Array Data type : float64 From 23431a4abd8dcb7a5f8f7d872ffdf8dd8c3a6277 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Fri, 10 Feb 2023 03:35:42 +0530 Subject: [PATCH 3/8] Minor fix --- docs/api/v3.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/api/v3.rst b/docs/api/v3.rst index 595d20f30..b741ff3eb 100644 --- a/docs/api/v3.rst +++ b/docs/api/v3.rst @@ -16,11 +16,11 @@ Code snippet for creating Zarr V3 arrays:: >>> import zarr >>> z = zarr.create((10000, 10000), - chunks=(100, 100), - dtype='f8', - compressor='default', - path='path-where-you-want-zarr-v3-array', - zarr_version=3) + >>> chunks=(100, 100), + >>> dtype='f8', + >>> compressor='default', + >>> path='path-where-you-want-zarr-v3-array', + >>> zarr_version=3) Further, you can use `z.info` to see details about the array you just created:: From cd9a75f990e7eb63fe68d909003e85360a066db0 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Fri, 10 Feb 2023 03:51:50 +0530 Subject: [PATCH 4/8] Minor indentation fix --- docs/api/v3.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/api/v3.rst b/docs/api/v3.rst index b741ff3eb..23cd06a8f 100644 --- a/docs/api/v3.rst +++ b/docs/api/v3.rst @@ -16,11 +16,11 @@ Code snippet for creating Zarr V3 arrays:: >>> import zarr >>> z = zarr.create((10000, 10000), - >>> chunks=(100, 100), - >>> dtype='f8', - >>> compressor='default', - >>> path='path-where-you-want-zarr-v3-array', - >>> zarr_version=3) + >>> chunks=(100, 100), + >>> dtype='f8', + >>> compressor='default', + >>> path='path-where-you-want-zarr-v3-array', + >>> zarr_version=3) Further, you can use `z.info` to see details about the array you just created:: From 98a9ad0c2df3e3e9fd3cc8dad03a079883ffbf10 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Wed, 15 Feb 2023 05:12:13 +0530 Subject: [PATCH 5/8] Update docs/api/v3.rst Co-authored-by: Jonathan Striebel --- docs/api/v3.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/api/v3.rst b/docs/api/v3.rst index 23cd06a8f..36f8b4524 100644 --- a/docs/api/v3.rst +++ b/docs/api/v3.rst @@ -3,10 +3,11 @@ V3 Specification Implementation(``zarr._storage.v3``) This module contains the implementation of the `Zarr V3 Specification `_. -Since Zarr Python 2.12 release, this module provides experimental infrastructure for reading and -writing the upcoming V3 spec of the Zarr format. Users wishing to prepare for the migration can set -the environment variable ``ZARR_V3_EXPERIMENTAL_API=1`` to begin experimenting, however data -written with this API should not yet be considered final. +.. warning:: + Since Zarr Python 2.12 release, this module provides experimental infrastructure for reading and + writing the upcoming V3 spec of the Zarr format. Users wishing to prepare for the migration can set + the environment variable ``ZARR_V3_EXPERIMENTAL_API=1`` to begin experimenting, however data + written with this API should be expected to become stale, as the implementation will still change. The new ``zarr._store.v3`` package has the necessary classes and functions for evaluating Zarr V3. Since the design is not finalised, the classes and functions are not automatically imported into From 4c1d151d6708bbe4e3a5bd5d846a83a231ea6213 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Wed, 15 Feb 2023 05:12:58 +0530 Subject: [PATCH 6/8] Update docs/api/v3.rst Co-authored-by: Jonathan Striebel --- docs/api/v3.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/api/v3.rst b/docs/api/v3.rst index 36f8b4524..a951e33d4 100644 --- a/docs/api/v3.rst +++ b/docs/api/v3.rst @@ -57,3 +57,21 @@ You can also check ``Store type`` here (which indicates Zarr V3). .. autoclass:: SQLiteStoreV3 .. autoclass:: LRUStoreCacheV3 .. autoclass:: ConsolidatedMetadataStoreV3 + +In v3 `storage transformers `_ +can be set via ``zarr.create(…, storage_transformers=[…])``. +The experimental sharding storage transformer can be tested by setting +the environment variable ``ZARR_V3_SHARDING=1``. Data written with this flag +enabled should be expected to become stale until +`ZEP 2 `_ is approved +and fully implemented. + +.. module:: zarr._storage.v3_storage_transformers + +.. autoclass:: ShardingStorageTransformer + +The abstract base class for storage transformers is + +.. module:: zarr._storage.store + +.. autoclass:: StorageTransformer From d8a4131d50e0db0831144e83e9c5a6c96c6bed4d Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Fri, 10 Mar 2023 21:10:24 +0530 Subject: [PATCH 7/8] Update docs/api/v3.rst Co-authored-by: Jonathan Striebel --- docs/api/v3.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/api/v3.rst b/docs/api/v3.rst index a951e33d4..c3233003d 100644 --- a/docs/api/v3.rst +++ b/docs/api/v3.rst @@ -75,3 +75,21 @@ The abstract base class for storage transformers is .. module:: zarr._storage.store .. autoclass:: StorageTransformer + +In v3 `storage transformers `_ +can be set via ``zarr.create(…, storage_transformers=[…])``. +The experimental sharding storage transformer can be tested by setting +the environment variable ``ZARR_V3_SHARDING=1``. Data written with this flag +enabled should be expected to become stale until +`ZEP 2 `_ is approved +and fully implemented. + +.. module:: zarr._storage.v3_storage_transformers + +.. autoclass:: ShardingStorageTransformer + +The abstract base class for storage transformers is + +.. module:: zarr._storage.store + +.. autoclass:: StorageTransformer From b17b17a841db2ed8d0b9bf82842c72cdc53e1cc4 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Fri, 10 Mar 2023 23:08:37 +0530 Subject: [PATCH 8/8] Fix broken links --- docs/api/v3.rst | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) diff --git a/docs/api/v3.rst b/docs/api/v3.rst index c3233003d..7665b2ddd 100644 --- a/docs/api/v3.rst +++ b/docs/api/v3.rst @@ -1,7 +1,7 @@ V3 Specification Implementation(``zarr._storage.v3``) ===================================================== -This module contains the implementation of the `Zarr V3 Specification `_. +This module contains the implementation of the `Zarr V3 Specification `_. .. warning:: Since Zarr Python 2.12 release, this module provides experimental infrastructure for reading and @@ -58,25 +58,7 @@ You can also check ``Store type`` here (which indicates Zarr V3). .. autoclass:: LRUStoreCacheV3 .. autoclass:: ConsolidatedMetadataStoreV3 -In v3 `storage transformers `_ -can be set via ``zarr.create(…, storage_transformers=[…])``. -The experimental sharding storage transformer can be tested by setting -the environment variable ``ZARR_V3_SHARDING=1``. Data written with this flag -enabled should be expected to become stale until -`ZEP 2 `_ is approved -and fully implemented. - -.. module:: zarr._storage.v3_storage_transformers - -.. autoclass:: ShardingStorageTransformer - -The abstract base class for storage transformers is - -.. module:: zarr._storage.store - -.. autoclass:: StorageTransformer - -In v3 `storage transformers `_ +In v3 `storage transformers `_ can be set via ``zarr.create(…, storage_transformers=[…])``. The experimental sharding storage transformer can be tested by setting the environment variable ``ZARR_V3_SHARDING=1``. Data written with this flag