Zarr driver #3896

rouault · 2021-05-28T13:57:18Z

Another approach at creating a Zarr driver than the one of PR #3411 (see analysis done in #3411 (comment))

Scope is:

read/creation/update of Zarr datasets
using nominally the multidimensional API
and also exposed with the classic 2D API (slicing needed for > 2D datasets)
support hierarchical organization in groups
support attributes, including a dedicated attribute to encode the CRS, and the XArray _ARRAY_DIMENSIONS attribute to encode the netCDF-like dimension concept
support consolidated metadata
support reading/writing datasets on cloud storage, .zip (but that comes for free using the VSI I/O layer)
support Zarr V2 and in-progress V3 specifications
support compression methods: zlib, gzip, zstd, lzma, lz4 (new optional build dependency), blosc (new optional build dependency), with an API allowing to register additional compressors
support most data types: bool, [u]int[8/16/32/64], float[32/64], complexfloat[32/64], strings, compound data types (limited to what GDALExtendedDataType supports)
limited support for filters with delta filter only as builtin, with an API allowing to register additional filters

Tasklist

rouault · 2021-07-01T16:46:53Z

This is now feature ready.

Zarr driver RFC

Who asked for the driver and why?

Zarr is an emerging format, natively capable for multidimensional raster and cloud friendly. A number of parties have expressed a desire to see it handled in GDAL.
The Zarr V2 format has just been submitted as a potential OGC community standard: https://www.ogc.org/pressroom/pressreleases/3275

This work takes a different approach than the one proposed in #3411 for the reasons mentioned in #3411 (comment)

It should also be noted that libnetcdf 4.8.0 has initial support for ZarrV2, but based on our testing, it lacks maturity, and having a dedicated driver in GDAL gives more control, particularly regarding the I/O layer where any VSI virtual file system can be used, or the possibility to easily add CRS support as we have done.

Read / Write / ReadWrite ?

Read, creation and update

Raster / Vector / Raster&Vector?

Raster, implementing classic 2D and multidimensional API

Driver name info

Short name : Zarr
Long name : Zarr
Description: Zarr

Should this be compiled in by default, optional build, or plugin?

Built-in by default, can be disabled in autoconf builds using standard mechanism (--disable-driver-zarr)

Does this driver require other drivers be built?

No, but it strongly benefits from building GDAL against liblz4, liblzma, libzstd and libblosc to benefit from all compression methods

Is there a standard(s) that the driver is working to implement?

There's a specification for ZarrV2 at https://zarr.readthedocs.io/en/stable/spec/v2.html, which is the one mentioned above as being submitted to OGC.
The driver also implements the in-progress Zarr V3 spec: https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/protocol/core/v3.0.html

How feature complete will the driver be?

It implements most of the Zarr V2 specification, except:

datetime64 (“M”) and timedelta64 (“m”) data types aren't supported (we could likely implement them as double if needed)
compound data types that use members of type array, since our multidimensional array abstraction doesn't support that
a few compression methods of NumCodecs aren't supported currently: ZFPY, BZ2. It should also be noted that this PR offers a C API to register compression methods that aren't implemented by GDAL, which can provide extensibility for other methods in needed.
only the Delta filter is supported. Similarly for compressors, this PR offers a C API to register filter methods that aren't implemented by GDAL, which can provide extensibility for other methods in needed.

Driver capabilities.

Supports: Raster
Supports: Multidimensional raster
Supports: Subdatasets
Supports: Open() - Open existing dataset.
Supports: Create() - Create writable dataset.
Supports: CreateMultiDimensional() - Create multidimensional dataset.
Supports: Virtual IO - eg. /vsimem/
Creation Datatypes: Byte Int16 UInt16 Int32 UInt32 Float32 Float64 CFloat32 CFloat64

Support for the following features :

DMD_MIMETYPE:

None

DMD_EXTENSIONS:

None

DMD_CONNECTION_PREFIX

None

GDAL_DMD_OPENOPTIONLIST

<OpenOptionList>
  <Option name="USE_ZMETADATA" type="boolean" description="Whether to use consolidated metadata from .zmetadata" default="YES" />
</OpenOptionList>

GDAL_DMD_SUBDATASETS: Yes
GDAL_DCAP_OPEN : Yes
DCAP_CREATE : Yes
GDAL_DCAP_VIRTUALIO: Yes

What external libraries does this driver depend on?

Zlib (build requirement of GDAL)
libdeflate optionaly (for faster Zlib compression/decompression)
liblzma optionaly
liblz4 optionaly
libblosc-c optionaly. Strongly recommended however, as this is the default compression method used by the Python Zarr implementation when creating a Zarr array
libzstd optionaly

Are there version requirements for the libraries?

The versions provided by Ubuntu 20.04 are sufficient to get the driver building and working.

What licenses do the libraries have?

Does the driver require a binary SDK?

No

Any external services required for the driver?

No

Any external service providers required for the driver?

No

Testing plan

This driver has an autotest script in autotest/gdrivers folder

Maintenance plan?

Mostly have a look on where Zarr V3 goes on. Current status in documentation for it is indicated as experimental

Who / what org is responsible for upkeep on this new driver?

Even Rouault

Where to report bugs?

GDAL gitub issue tracker

Draft help text

see gdal/doc/source/drivers/raster/zarr.rst in PR for details

What are the conditions for removing the driver?

If nobody longer interested in keeping it in a buildable and working state

Does the driver already have a draft implementation?

Yes

Who implemented it? Or who will implement this driver? Does an implementer(s) need to be found?

Even Rouault

Is there funding needed?

No, funding of the work has already been provided

Who will review the code / PR?

TBD

rouault · 2021-07-05T13:09:08Z

Is anyone interested in giving this some review ? I know this is a bit overwhelming. I could offer some funding if that might help and being practical to setup.

tbonfort · 2021-07-05T13:20:59Z

Is anyone interested in giving this some review ? I know this is a bit overwhelming. I could offer some funding if that might help and being practical to setup.

cc @thomascoquet ?

sgillies · 2021-07-05T15:12:06Z

~~@rouault I've been using zarr (the python package) at work and have some data for testing. I'll try it out later today.~~ I won't be able to provide a serious PR review after all.

…ation for libdeflate code path

…r now)

rouault · 2021-07-13T14:53:53Z

I'll merge this PR next week if there is no objection or further comments

SpaceIm · 2022-01-04T21:20:11Z

gdal/configure.ac

+AC_ARG_WITH(blosc,[  --with-blosc[=ARG]       Include blosc support (ARG=yes/no/installation_prefix)],,)
+
+if test "$with_blosc" = "" -o "$with_blosc" = "yes" ; then
+  AC_CHECK_LIB(blosc,blosc_cbuffer_validate,HAVE_BLOSC=yes,HAVE_BLOSC=no,)
+
+  if test "$HAVE_BLOSC" = "yes" ; then
+    LIBS="-lblosc $LIBS"
+  else
+    if test "$with_blosc" = "yes" ; then
+      AC_MSG_ERROR([libblosc not found])
+    else
+      echo "libblosc not found - BLOSC support disabled"
+    fi
+  fi
+elif test "$with_blosc" != "" -a "$with_blosc" != "no"; then
+
+  AC_CHECK_LIB(blosc,blosc_cbuffer_validate,HAVE_BLOSC=yes,HAVE_BLOSC=no,-L$with_blosc/lib)
+
+  if test "$HAVE_BLOSC" = "yes" -a -f "$with_blosc/include/blosc.h" ; then
+    LIBS="-L$with_blosc/lib -lblosc $LIBS"
+    EXTRA_INCLUDES="-I$with_blosc/include $EXTRA_INCLUDES"
+  else
+    AC_MSG_ERROR([libblosc not found])
+  fi
+
+else
+    HAVE_BLOSC=no
+fi
+
+AC_SUBST(HAVE_BLOSC,$HAVE_BLOSC)


Too fragile, c-blosc may depend on lz4, snappy, zlib and zstd. A naive AC_CHECK_LIB may break due to missing dependencies in LIBS. c-blosc provides a pkg-config file: blosc.pc

Too fragile, c-blosc may depend on lz4, snappy, zlib and zstd. A naive AC_CHECK_LIB may break due to missing dependencies in LIBS. c-blosc provides a pkg-config file: blosc.pc

too fragile for static builds of c-blosc ? should work fine hopefully for dynamic builds of c-blosc. Anyway autoconf builds will be soon deprecated, so any improvements if any needed should go to the CMake builds

too fragile for static builds of c-blosc ?

yes (in conan-center, by default all libs are built against static dependencies so we quickly see these kind of issues).

Anyway autoconf builds will be soon deprecated, so any improvements if any needed should go to the CMake builds

Great ! I'll check that while packaging the first release using CMakeLists.

Great ! I'll check that while packaging the first release using CMakeLists.

early testing of CMake builds of master would be appreciated

rouault added this to the 3.4.0 milestone May 28, 2021

rouault marked this pull request as draft May 28, 2021 13:57

rouault mentioned this pull request May 28, 2021

[WIP] Add Zarr driver #3411

Closed

6 tasks

kylebarron mentioned this pull request Jun 4, 2021

Zarr loader visgl/loaders.gl#1297

Open

rouault force-pushed the zarr branch from 62593c5 to 9137c51 Compare June 11, 2021 10:31

rouault force-pushed the zarr branch 2 times, most recently from 62a8e59 to 9f325dd Compare June 18, 2021 12:55

rouault force-pushed the zarr branch from 9f325dd to 57db382 Compare June 25, 2021 12:58

rouault changed the title ~~[WIP] Zarr driver~~ Zarr driver Jul 1, 2021

rouault marked this pull request as ready for review July 1, 2021 16:22

rouault mentioned this pull request Jul 3, 2021

Use singular for "spec/codec" zarr-developers/zarr-specs#119

Merged

rouault force-pushed the zarr branch from e98ddfa to 595860b Compare July 10, 2021 09:33

rouault added 14 commits July 12, 2021 12:09

Zarr driver: initial piece of multidimensional read side

690bbb0

Zarr: test invalid values in .zarray

f90ea6f

Zarr: test negative array steps

3a9a5f1

Zarr: do not restrict some variables to size_t

b89a50a

Zarr: IRead() performance improvements

e6d357b

Add cpl_compressor.h to register compression/decompression functions

199c937

Add support for Blosc compressor

7f5434c

Docker alpine-normal: add cblosc

7dd10a1

Zarr: add support for compressor

6577877

CPLZLibDeflate(): honour nLevel parameter, and improve auto-mem alloc…

952fb11

…ation for libdeflate code path

Add ZLib compressor

01b3560

Add LZMA compressor

84e1ac5

Add ZSTD compressor

c473d18

Add LZ4 compressor

4e3ebca

rouault added 25 commits July 12, 2021 12:11

Zarr: write side: add metadata support for compression method

1b55d99

Zarr: add write support for array attributes

710438d

Zarr: add write support for CRS

37dc8d6

Zarr: add write support for XArray's _ARRAY_DIMENSIONS

2b93b07

Zarr: implement array value writing

7622fe2

netCDF: move CF constants to netcdf_cf_constants.h header

dc91257

Zarr: add support for unit, scale, offset using CF conventions

ae378de

Zarr: implement GetCoordinateVariables()

9fa7377

Zarr: implement classic raster Create() API

d2fcad9

Zarr: check for too large block size

c6b13bd

Zarr: add ZarrV3 group creation

f598f91

Zarr: add ZarrV3 array creation

c56eb1e

Doc building: add a supports_multidimensional directive

6291288

Zarr: add documentation page

9ea53b4

Zarr: set TYPESIZE with Blosc compression

b3a28a7

/vsicurl/: fix to be able to read /vsis3/zarr-demo/store/spam

b2e4b40

Add CPLJSONObject::DeleteNoSplitName()

6356d9b

Zarr: add creation/update support for consolidated metadata .zmetadata

99962ae

Compressor API: add support for Delta filter restricted to dtype=astype

e3b5f77

Zarr: add read/update support for filters (limited to delta filter fo…

b36c2d8

…r now)

Zarr: add read support for NCZarr way of setting fill_value

0f006be

Zarr: add creation option FILTER (Zarr V2 only)

80ec6da

Zarr: turn on -Weffc++ -Wold-style-cast

faf0616

Zarr V3: use singular form for codec URL

dc63e21

zarr/GNUmakefile: use -iquote

6c04756

rouault force-pushed the zarr branch from 595860b to 6c04756 Compare July 12, 2021 10:12

rouault merged commit e85de9a into OSGeo:master Jul 19, 2021

SpaceIm reviewed Jan 4, 2022

View reviewed changes

rabernat mentioned this pull request Apr 6, 2022

Writing GDAL ZARR _CRS attribute not possible pydata/xarray#6448

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zarr driver #3896

Zarr driver #3896

rouault commented May 28, 2021 •

edited

Loading

rouault commented Jul 1, 2021 •

edited

Loading

rouault commented Jul 5, 2021

tbonfort commented Jul 5, 2021

sgillies commented Jul 5, 2021 •

edited

Loading

rouault commented Jul 13, 2021

SpaceIm Jan 4, 2022

rouault Jan 4, 2022

SpaceIm Jan 4, 2022

rouault Jan 4, 2022

Zarr driver #3896

Zarr driver #3896

Conversation

rouault commented May 28, 2021 • edited Loading

Tasklist

rouault commented Jul 1, 2021 • edited Loading

Zarr driver RFC

Who asked for the driver and why?

Read / Write / ReadWrite ?

Raster / Vector / Raster&Vector?

Driver name info

Should this be compiled in by default, optional build, or plugin?

Does this driver require other drivers be built?

Is there a standard(s) that the driver is working to implement?

How feature complete will the driver be?

Driver capabilities.

Support for the following features :

What external libraries does this driver depend on?

Are there version requirements for the libraries?

What licenses do the libraries have?

Does the driver require a binary SDK?

Any external services required for the driver?

Any external service providers required for the driver?

Testing plan

Maintenance plan?

Who / what org is responsible for upkeep on this new driver?

Where to report bugs?

Draft help text

What are the conditions for removing the driver?

Does the driver already have a draft implementation?

Who implemented it? Or who will implement this driver? Does an implementer(s) need to be found?

Is there funding needed?

Who will review the code / PR?

rouault commented Jul 5, 2021

tbonfort commented Jul 5, 2021

sgillies commented Jul 5, 2021 • edited Loading

rouault commented Jul 13, 2021

SpaceIm Jan 4, 2022

Choose a reason for hiding this comment

rouault Jan 4, 2022

Choose a reason for hiding this comment

SpaceIm Jan 4, 2022

Choose a reason for hiding this comment

rouault Jan 4, 2022

Choose a reason for hiding this comment

rouault commented May 28, 2021 •

edited

Loading

rouault commented Jul 1, 2021 •

edited

Loading

sgillies commented Jul 5, 2021 •

edited

Loading