Data file formats

[ali-ramadhan]

For Oceananigans.jl we settled on supporting NetCDF output early as it's the most ubiquitous and familiar data format (see CliMA/Oceananigans.jl#31).

We've been using NetCDF.jl but it seems like an inactive package so we're thinking of switching to the more active and thoroughly tested NCDatasets.jl. NetCDF.jl has some missing features and seems slow. Output writing and compression was so slow compared to time stepping on the GPU that we ended up introducing asynchronous output writing (CliMA/Oceananigans.jl#137).

Anyways, just curious if NetCDF output is something you guys have looked at yet. If not, might be worth it to work on a common solution with features we all need (fast NetCDF output, option for asynchronous GPU output, standardized names, etc.)?

[ali-ramadhan]

Another option, not sure if it is good as I have never used it, is a meta package like https://www.olcf.ornl.gov/center-projects/adios/ which supports parallel io and is somewhat file format agnostic.
_{View in Slack}

[ali-ramadhan]

<@UCSM0C54P> re: your NetCDF issue, why not HDF5?
_{View in Slack}

[ali-ramadhan]

if you do need NetCDF compatibility (e.g. for exporting to another program), it might be possible to enforce the necessary restrictions at the Julia level
_{View in Slack}

[ali-ramadhan]

My short exposure to HDF5 has all been positive
_{View in Slack}

[ali-ramadhan]

The main complaints I have heard about HDF5 have been when people try to update or mutate data, and corrupt their dataset
_{View in Slack}

[ali-ramadhan]

I think if we want to encourage broad use netCDF is widely ingrained in climate communities. Some ability to produce netCDF for downstream use might be wise. Possibly that is not sufficient for full DG info though? What is the thinking on street re targeting netCDF/CF style and engaging a bit with Pangeo community ( https://pangeo.io ). Pangeo has been leaning on netCDF to transparently include a zarr backend option ( https://zarr.readthedocs.io/en/stable/ ), which plays well with cloud object stores out of the box.
_{View in Slack}

[rabernat]

I found this repo thanks to a tweet. Kudos to everyone for following modern software development best practices in terms of CI, docs, etc! This bodes very well for the CLIMA project!

It's correct that in Pangeo we are exploring storage options beyond netCDF. The primary reason is that HDF5, the library underlying netCDF4, does not currently play very well with object storage (e.g. Amazon S3, Google Cloud Storage, etc; see blog post about why). We are very excited about using the cloud as a shared environment for the analysis of big climate datasets, so we have been experimenting with other options.

Zarr is the thing we are most excited about. Zarr was originally developed for genomic analysis by researchers who were frustrated by the poor parallel write support in HDF5. It uses an open spec and is quite simple, yet it performs very well on single-workstation, HPC, and cloud environments. Here is a little talk I gave about Zarr at AGU. TileDB is a similar product, the difference being that it is backed by a company and, while open source, doesn't have an open spec. After the experience of HDF-group going commercial, I think the community is now a little wary about that.

Zarr has been adopted in a number of different fields and implemented in a number of different languages (python, C++ and Java). @meggart of MPI has a working julia implementation. (See also https://github.com/zarr-developers/zarr/issues/284.)

Although Zarr is working great for us, it is still experimental in the sense that it is not yet a community standard. The netCDF data model and the CF conventions themselves are excellent. They are the result of decades of tedious yet important work to define common standards for exchanging data. Going forward, I hope we can preserve these while experimenting with new underlying storage containers like zarr. netCDF != HDF. In this vein, it's been great to learn that the Unidata netCDF group themselves are working on creating a zarr-based backend for the netCDF core C library. I would expect them to release something within one year.

In the meantime, it would be great to see the CLIMA project experiment with zarr. The first step would be to check in at https://github.com/meggart/ZarrNative.jl to understand better where the julia library stands.

[simonbyrne]

Another thing we might want to look at, especially if we're looking to use HDF5: https://www.openpmd.org

[simonbyrne]

So in case anyone is interested, here is brief summary of different data formats.

HDF

HDF (Hierarchical Data Format) is a set of file formats and libraries, maintained by the HDF Group, which represent a hierarchies of groups and datasets (like a file system, stored in a file)

• HDF4 is an old format, with many hard-coded limitations (limited types, 32-bit integer indexing).
• HDF5 is newer, and more general, supporting different data layouts, compression, etc. It is extremely complex, but an open standard

The hdf5 library can optionally be built against MPI, for parallel I/O.

Given its standard but flexible nature, many other software packages have used HDF5 as the basis for their own format:

• Matlab v7.3 .mat files
• Julia .jld and .jld2 formats
• NetCDF-4 (see below)

Packages

• HDF5.jl wraps the hdf5 library. It is an old package, and appears to be intermittently maintained.
  • There is an open PR for MPI support.
• JLD.jl is built on top of HDF5.jl, for convenient saving/loading of arbitrary Julia objects.
• JLD2.jl is pure-Julia implementation of a subset of HDF5, and supports similarly saving/loading of Julia objects.

NetCDF

NetCDF is developed by Unidata at UCAR is the name for any one of a set of file formats which can be read/written by the netcdf library. These are:

1. NetCDF Classic (PnetCDF calls this CDF-1)
2. NetCDF 64-bit offset, which is a a variant of 1 with 64-bit offset for large arrays (PnetCDF calls this CDF-2)
3. PnetCDF (see below) supports an additional variant of 2 with support for 64-bit integer values, which it calls CDF-5 (this does not appear to be supported by the netcdf library).

Formats 1-3 are often labelled NetCDF-3

4. NetCDF-4, which is a subset of HDF5. It supports additional features such as custom datatypes.
5. NetCDF-4 Classic, which is a subset of NetCDF-4 limited to the the same features as the NetCDF Classic format (e.g. no custom datatypes), so that datasets can be losslessly converted between the two formats.
6. It can also read (but not write) HDF4 files.

Parallel I/O is supported by:

• Formats 1,2 and 3: the PnetCDF package, developed by ANL, which is a completely separate library built on top of MPI. It can either be used standalone, or netcdf can be built with PnetCDF support (--enable-pnetcdf) to allow it to use the same interface as formats 4 & 5.
• Formats 4 and 5: directly if the underlying HDF5 library was built with MPI support.

Packages

There are 2 Julia packages which wrap the netcdf library:

• NetCDF.jl is an older package and had been inactive, though recently appears to have had some attention
• NCDatasets.jl is a newer package, with a slightly different interface: this is what we're currently using, and by default it uses the NetCDF-4 format.

[kpamnany]

From here:

NETCDF3_CLASSIC was the original netcdf binary format, and was limited to file sizes less than 2 Gb. NETCDF3_64BIT_OFFSET was introduced in version 3.6.0 of the library, and extended the original binary format to allow for file sizes greater than 2 Gb. NETCDF3_64BIT_DATA is a new format that requires version 4.4.0 of the C library - it extends the NETCDF3_64BIT_OFFSET binary format to allow for unsigned/64 bit integer data types and 64-bit dimension sizes. NETCDF3_64BIT is an alias for NETCDF3_64BIT_OFFSET. NETCDF4_CLASSIC files use the version 4 disk format (HDF5), but omits features not found in the version 3 API. They can be read by netCDF 3 clients only if they have been relinked against the netCDF 4 library. They can also be read by HDF5 clients. NETCDF4 files use the version 4 disk format (HDF5) and use the new features of the version 4 API.