Proposal: Reframe NetCDF as a reference structural encoding of CF so other structural layers can be defined.

[dblodgett-usgs]

Topic for discussion

Proposal: Reframe NetCDF as a reference structural encoding of CF so other structural layers can be defined.

This proposal follows from #463 and #484. It aims for minimal and incremental modification to enable CF to be supported by structural encodings beyond NetCDF. It prioritizes compatibility with existing NetCDF and Zarr data but also refines CF in a way that cleanly separates it from NetCDF, unblocking the potential for future structural layers.

1. Premise

CF defines constructs (what a coordinate variable, a cell method, a grid mapping means) and relies on a structural data model to hold them (dimensions, variables, attributes, groups). Today the structural model required to conform with CF is netCDF, required by a single sentence in Section 1.3: "The terms in this document that refer to components of a netCDF file are defined in the NUG."

Appendix I describes the netCDF encoding "CF-netCDF" and establishes CF independence from the NetCDF encoding as a design goal (Hassell et al. 2017).

This proposal makes the NetCDF encoding of CF explicit and substitutable. It evolves CF to state the components it requires of any structural layer, keeping the (NetCDF User's Guide) as a reference encoding, and scopes the handful of sections that describe netCDF as such.

This proposal does not change any CF semantics, restructure the document, deprecate the netCDF encoding, or define a Zarr (or other) encoding within CF.

2. Structural layers

• NUG (NetCDF User's Guide) — the reference layer; components, attribute conventions. Nothing changes for CF-netCDF datasets.
• NZ (zarr-conventions/nz) — a structural layer on Zarr v3 based on xarray encoding assumptions. NZ is written to supply the same components the NUG supplies (named shared dimensions, typed arrays incl. scalars, group/array attributes, _FillValue, dimension coordinate arrays). It documents what xarray already writes and is registered under the zarr-conventions framework.
• Designed to admit others on the same terms, e.g. a tabular layer on Parquet / GeoParquet for Discrete Sampling Geometries (ch. 9; cf. @ethanrd in Separating CF semantics from the netCDF encoding #484), or OGC CoverageJSON, whose domain axes and typed ranges are the same components.
• How a structural layer is defined:
  • Its own documentation conventions state how it supports each required component.
  • A dataset declares both in its Conventions declaration (e.g. "NZ-1.0 CF-1.14").

3. The edits to be applied in one PR

Section 1.3 Terminology (+ one sentence in Section 1.1).

Replace the NUG-binding sentence with the list of components CF requires of a structural layer:
1. named, sized dimensions shared within a group
2. typed N-d variables incl. scalars with an ordered dimension list
3. attributes at variable and group/global level
4. a root group (groups optional)
5. coordinate variables (1-D, name = dimension)
6. reserved _FillValue.

Define the NUG as the reference encoding.
Define "most rapidly varying dimension" by declared (CDL) order rather than storage.

Why: This core dependency is CF's only required binding to netCDF. Spelling out the contract with the NUG makes it substitutable. Declared-order definition allows the "(CDL order)" rule in ch. 7, ch. 9 and the conformance doc to go unchanged.

Chapter 2 scoping

Chapter 2, "NetCDF Files and Components" will need to be reframed to emphasize that NetCDF is a reference implementation of the CF constructs.

1. title/preamble become "Components of a CF Dataset" framed using the NUG as the reference implementation.
2. Section 2.1 filename needs an addition that other encodings identify a dataset by their own means (a Zarr hierarchy by its root metadata, for example).
3. Section 2.2 data types incl. strings needs subtle changes indicating what CF requires and that NetCDF is a reference encoding of it that takes its own opinions.
4. Section 2.3 names requires only light framing changes to imply that NetCDF is an example encoding.

Regarding strings, CF already defines the base requirement and implementation options as a "may be" statement: Section 2.2 says an n-dimensional array of strings "may be implemented as" a string variable with n dimensions or a char variable with n+1. Both netCDF forms (NC_STRING, and NC_CHAR with a trailing string-length dimension) and both registered Zarr v3 string types (string, fixed_length_utf32) are arrays of Unicode strings, and they map losslessly in both directions. The edit therefore would state the base requirement and that in netCDF, it is stored in the two existing forms.

Why: These sections describe the netCDF structural encoding of CF as an exemplar. We have verified that xarray's Zarr v3 output for CF-netCDF string variables already lands on registered Zarr types, so no new string concept or deliberation is needed. All other adjustments are reframing to provide clarity that NetCDF is the reference encoding.

Section 2.5.1 Missing data

Restate the missing-data rules in CF rather than deferring to NUG Appendix A:

1. Values equal to _FillValue (a scalar of the same type as the stored data) are missing.
2. Values outside valid_range, or below valid_min or above valid_max, are missing.
3. Values equal to missing_value are missing.
4. When a single missing value suffices, _FillValue is the recommended attribute.
5. Where the structural encoding defines a default fill value for a data type — as the NetCDF library does — values equal to that default are treated as missing when _FillValue is absent.

The CF rules already in this section (missing data permitted in data and auxiliary coordinate variables but not in coordinate variables; missing values interpreted before unpacking; actual_range) are unchanged. The paragraph on the NUG 2.3 to 2.4 change and the description of the NetCDF library's pre-fill behaviour remain as NetCDF-scoped notes.

Why: This section is the heaviest deference to the NUG in the document — CF's definition of "missing" currently lives in a document CF does not version or control, a point @larsbarring raised in #463. @JonathanGregory proposed stating these rules in CF in his #463 summary. Item 5 preserves current CF-netCDF reader behavior while making explicit a property another structural encoding may lack: Zarr's storage fill_value initializes unwritten chunks and carries no masking semantics, so an array without _FillValue has no missing-data indicator there.

Section 2.6 Attributes (incl. Section 2.6.1)

1. Section 2.6.1: Conventions is the root-group attribute by which a dataset declares the conventions it follows. CF already states the list syntax (blank-separated, or comma-separated when a name contains blanks), so the sentence attributing that syntax to NUG Appendix A becomes an informational note.
2. Section 2.6: add one paragraph for structural encodings that do not declare attribute data types. CF's typing rules are written against declared attribute types — _FillValue, missing_value, valid_*, flag_values and flag_masks take the type of their variable; actual_range takes the unpacked type; scale_factor and add_offset must be float or double and determine the unpacked type. Proposal: Where attribute types are not declared, the same-type rules resolve from the variable's data type, and scale_factor and add_offset are taken as double.

Why: Without this paragraph, CF's typed-attribute rules cannot be applied where attributes are JSON values, which is the case for Zarr. The double default is the guidance Section 8.1 already gives for packed data whose attribute types do not conform ("unpacked to double type, in order to minimize the risk of loss of precision"), so existing CF-netCDF datasets don't read differently.

Section 2.7 Groups

1. Restate the section on structural layers, such as NetCDF4, that provide group functionality: a root group, a tree of named child groups, group-level attributes, and variables and dimensions that belong to one group and are identified within it by name. A layer without groups would be treated as a "root-only" case.
2. Replace "the same netCDF dimension ID" with a definition of dimension identity that does not depend on an ID. Name and length can be used instead of ID.
3. The quotation from the NetCDF groups documentation becomes an informational note on how NetCDF scopes dimensions rather than the principle the rules are based on.

The search strategies (absolute path, relative path, proximity, lateral) and the rules for Conventions, external_variables, title and history in non-root groups are defined within CF and do not need to change.

Why: Section 2.7 came in with CF-1.8 and is CF's own name-resolution convention; the dimension-ID clause is its only NetCDF-bound sentence. Stating identity by name and length is what a layer that scopes dimension names per group needs in order to apply the section at all.

Reference wording

1. Section 3.1 already limits CF's dependence on UDUNITS to "the definition of legal units strings" and states that CF does not require the UDUNITS software for conversion. Align Section 1.4 and the bibliography entry with that scoping — the reference is to the UDUNITS unit grammar and database, not to the software package.
2. Section 8.1: strike "At the current time the netCDF interface does not provide for packing data," which describes a library rather than the convention.
3. Conformance document: mirror the Section 2.7 change in its own 2.7 section (the dimension-ID sentence, and "NUG-coordinate variables" becomes "coordinate variables").

Why: These are alignments with scoping CF already states, not changes in meaning.

Not changed

Appendix A's cross-references to NUG Appendix A stay as attribution — CF defines every attribute it requires in its own text (Sections 2.5.1, 2.6.2, 3.1, 3.2 and 8.1), so the citations record provenance rather than deference. Appendix I is unchanged; it already describes the encoding-independent data model this proposal relies on. CDL remains the notation for examples. The unlimited-dimension mentions in Chapter 9 and Appendix H are already conditional on wanting to append and do not need to be edited; the same holds for Section 8.4, which presents the NetCDF library's quantization attributes as one implementation among others. Incidental "netCDF file" wording in Chapters 4, 5, 7 and 9 and Appendix D can become "dataset" as an optional follow-up.

4. Why this path

Parsimonious. Six sections and one PR that do not introduce ideas beyond what Appendix I already introduces, "CF-netCDF", and every CF-netCDF dataset remains valid verbatim — the condition Section 1.2 item 10 sets for any change.

Incremental. Each edit can be checked as a thought experiment against data and software that exist today — netcdf-c, xarray, cf-python, the cf-checker — rather than as a restructure every reader and every training document has to absorb before its value can be judged.

Compatibility-first. The structural layers are written from what implementations already do: the NUG from NetCDF, NZ which documents xarray's Zarr V3. Conformance of existing data against a structural layer defined independent of CF is therefore testable now rather than after a new encoding is designed and adopted.

Progressive. The edits unblock the use cases raised in #484 — ESA's Sentinel archive on Zarr, Copernicus ARCO ERA5, ERDDAP's Parquet output — without limiting the potential to pursue a more significant Part I / Part II restructure, a formal grammar, or other formalization later. Each of those will be easier once the binding between CF and its structural layer is explicit.

Governance-light. CF defines the components it requires. Who publishes an encoding for a given container, and whether that encoding is in CF as an appendix, remains a decision the community can make case by case, as discussed in #463.

5. Proposed next step

I will draft the PR against cf-conventions and open it for discussion, with the Section 1.3 component list and the Section 2.5.1 and 2.6 restatements as the parts most in need of review. I would ask the maintainers of the groups material (Section 2.7), the Discrete Sampling Geometries chapter, and the conformance document to review the respective edits. The target is CF-1.14.

Three questions for this thread:

1. Does the six-component list in Section 1.3 capture everything CF depends on from the NUG? If a chapter relies on a NUG property not in the list, that is a seventh component rather than an argument against the approach.
2. Does the attribute-typing paragraph belong in Section 2.6, with the other attribute rules, or in Section 8.1 with the double guidance?
3. After this PR, should the NetCDF-scoped sentences be collected into an appendix — a small step toward the reference-encoding part of the Separating CF semantics from the netCDF encoding #484 proposal — or stay in place?

LLM Agent Disclosure: Claude Fable 5.1 was used to assist research and outlining this discussion entry. The outline was filled in and edited by @dblodgett-usgs. All LLM generated text has been reviewed and validated.

"Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government."