Separating CF semantics from the netCDF encoding

[pvanlaake]

Topic for discussion

1. Introduction

Over the past months, two discussions on this forum have explored how CF relates to storage formats other than netCDF: #461 on CF and Zarr, and #463 on moving CF away from its tight coupling with netCDF. A recurring theme in both is that the CF conventions document currently mixes two different things: what CF metadata means or tries to achieve (the semantics), and how that metadata is written into a netCDF file (the encoding).

At the 2026 CF Workshop a breakout session dedicated itself to this topic, with participants from across the CF community. This discussion reports on that session and proposes a concrete way forward. What follows is largely a summary of the discussion at the breakout session - all participants in that session are gratefully acknowledged.

The starting point is the purpose of CF itself, as stated in section 1.1 of the conventions:

The purpose of the CF conventions is to require conforming datasets to contain sufficient metadata that they are self-describing in the sense that each variable in the file has an associated description of what it represents, including physical units if appropriate, and that each value can be located in space (relative to earth-based coordinates) and time.

The same section goes on to note that although CF is specifically a netCDF standard, most of its ideas are of wider application, and that the metadata objects could be contained in file formats other than netCDF. Nothing in the stated purpose depends on netCDF. The proposal here is to make that explicit in the structure of the conventions.

2. Why do this?

Enable CF in other containers. CF metadata is already being written to Zarr, both through the netCDF API (NCZarr) and using GeoZarr conventions (as was demonstrated at the workshop). Other containers may follow. Today, anyone wishing to use CF outside of netCDF has to reverse-engineer which parts of the conventions are essential semantics and which are artefacts of the netCDF encoding. Making that separation once, authoritatively, within CF, gives every container a common semantic target and makes the result verifiable against the conventions rather than against a particular implementation. It also lets each container encode CF in a way that suits its own design, while remaining semantically equivalent to CF-netCDF. It was observed at the workshop that some current CF mechanisms exist to work around limitations of netCDF; for example the space-separated lists of variable names in attributes such as coordinates and ancillary_variables, which software must parse and then check for existence. Containers with native support for references between objects could express the same semantics more directly.

Write the semantics once, in a machine-readable form. If the semantics are described in a structured, machine-readable format, then multiple materials can be generated from a single source: the conventions text itself, diagrams of the data model, tutorial and example material for each encoding, validators and compliance checkers, and so on. This reduces the risk of these materials drifting apart, and lowers the cost of maintaining them. At the workshop it was noted that such a single source would also be a sound basis for tools built on LLMs: they produce poor results unless properly instructed, and a precise, machine-readable description of CF is exactly the kind of baseline information they need. Any such generated materials would still need human review and testing against known-correct results, and misunderstandings found in that way can feed back into improving the text.

It is worth stating what this is not. It is not a proposal to change the CF semantics, to add new conventions, or to deprecate the netCDF encoding. CF-netCDF datasets written today will be exactly as valid in the new structure as they are now.

3. What will this effort look like?

The work is partly done already. The CF data model (Appendix I) identifies the constructs of CF and their relationships independently of netCDF (although there are references to typical netCDF objects such as a grid_mapping variable whose semantics (describe the CRS of a data variable) may take on a different form in other encodings). What it does not cover is the full semantic content of the conventions text: the meaning of individual attributes, the rules and constraints, the controlled vocabularies, the guidance on interpretation. The data model and the semantics together describe a dataset and its variables; neither is complete on its own.

The proposed effort is:

1. Go through the conventions text, section by section, and extract the semantics, i.e. everything that remains true regardless of the container. Similarly for the CF data model, revise/restructure to focus on the logical model (i.e. the relationship between data variables and associated constructs) rather than netCDF-specific materialisations of the logic (e.g. the grid mapping variable).
2. Write these into a new document which could become Part I: CF semantics. It should build on, and be consistent with, the revised CF data model.
3. The existing conventions text, stripped of what has moved to Part I, might then become Part II: Reference encoding, describing how the semantics are encoded using the netCDF API.

Encodings for other containers could later be added as additional parts, subject to whatever process the community agrees for that. There may also be encodings managed by third parties, not under the management and review of this community, and thus not part of the conventions document.

Several points raised at the workshop bear on the shape of the result:

• Don't start with netCDF. The document should open with the semantics; the netCDF encoding should come after. Moving the data model towards the front of the document was suggested as one step in that direction.
• One specification. There should be a single CF specification, not one per format. The semantics are defined once; each encoding shows how they are realised, with its own examples. One suggestion was to compile multiple "flavours" of the document from a common source, so that a reader interested in a particular encoding sees only the material relevant to it (e.g. a 2D-only container would not need the sections on features it cannot represent), in both HTML and print form.
• Semantics first, grammar later. A formal grammar for CF was felt to be valuable but hard to do well. The room generally agreed to focus on the semantics first and leave a formal grammar for later.
• Examples. Most readers learn CF from the examples, which are currently all in CDL. A semantics section without concrete examples risks being harder to understand, not easier. How to present examples in Part I needs careful thought.
• Terminology. New terms may confuse existing users. Where new terminology is unavoidable it should be introduced sparingly and defined clearly.
• Prior art. OGC practice distinguishes between a conceptual (UML) model and its properties, and OGC has an older mapping from netCDF/CF to its coverage model. OGC standards documents were also mentioned as a good example of formatting. These are worth looking at before settling on a structure.

The structure of the new document is deliberately left open at this stage. Agreeing on it is one of the first things this discussion should do.

4. Process

Who are the users? A clear message from the workshop was that the users of the documentation should be identified before the conventions are restructured, and that different audiences may need different documents. Software developers, specification authors, data managers, user-facing support staff and scientists all interact with the conventions differently. One view was that the semantics are primarily for the more sophisticated users (the CF community, software developers, standards bodies), while most scientists would reach CF through the tools they use. Another was that user-facing support staff are a major audience in their own right, and that the core documentation should remain approachable enough that it can be recommended to scientific communities without extensive additional explanation. It was also suggested that the resulting structure should be checked through usability testing, so that we do not end up with an architecture that is elegant but hard to navigate.

Transition cost and governance. Many users have learned to navigate the current document, and organisations have built training material and guidance around it. Any restructuring must show a clear benefit over the cost to absorb the new structure. Opening CF to more containers may also bring more stakeholders and competing requirements, which adds governance complexity; the CF Committee will need to manage the number of encodings taken on. On the other hand, it was also argued that broadening CF to more containers would grow the community, which would help carry the extra workload. A clean boundary between semantic and encoding concerns is what makes that manageable.

Proposed way of working. Following the usual CF practice:

• Volunteers (preferably in small groups) take on one or more sections of the conventions and extract the semantics from them.
• Progress and questions are discussed here, and/or in a small working group that meets regularly and reports back to this discussion.
• Experts on specific parts of the conventions (e.g. DSG, grid mappings, UGRID, cell methods) are explicitly invited to contribute to or review the extraction of their sections.

Timeline. The target is to present the CF semantics (Part I) at the 2027 CF Workshop. A possible sequence:

• By end of 2026: agree on the user groups, the target structure of Part I, and the format for the machine-readable semantics; set up working group(s).
• First half of 2027: section-by-section extraction.
• Summer 2027: consolidate into a draft Part I, review for consistency with the data model and the current text.
• 2027 CF Workshop: present the draft for discussion.

Questions for discussion

To get the ball rolling:

1. Do you support the Part I / Part II structure as the target, or would you propose a different one?
2. Who are the primary audiences for Part I and Part II respectively?
3. What format should the machine-readable semantics take, and what should be generated from it first?
4. Where should the work live: a branch of the cf-conventions repository, or a separate repository?
5. Would you like to volunteer for a section or join (one of) the working group(s)?

Any and all comments and suggestions welcome.

[davidhassell]

Hi @pvanlaake,

I neither agree nor disagree with this, yet, because I realise that I may not understand it properly.

It was mentioned above that "There should be a single CF specification", and there already is, and it works for all known encoding formats (CDL, netCDF-3, netCDF-4, Zarr, ...). Describing everything without reference to an encoding, and then separately describing an encoding for the stuff you've previously described, possibly more than once, seems to be less easy to understand and is necessarily duplicative. An important question is what you want to do about the fact that CF is currently a convention for the netCDF API. Is that up for debate? I see no use case for which the netCDF API is insufficient and, given that, what is wrong with describing the encoding for netCDF API?

Looking forward to discussing and learning more on these (and other) points and questions,
David

[ethanrd]

Hi David,

Because there is a fairly straightforward mapping between netCDF and Zarr, Zarr is perhaps not the strongest argument for splitting CF semantics out from the encoding. And there is the counterargument of having multiple CF encodings in Zarr. However, I think there are data formats and use cases that don't map as clearly to netCDF.

For instance, mapping CF semantics to tabular data formats (e.g., Parquet and IceBerg) seems like it could be quite interesting and potentially useful for station, sounding, and other observational data that can be described according to CF Chapter 9 "Discrete Sampling Geometries".

[davidhassell]

Hi Ethan,

Thanks - that's interesting! I'll go away and think more on that context :)

[dblodgett-usgs]

Dear Patrick,

I'll echo @davidhassell and say that, while I don't disagree with this in concept, I also don't agree that this level of formalism is needed.

The CF convention has always resisted modularity, favoring direct implementation-ready components. My read of the proposed path to establish a cleaner separation between semantic concerns and syntactic concerns is assuming a modularity that is not a natural fit for the way the CF convention is constructed today. The convention is itself normative, but it doesn't have a formal normative component structure. Without that sort of structure, establishing even an internal modularity is a challenge that would likely need to be addressed at a more fundamental conceptual level (thinking the CF constructs appendix) before trying to address the topic of semantic and syntactic separation.

Similarly ambitious efforts have been attempted in the past. e.g. https://github.com/diwg/cf2, https://binary-array-ld.github.io/netcdf-ld/ -- I'm not drawing a similarity in goals, but want to point out that incrementalism has won out over step-change in the CF community over the years.

I think the way to achieve the goal of a cleaner separation of concerns to facilitate additional syntactic layers for CF semantics is through targeted incremental refinements of the specification. Targeted in that they solve an implementation-level goal or otherwise address a narrowly scoped aspect of the specification such that people can run small thought experiments about how this would impact those who hold data encoded in CF or write software that assumes CF conventions.

I would suggest an initial pull request to adjust the normative references to the NUG and UDUNITS such that they are scoped to semantic interoperability only -- that would open the door to additional implementations that are compatible with NUG and UDUNITS semantics but take different syntactic approaches to encoding the CF constructs. I don't see that initial step as needing more than a person to draft the changes and open a pull request for discussion.

From there, additional encodings of CF could be adopted as appendices or implemented as external conventions that import a version of CF.

Am I missing something important that requires so much more work? Maybe I'm not seeing the problem with CF as it stands in the same way.

Regards,

Dave

[pvanlaake]

Dear David and Dave,

Thank you for your comments. It's unfortunate that you could not attend the hackathon where your comments could have been discussed with the wider group. The participants in the hackathon largely agreed on the principle of separating the semantics from the encoding. Which is not to say that the outcome of the effort to separate the two will be accepted by the community - the agreement was to have a proposal for conventions semantics and netCDF encoding ready for presentation at next year's workshop. There is plenty of time therefore to continue the discussion on how to do this.

In terms of the rationale for doing this, I'll refer you back to discussion #463. In a comment @ChrisBarker-NOAA indicated that the CF Committee reached consensus that "CF should look at containers (file formats) other than netCDF". In his elaborate summary of the discussion up to that point @JonathanGregory concluded by saying:

We have always had in mind, as the conventions document says, that CF could be applicable to other containers than netCDF. During the development of CF, a lot of thinking has been done by many people that could be useful in any file format for the kind of data that CF deals with.

Quite a lot of the conventions text could be written in the terms of the data model, without referring to any encoding, but this might not be suitable for all uses of the document. It's quite often remarked that many just look at the examples. Perhaps that means we need to provide the CF conventions document in separate versions for netCDF and Zarr, maintained from the same source for consistency. Each encoding needs its own translation of the examples and the terminology.

That would really be a big job, but it could be done a bit at a time, in a branch of the conventions repo, starting with nothing and gradually copying and adapting sections, in the priority order of use cases. If we undertook such a complete rewrite of the document, that might also be an opportunity to implement BCP14 language or some adaptation of it, as we have decided to do, I believe (but without a definite timetable). Once completed, this new version would deserve to be called CF 2.0.

@davidhassell wrote:

An important question is what you want to do about the fact that CF is currently a convention for the netCDF API. Is that up for debate?

I think this is not phrased correctly, although you have the literal text of the convention to back you up. The intent is clearly that CF is a convention that is currently encoded for the netCDF API but more widely applicable. From section 1.1:

The main purpose therefore, is to propose a clear, adequate and flexible definition of the metadata needed for climate and forecast data. Although these conventions are specifically targeting the netCDF format, most of the ideas are of wider application. The metadata objects could be contained in file formats other than netCDF. Conversion of the metadata between files of different formats will be facilitated if conventions for all formats are based on similar ideas.

So there is no need for a debate on this. The conventions text is currently written for the netCDF API, but that is a targeting, not a principle.

In his comment in the other discussion, @dblodgett-usgs wrote:

I am happy to support either the in-CF-document path or the third-party-conformance path you describe, and I think NZ can serve both. What I would like to avoid is letting the governance question block the near-term incremental progress.

None of this work impedes other developments. Jonathan expressed his support for it in his summary. In fact, I think such efforts may benefit from the work separating the semantics from the encoding.

In practice, CF-encoded data is already being stored in different containers, with varying levels of conformity with the conventions. ESA is moving its entire Sentinel archive from its SAFE (netCDF) format over to GeoZarr (Zarr extended with several GeoZarr conventions). Copernicus C3S is using a Zarr data store for ERA5 data in ARCO retrievals. ERDDAP support Parquet files with CF metadata. The participants in the hackathon agreed that this community should make an effort to serve those environments as well, next to netCDF.

[dblodgett-usgs]

Dear Patrick -- you may be misreading my comment as disagreement. I agree that we should work to reframe CF conventions as format agnostic. Where I think we don't align is in our assessment of cost and benefit of change on the magnitude of change needed to achieve the goal of multi-format compatibility.

For example, I do not agree with extracting machine-readable semantics from the current spec as a first order of business. From a semantic point of view, I think the specification does quite a good job of conveying the meaning of its constructs as is. Further, I don't see very many opportunities to use formally encoded semantics in this space without some fairly large commitment from an institutional player or open-source development community to pick the work up and use it in a production context.

So let me just answer your questions directly.

• Do you support the Part I / Part II structure as the target, or would you propose a different one?

No -- I prefer to leave the structure as is and make targeted edits that reduce the reliance on the NetCDF API. I favor an approach that maps CDL (as is used in many examples) to alternate encodings as has been done for Zarr and could be done for others.

• Who are the primary audiences for Part I and Part II respectively?

I don't know and the fact that this is even a question is an indication that we don't have a clear source of value for these proposed changes. My comments above re: CF resisting modularity should be read in this context.

• What format should the machine-readable semantics take, and what should be generated from it first?

I find the CF spec quite machine readable in the era of LLMs to be honest. If there's one "most ubiquitous" machine readable format that could work for this application, I would suggest JSON-Schema in line with Zarr-conventions. The RDF-compatible encoding world (RDF-XML, TTL, JSON-LD, etc.) is just not widely adopted and tends to be overly opaque for those who aren't natives to triples as a system. More generally, by choosing ANY formalism other than prose, you introduce buriers to those who have different technical foundations -- which is one of the main reasons NOT to adopt a formalism.

Where should the work live: a branch of the cf-conventions repository, or a separate repository?

This question was what made me think of CF2. If you have to ask this question about what you are doing, you have to have a very strong problem statement and you have to have clear buy in that your branch or seperate repo will, once some measurable objective has been achieved, be merged back into the main product. Without that commitment, you risk forking the developer community or ending up doing something that just doesn't achieve relevance.

Would you like to volunteer for a section or join (one of) the working group(s)?

I would volunteer to help reduce the reliance on the NetCDF API (which is why I'm taking time to discuss this here) but don't feel that a working group is required to achieve its goals.

---

Perhaps others from the hackathon could offer their perspective. I would very much change my stance here if there were institutional representatives or developer communities with clear user needs were piping up.

Regards --

Dave