Consolidation of `cf-json` and `nco-json`

[kwilcox]

Hey folks, I think most of you were involved in the conversation about a year and a half ago regarding CovJSON, cf-json and nco-json. Most was on the cf-metadata mailing list and carried over to covjson/specification#86.

I'd like to focus in this issue on resolving the differences between cf-json and nco-json so we can focus on developing and promoting a single spec. I myself use nco-json extensively. Here are what I consider the main differences and please post any objections or clarifications as it has been a long while since we had this conversation

1. data is required in cf-json and optional in nco-json
2. nco-json supports groups. I've barely kept up with CF-2.0 discussions but supporting this now might be a good idea?
3. Support for flattening the data arrays. The client is then responsible for reshaping to the defined variable shape
4. Support for typing both variables and attributes

@czender @BobSimons @ChrisBarker-NOAA @rsignell-usgs @pedro-vicente @davemetocean @aportagain

[czender]

Kyle,
First a general comment and then some clarifications. NCO-JSON can losslessly express in JSON any CF-compliant file so long as CF is based on netCDF (i.e., presumably for a long time). My understanding (someone please correct me) is that CF-JSON is/was intended to deal with higher level structures that could benefit from additional abstraction, e.g., a CRS object (?). In any case it is unclear that CF-JSON was designed or intended to represent the entire netCDF feature space (as opposed to the CF subset of that space), so I do not think NCO-JSON and CF-JSON are necessarily in conflict. In fact, my impression was that CF-JSON did adopt NCO-JSON about two years ago, and that now CF-JSON has no differences from NCO-JSON in pedantic mode aside from possibly some CF-specific higher level abstractions. I hope someone from CF-JSON clarifies this.

1. NCO-JSON requires data for variables always, but in non-pedantic mode omits data from attributes. Is this what you're referring to? All the early adopters of NCO-JSON that I am aware of seem to prefer pedantic mode, which requires data for attributes too. I'm having second thoughts about making non-pedantic mode the default for NCO, and am considering make pedantic mode the default. Then it could be said that NCO, like CF-JSON, requires data (unless the user opts-in to the non-pedantic mode). Thoughts?
2. Groups seem to be on their way into CF. Maybe in CF-1.8. Maybe even this year.
3. NCO-JSON supports flattening arrays but structures them by default. Flattening is enabled by adding four to the base NCO-JSON format, e.g., --jsn_fmt=4. I do not think flattening by default is a good idea.
4. NCO-JSON requires typing for all attribute types only in pedantic mode. Point 1 above would also address this. More thoughts on this?
   Best,
   Charlie

[kwilcox]

In fact, my impression was that CF-JSON did adopt NCO-JSON about two years ago, and that now CF-JSON has no differences from NCO-JSON in pedantic mode aside from possibly some CF-specific higher level abstractions. I hope someone from CF-JSON clarifies this.

Also hoping for clarification on if cf-json features were intended to be a sub-set, super-set or the same set as nco-json. It seems to me like we should all advocate together for the same format and I don't quite understand why we have two formats!

1. I was only referring to the data field on variables which contains the data in array form. The cf-json spec says it is required. It is not required in nco-json (I can dump as JSON using ncks -mM --json and it is omitted). I'm advocating for this to remain optional. My usage of nco-json revolves around allowing analysts to curate metadata records (attributes) externally to the data. It is stored and indexed in nco-json. The nco-json is applied to netCDF files later on in a pipeline. I don't require the data field on variables, this use-case of nco-json could be called "attributes-only" or "data-less".

2. Excellent, let the confusion begin!

3. When I do use the data field on a variable I always flatten because it is more efficient to load as a flat array and reshape into the already-defined-in-the-json shape of the variable, but I agree, it shouldn't be the default.

4. I would like to see cf-json support typing of attributes like nco-json does in levels 2 and 1 of the JSON output (pedantic and pedantic-light mode). I use level 1 with a custom rule that I specifically specify the type unless it is a NC_STRING.

[ChrisBarker-NOAA]

As I see it, CF was/is built on the netcdf data model. Thus you should be able to "do" CF in any format that fully support netcdf. With that in mind, ideally, "CF JSON" would simply be:

"Use nco-json, and follow all the same rules that CF has"

And it sounds like we are close to that, except that maybe it's:

"Use nco-json in pedantic mode, and follow all the same rules that CF has".

So ideally, if there are any discrepancies still, we hash those out for nco-json.

On to some points:

I was only referring to the data field on variables which contains the data in array form. The cf-json spec says it is required. It is not required in nco-json (I can dump as JSON using ncks -mM --json and it is omitted). I'm advocating for this to remain optional. My usage of nco-json revolves around allowing analysts to curate metadata records (attributes) externally to the data. It is stored and indexed in nco-json. The nco-json is applied to netCDF files later on in a pipeline. I don't require the data field on variables, this use-case of nco-json could be called "attributes-only" or "data-less".

Interesting -- I had thought of "optional data" as being when you have a variable only in order to store the attributes, but don't really have any meaningful data to attach. IN that case, while I think netcdf simply does not allow data-less variables, JSON does, so we should allow it -- no data means, well, no data that I care about.

But this use case -- there IS real data, but I want a meta-data-only representation is tricky. In theory, you could want that with netcdf as well -- but I suspect there is any CF legal way to specify metadata-only.

Frankly, I think the the way to handle this is to alter CF, rather than being JSON specific, this is off the cuff, but something like:

"""
If a variable has an attribute "meta-data-only" then that attribute should be set to a description of the data type and shape of the data that is associated with the variable, but the actual data will be a minimal length 1 array.
"""

In any case, whether in all of CF or jsut JSON, it needs to be clear whether no data means there is no data, or if it means I've left the data out -- and if you've left the data out, it's probably a good idea to specify the data type and shape (dimensions?)

3. When I do use the data field on a variable I always flatten because it is more efficient to load as a flat array and reshape into the already-defined-in-the-json shape of the variable

Really? why is that? makes no sense to me -- I would expect the JSON parser in a JS interpreter to be about as efficient as possible, and if you are going to generate all those sub-arrays anyway, there is no memory advantage. Personally, I'd much rather "one way to do it", and that way should be keeping the structure in the JSON.

[ChrisBarker-NOAA]

@czender: what does this have to do with the Consolidation of cf-json and nco-json?

Did you accidentally post to the wrong issue?

[czender]

Ooops! Ignore my TMTS (Too-many-tab-syndrome) behavior

[czender]

I thought there would be unanimous endorsement of making pedantic (jsn_fmt=2) NCO-JSON the default, as I offered above. Kyle uses a variation where all attributes except NC_STRING need to be explicitly typed, and clearly there are other variations that NCO could though currently does not accept as different format variations. More importantly, Kyle's usage points to the tension between the design of JSON, which embodies minimalism and not overcomplicating things (e.g., why type strings?) with the desire of some for completely lossless representations (which ERDDAP likes). Rather than declaring one the winner and THE standard, I would prefer that the standard encompass both and allow some variations (as NCO currently implements with the jsn_fmt parameter). Thoughts? Does that make sense or should there be ONLY one netCDF JSON standard and any variation is non-standard?

[kwilcox]

I'm in favor of a more open ended spec allowing all of current jsn_fmt options to be correct. I can see the use-case for non-pedantic files and don't want to restrict someone who doesn't need to save the attribute types from reducing the complexity of their JSON. I don't think we really need to specify the type of format the file is in... parsers and clients of the spec will have to deal with both typed attributes and non-typed attributes and act accordingly. The spec could specifically state that non-typed attributes should be interpreted as string, float or int (similar to jsn_fmt=1).

[czender]

It sounds like @kwilcox and I agree that it should all be valid. I guess I'm still looking for guidance on whether the NCO default should be jsn_fmt=0, 1, or 2. Ideally that choice would be consistent with the wishes of the largest fraction of the userbase. It's almost a guess at this point, yet based on my informal polling at AGU in December, I think the ERDDAP folks generate the most NCO-JSON. Am I wrong? And Kyle, would it be helpful if NCO added a new jsn_fmt=3 that types all attributes except NC_STRING?

[ChrisBarker-NOAA]

@czender wrote:

the tension between the design of JSON, which embodies minimalism and not overcomplicating things (e.g., why type strings?) with the desire of some for completely lossless representations (which ERDDAP likes).

I"m not sure that quite captures the difference -- I don't know that JSON embodies minimalism, but rather it IS a format with minimal complexity. It was originally designed to be a serialization for Javascript, which is also a pretty simple language (or at least has a limited type system):

• There is one only type of number.
• There is only one text type.
• There is only one "array" type.
• There is only one "mapping" type.

netcdf and CF support a wider set of types, so there is a mismatch. It does seem that there are enough use-cases for simpler JSON that having only one full featured mode is not going to fly.

But I wonder if we can more clearly define the "simpler" use cases, and thus have semi-lossless conversion without being pedantic -- that is, there is a subset of netcdf data that can expressed in a simpler form, and it is clearly defined how to go from JSON to netcdf without a full spec. I"ve lost track a bit, but somethign like:

If there is no datatype specified, then numerical data are doubles, and string data are unicode text with a standard encoding. (Sorry, I've really lost track of how to express that in netcdf :-) )

If there is no dimensionality provided for an array, it is the dimensionality it is presented in -- that is, 1D is 1D, nested for higher dimensions.

If there is no data, then the data is irrelevant (that is, an attribute only variable) -- in that case.

[note: we could maybe make a distinction between integers and floats if we wanted]

When going from netcdf => JSON, we could also have well defined rules:

All numeric types are JSON numbers
All text types are JSON strings
All ND arrays are displayed as nested
All empty data arrays are left off.

This would allow almost loss-free round tripping ....

In fact, the JSON writer would check for values that are out of bounds and raise a warning or error. Though now that I think about it, out of bounds values are going to be a problem anyway...

Finally, maybe the default behaviour could be the less pedantic mode when there is no ambiguity.

-CHB

[czender]

@ChrisBarker-NOAA These are helpful suggestions, and they appeal to my minimalistic side. Moreover, they largely though not completely consistent with how the default NCO implementation, aka jsn_fmt=0, prints attributes. Level 0 formats attributes without types if possible. One difference is that NCO currently has no way of printing variables that way. It calls for a new formatting level which simplifies the printing of variable data similar to attribute data. I think it is worth doing because, as you say, it preserves the essential data in the sprit of JSON, with minimal overhead.

[ChrisBarker-NOAA]

Level 0 formats attributes without types if possible.

How often are non-string attributes actually used? For my part, I always thought that strings were the only option for attributes (until the conversation about JSON...)

But anyway, it is fully specified and lossless to have default types for JSON. That is, specifying that:

[1.0, 2.3, 4.5]

is an array or doubles is exactly the same as not specifying the type.

So could we say that anywhere a type is not specified it IS the JSON type?

Then folks converting netcdf to JSON would, by default get "untyped" JSON for doubles, and could opt in for untyped (uptyping) for floats, for example.

I'm still a bit stuck on strings, as I am very confused about how one is supposed to handle unicode in netcdf.

[kwilcox]

How often are non-string attributes actually used? For my part, I always thought that strings were the only option for attributes (until the conversation about JSON...)

Non-string attributes are essential. _FillValue, missing_value, valid_min|max|range, scale_factor, add_offset are some "standard" based one that need to be typed to be used correctly.

I also wanted to clear up that the intention of cf-json and nco-json was to be a a convenient language-agnostic, text-based, and easily-parse-able (in contrast to CDL) data format. While JSON may have its roots in JavaScript, its usage is well beyond that.

1. data is required in cf-json and optional in nco-json

I'm glad this spurred more ideas and thoughts than I thought it would. I've noticed other groups talking about a generic text-based metadata format and nco-json without a variable's data included is what I've been using for a year or so. I'd like to avoid having to call this kyle-json and somehow build this into the spec. I have never really thought about how someone would "know" if it was data-less or not, and even if that is something that would be required. Someone could just check for the existence of the "data" attribute if they need it. And if they don't need it then can just ignore it?

2. nco-json supports groups. I've barely kept up with CF-2.0 discussions but supporting this now might be a good idea?

I'd like to see cf-json be a formalized nco-json, i.e. the nco-json of an already CF compliant netCDF file. So as CF starts thinking about groups, cf-json needs to adapt how nco-json does groups accordingly. A netCDF file is to nco-json as CF netCDF file is to cf-json. cf-json is still compliant nco-json.

3. Support for flattening the data arrays. The client is then responsible for reshaping to the defined variable shape

I'm good with always keeping the data in the shape of the variable. We decided something already. High fives. ✋

4. Support for typing both variables and attributes

For a lossless round trip of netCDF -> JSON -> netCDF, both the attributes and variables need to be typed. There is no alternative. If cf/nco-json wants to support round-tripping (I think it should), then being able to specify variable and attribute types is a requirement. I think we agree that the ability to type is required, but it should also be optional if someones use-case does not require a 1:1 representation. Type or don't type, it is up to the user. I'd prefer if variables required a type.

[aportagain]

Thanks @kwilcox for bringing this discussion here, and everyone else for their contributions so far! And apologies for not getting involved earlier, unfortunate timing, this issue was opened the day after I went on annual leave and I just got back yesterday :) I'll have a chat with some of our (internal) active users of CF-JSON over the next few days to get their opinions and then reply in more detail.