Dear Charlie @czender

Thanks to you and others at the hackathon for considering this.

I agree with you that it is a good idea to split up the description of the compression into a number of pieces rather than gluing them together in something like _QuantizeBitGroomNumberOfSignificantDigits. Your suggestion for the keywords look CF-friendly. I generally favour spelling things out rather than using abbreviations unless it's unbearably cumbersome or the abbreviation is widely known.

I may be misunderstanding your CDL. I can only see two metadata schemes sketched out: a container variable with attributes, or a string-valued attribute with key-value pairs. Either of those is CF-compatible. Of those two schemes, I think the container variable is better, because (a) it can be shared by many data variables, thus reducing redundancy, (b) it makes the data variables themselves more readable, (c) since some of the values are numeric, it feels nicer not to have to extract them from a string.

What would it mean if the family is unknown? Is this there because you foresee there might be some kinds of lossy compression which don't work by quantisation? Do you have some in mind? If not, then maybe we don't need this level yet. It could be introduced when there is a need for it.

Would the vocabulary for these schemes be defined by an appendix of the convention, or by other documents (like the standard name table), do you think?

Best wishes

Jonathan

Apologies everyone for my long absence from updating this issue. Thanks @JonathanGregory for your comments last October. I updated the issue in December 2022, and then @davidhassell provided more feedback in March 2023 that included comments on issues Jonathan had raised earlier (above). I responded to David's feedback in May 2023 and I wanted to retain the content of that offline exchange so that others could follow our discussion. With his permission, below I quote David's comments from March (double indented) and my responses to those (single indented).

My responses to your message of 20230316 are interleaved:

I think that option #2 is preferable for the same reasons a), b), and c) that Jonathan describes.

I favor #2 as well.

Also, when the container is shared between 2+ variables in one dataset, it is likely (as opposed to unlikely) that the parameters will differ (such as NSD, NSB).

I expect many users will apply the same (overly conservative set of) lossy compression parameters to save themselves the time it would take to optimize the parameters on a per-variable basis. Still, it is necessary to allow the flexibility to have per-variable parameters.

I am not so concerned about cluttering up the variable with extra attributes, as the parameters are of key scientific importance. As they are logically properties (in the CF data model sense), an application would present them as such in any case, however they were stored.

Agreed.

If lossy compression is known to have been applied, by the presence of some underscore attribute, then providing a lossy compression container should be mandatory.

OK by me for CF-compliance. Writing the container and the per-variable attributes is likely to be the main chore for developers. The lossy algorithms themselves are often/usually library functions that already exist and are callable by software that knows nothing about CF or netCDF.

The relevant parameters should be mandatory when a lossy compression container is provided.

Agreed. Any variable referencing the container variable must have the required per-variable attributes to be CF compliant.

The prior and post metrics should be optional. If present then they may inform the user whether they want to use the data, and if so how they use it. The data is still fully defined if they are missing.

Agreed. Good point. This reduces the number of required parameters from a minimum of six to a minimum of four, three of which would be in the container. Hence a minimum of two attributes per variable: one to point to the container, and one to provide a per-variable parameter like NSD.

The six properties of lossy compression ...

1. Lossy algorithm family (e.g., Quantize or unknown)
2. Algorithm name (BitGroom, BitRound, Granular Bitround, BitShave)
3. Implementation (potentially including library name, client name, author name, software version)
4. Algorithm input parameters (e.g., NSD, NSB)
5. A priori metrics (e.g., maximum relative or absolute error can be predicted/guaranteed for some algorithms)
6. A posteriori metrics: DSSIM Structural Similarity Image Metric (these can be computed for any algorithm by comparing the raw and compressed data)

... should be described in a new appendix, in a similar way to how grid mappings are described. Only the algorithms detailed in this appendix are allowed.

Agreed

1., 2, would have a CV of allowed string values. What about 3. - CV or free text?

I can't see any practical way around allowing Free text for the Implementation attribute.

Could the name "lossy compression" not be the best one? What we are describing is not really compression, right? - lossless compression is applied later.

Names are important and I am certainly open to alternative names. You are correct that quantization algorithms only (lossily) pre-condition data for lossless compression, they do not compress anything themselves. That said, the only purpose for quantization that I am aware of is subsequent compression. And it is intended (though not required) that this proposed CF convention be extensible to non-quantization lossy codecs (e.g., Sz, Zfp). Hopefully whatever terminology is adopted can meet these dual roles.

And let me also respond directly to Jonathan's question which has not yet been directly addressed:

What would it mean if the family is unknown? Is this there because you foresee there might be some kinds of lossy compression which don't work by quantisation?

Yes

Do you have some in mind? If not, then maybe we don't need this level yet. It could be introduced when there is a need for it.

The proposal is currently only fully flesh-out for Quantization. Other potential lossy compression algorithms include Sz and Zfp. I know little about those algorithms other than their homepages. Whether they might comprise their own "families", or both be members of a single family is unclear to me. While Sz and Zfp seem like two likely candidates for codec species to implement support for under this proposed convention, I'm still unsure what the next logical "family" of lossy algorithms is. In that sense I agree with Jonathan that other values of family are currently unknown, and thus the family parameter could be optional until there is a well-defined implementation for a family of non-quantization algorithms. Feedback welcome.

Hi @davidhassell and @JonathanGregory,

I am back in Santander today and all next week, courtesy of @cofinoa :) My goal next week is to draft and submit the CF PR for this issue. It would be helpful to have some guidance on a few high level question before, so I start off on the right foot with the PR:

1. Shall I leave the Technical Proposal Summary in this issue as is, extend it to explain that after due consideration in the comments below we settled on method number 2 above, or instead revise and condense it to include only the necessary details for method number 2?

2. For the PR, I was thinking of adding a Section 8.4 "Lossy Compression by Quantization". The only methods I plan to discuss are Quantization-based, even though the proposed encoding accommodates more general algorithms. And an Appendix L on "Metadata to Encode Lossy Compression Properties", in which the controlled vocabularies and examples for the 3 support Quantization methods would be detailed. Does that sound about right? Or...? Of course, the titles and sections could be changed during review.

There is also one recent update to report. NCO has supported the method number 2 draft proposal since version 5.2.0. So if you use NCO to perform lossy compression, the output dataset will contain the container variable and the minimal amount of lossy metadata in the proposed format. My upcoming EGU24 poster shows some of this:

EGU24-13651
Why and How to Increase Dataset Compression in RDIs and MIPs like CMIP7
by Charles Zender
Session details here
The display is Thursday, 18 April 2024, 08:30-12:30. Please visit between 10:30-12:30 if you'll be at EGU.

Here is an example:

zender@spectral:~$ ncks -O -7 --cmp='btr|shf|zst' ~/nco/data/in.nc ~/foo.nc
zender@spectral:~$ ncks -m --hdn -C -v prs_sfc,compression_info ~/foo.nc
netcdf foo {
  dimensions:
    lat = 2 ;
    lon = 4 ;
    time = UNLIMITED ; // (10 currently)

  variables:
    char compression_info ;
      compression_info:family = "quantize" ;
      compression_info:algorithm = "BitRound" ;
      compression_info:implementation = "libnetcdf version 4.9.3-development" ;

    float prs_sfc(time,lat,lon) ;
      prs_sfc:_QuantizeBitRoundNumberOfSignificantBits = 9 ;
      prs_sfc:lossy_compression = "compression_info" ;
      prs_sfc:lossy_compression_nsb = 9 ;
      prs_sfc:long_name = "Surface pressure" ;
      prs_sfc:units = "pascal" ;
      prs_sfc:_Storage = "chunked" ;
      prs_sfc:_ChunkSizes = 1145, 2, 4 ;
      prs_sfc:_Filter = "32015,3" ;
      prs_sfc:_Shuffle = "true" ;
      prs_sfc:_Endianness = "little" ;
} // group /

Dear Charlie @czender

Thanks for the update and for being willing to work on this.

There's not a definite rule about what to do with the first posting in the issue when it gets outdated. My own view is that it's good to keep the proposal as originally stated, because it is needed in order to understand how the discussion began. If I were you, I would make a new posting to this issue, perhaps with some of the headings and text repeated from the first posting if that suits your purposes, and edit the first posting itself to give a link to the new version e.g. under the latest moderator summary section. But I think any way of proceeding which is clear and suitable would be acceptable, to be honest!

I think your plan for a new subsection and appendix are fine. Whether to have an appendix depends on how long the detail is. If it's not huge, it could be in the subsection e.g. all the detail about cell_methods, which is quite complicated, is in Sect 7.3, not an appendix. Appendices are useful for large detailed sections, like Appendices H I and J, or for lists which might get added to, like Appendix A (and most others).

That's great news that you've already implemented the draft proposal!

I hope you enjoy your time in Spain.

Cheers

Jonathan

Hi Charlie,

This is good news! I agree with Jonathan's comments.

In particular (largely just restating items Jonathan said :))

• A new post stating that what options 2 is, and that we've agreed on it, would be a good thing.
• I think Section 8.4 "Lossy Compression by Quantization" is a good title
• The PR could be written with an appendix or subsection - which ever you prefer - and we can make a final decision on whether or not that's the best place once it exists.

All the best,
David

Hi @czender - is the PR ready for review?

@davidhassell not yet. It still needs formatting, examples, and clean-up. Maybe next week. Don't worry, I'll request your review when ready.

Dear All,

I submitted #519 to implement the core of this proposal. As noted above, it does not include anything about lossy compresion error metrics. It could, but I was running out of steam. I would appreciate anyfeedback on the core of #519 before deciding whether to extend it to include error metrics.

One issue with error metrics is that there is a large menagerie of them. I think someone with a better background in statistics could do a better job than me, at least with the "Post metrics". There is one (and only one, AFAICT) "Pre metric" that is easy to implement and to understand. Specifically is is the maximum relative error incurred by BitRound. This error is simply 2^(-NSB+1) for all values. Hence it is easy to understand, and to compute either before or after quantization. I have added this single error metric to the reference implementation in NCO. The output looks like this:

zender@spectral:~$ ncks -O -7 -C -v ps,ts --qnt_alg=btr --qnt default=9 --qnt ps=13 --cmp='shf|zst' ~/nco/data/in.nc ~/foo2.nc
zender@spectral:~$ ncks -m -C -v ps,ts,compression_info ~/foo2.nc
netcdf foo2 {
  dimensions:
    lat = 2 ;
    lon = 4 ;
    time = UNLIMITED ; // (10 currently)

  variables:
    char compression_info ;
      compression_info:family = "quantize" ;
      compression_info:algorithm = "bitround" ;
      compression_info:implementation = "NCO version 5.2.5-alpha02" ;

    float ps(time,lat,lon) ;
      ps:standard_name = "surface_air_pressure" ;
      ps:units = "Pa" ;
      ps:lossy_compression = "compression_info" ;
      ps:lossy_compression_nsb = 13 ;
      ps:lossy_compression_maximum_relative_error = 6.103516e-05f ;

    float ts(time) ;
      ts:standard_name = "surface_temperature" ;
      ts:units = "K" ;
      ts:lossy_compression = "compression_info" ;
      ts:lossy_compression_nsb = 9 ;
      ts:lossy_compression_maximum_relative_error = 0.0009765625f ;
} // group /

Here at EGU24 today, a few people mentioned they would be more inclined to use lossy compression if the error metrics were included with the data. Any of the metrics mentioned above are possible to add if the software has access to both the quantized and raw arrays. So you can imagine, e.g., correlations, SNR, etc. being computed and added by software that wants to do so (and has access to raw and quantized data). Thoughts on whether to add the maximum_relative_error, and whether to add more complex metrics, to the PR are welcome. Personally I am only inclined to add the maximum_relative_error myself, and only to add it to the convention once the initial PR has been finalized (because I'm learning, again, how much work it is to draft these conventions!).

Hi Charlie,

I think it makes sense to wait to add error metrics until after the initial PR is finished. It sounds like a topic that warrants a separate discussion of its own. (And it also sounds like that will help keep this issue from stalling out, which is good.)

Hi Charlie,
Thanks for getting this together - I'm about to dive into the PR (today/tomorrow). I'm also good with tackling the metrics later.
David

Hi Charlie,

I'm currently reviewing the text, but before I carry on, I thought it would be good mention the term lossy comrpession variable. I not sure that this is a good name. I think that it's fine to describe quantization in chapter 8, because it's primary use case is to compress, but the container variable described is not generic to lossy compression (e.g. lossy compression by coordinate subsampling also has a container variable called interpolation variable). Rather, it is specific to quantization, so wouldn't quantization variable be better? I would follow this through to other uses of "lossy compression", e.g. I might call the per-variable attribute quantization.

What do you think?

Cheers,
David

Hi David,

Thanks for editing this. Let me explain the intent. And then you and @JonathanGregory can decide:

A bit further on in the "family" section the PR says: "Other potential families of lossy algorithms include rounding, packing, zfp, and fpzip." And that's not a comprehensive list. Other lossy algorithm families could be, e.g., discrete cosine transform, layer packing, logarithmic packing...AFAICT, the lossy_compression variable as outlined in the PR could fit all of these families, and would always signal to the user that lossy compression of some sort had been employed, with the family and algorithm attributes bringing more specificity.

If we change the name to quantization variable, then may as well eliminate the family variable completely. And, if CF is extended later to include other lossy compression algorithms, then we could expect to have a zfp variable, a fpzip variable, a logarithmic packing variable. In other words, turning the generic lossy compression variable into a more specific like quantization variable sets a precedent that would naturally be followed by subsequent algorithms placed in CF. If that sounds desirable to folks, fine, I'll change the PR accordingly.

Note that we discussed what to name the container variable at the 2022 workshop. I think there were folks on both sides of the question you are (re-?) raising. The current PR implements what we hammered out at that workshop. Of course, it's better to change things now if in hindsight you/we think the workshop conclusions could be improved.

As a data producer / consumer, I strongly favor having a single lossy_compression variable/attribute that could be relied on as an indicator of its presence.

It's much easier to check for that and then decide how to deal with it than to check for a whole bunch of different indicators, and the failure state is much better if the compression uses an algorithm that you didn't know existed. It's also more communicative to folks who may have never encountered the issue before; if I'm scanning through the headers of a file, the term lossy_compression is going to catch my attention in a way that quantization or zfp would not.

Hi Charlie and Seth,

You both put the reasons for using lossy_compression well, and I agree with them. Thanks for explaining things.

However I'm concerned that we are being misleading in the case that a creator wants to remove false precision for scientific reasons, but doesn't want to use native compression (or indeed can't - netcdf-3). In that case we haven't done any lossy compression, but have done a lossy filter (is that the right word?).

I retract my suggested renaming, but (thinking out loud, here!) wonder if we need two containers: "lossy_compression", and another one for algorithms that have changed the data without compressing it ("filter", perhaps?). The former would not be needed at this time as there is no use case, yet.

Thanks,
David

Hi David,

That's an interesting point, and it will deserve thoughtful consideration if and when we get a use case for it. There are a lot of potential uses for filtering, and I think if we're going to represent them in CF, we want to set ourselves up to use an umbrella approach that can handle all of them in a unified way, rather than dealing with each of them completely independently of the others. So we should try to remember this (potential) use case if/when that comes up.

(I think CF is starting to bump up against the issue of explosive combinatorics making namespaces unmanageable in a number of areas, and I'm concerned about forestalling more of these problems in the future.)

But since we aren't currently (AFAIK) considering any use cases for filtering, we should leave that aside for now.

Cheers,
--Seth

Hi All,

@davidhassell your suggestion is absolutely correct. And thank you @sethmcg for your thoughts. Here's my response to your suggestion:
Quantization per se does not compress and is perhaps better termed a pre-conditioner (as I describe it in the PR) or a filter (in HDF5-speak) whose raison d'être is to be followed by a lossless compression scheme. Quantization can therefore be applied to netCDF3 files as you mention. A few pertinent technical qualifications before responding: I think the netCDF library implementation will not quantize netCDF3-format datasets (it either ignores the request or fails with a suitable error, I can't remember which). The NCO implementation does support quantization of netCDF3 files (becuase, why not?).

@JonathanGregory raised a similar point, above, which I copy along with my response here:

Could the name "lossy compression" not be the best one? What we are describing is not really compression, right? - lossless compression is applied later.

Names are important and I am certainly open to alternative names. You are correct that quantization algorithms only (lossily) pre-condition data for lossless compression, they do not compress anything themselves. That said, the only purpose for quantization that I am aware of is subsequent compression. And it is intended (though not required) that this proposed CF convention be extensible to non-quantization lossy codecs (e.g., Sz, Zfp). Hopefully whatever terminology is adopted can meet these dual roles.

It might be better to segregate, as you suggest, these dual roles into different container variables. Let's consider the implications of a distinct container variable to describe transformations that do no compression. One example is the Shuffle filter. Byte-Shuffle and bit-Shuffle rearrange (bytes and bits, respectively) to enhance subsequent compression, similar to quantization. However, shuffling leaves the values in a non-IEEE format that cannot be stored in a netCDF3 file. A reverse transformation (unshuffle?) is required before using the data so Shuffle must be and is implemented as an HDF5 filter so that it is done transparently for the user. Quantization is IEEE-compatible and has no such requirement so it's fine for netCDF3 files. Which begs the question: Do we really want a container variable for transformations that do no compression? Which transformations are you thinking of? Sure, quantization fits the bill. But is that enough? Are there other transformation algorithms you have in mind that would justify creating a generic container variable to hold them?

Deciding on the right level of generality is indeed hard as there are bound to be trade-offs. Perhaps quantization is just in a category by itself. Which would support your recent suggestion of naming the container "quantization" instead of "lossy compression". I wrote the current PR based on the discussion at the 2022 workshop. Let me reiterate that I'll go along with (and rewrite the PR for) whatever you and @JonathanGregory and any other interested people like @sethmg and @cofinoa decide is best for CF.

We seem to be stalled on this issue. Would everyone interested in this proposal, in some form, seeing the light of day please respond with what changes, if any, you would like to see in the current PR? Helpful responses include, though are not limited to:

1. No Major Changes to PR, though possibly minor changes (please include those minor suggestions)
2. Replace lossy_compression container with quantization-specific container as suggested (and retracted) by @davidhassell above
3. Replace lossy_compression container with something like transformation or filter container that could be shared by other algorithms that alter though do not compress data as suggested by @davidhassell
4. Other (please be as specific as possible)

Mahalo,
Charlie

Hi Charlie,
1: The PR looks good to me as-is.

Dear Charlie @czender

I'm sorry that I've kept you waiting. Below are some minor review comments. None of these imply any change to the proposed convention; they're just about the text. I hope they make sense and you can work out what each applies to.

Regarding the question of what the container attribute should be called, I tend to think that "lossy compression" sounds too general, because Sect 8.3 is about "lossy compression" as well, and the basic "packing" of Sect 8.1 is a kind of lossy compression. Furthermore, as you write in the preable of the new section, this convention is actually not about compression at all, but about modifying the data to make it suitable for compression. The title of the new section being "Lossy Compression by Quantization" makes me prefer "quantization" as a name for the container, and dispense with family, as you suggest in your earlier comment.

You also sensibly remark that there may be other algorithms that come along, which could also be described as lossy compression and might want similar container variables. That's true, but also they might not! I don't think we are tying our hands. The emergence of new use-case in future, which needs a similar treatment but can't be described as quantization, might suggest a suitable name for a container that could handle both quantization and the new method. We could then, for example, make quantization into an alias.

The new attributes should be described in Appendix A, within which you will also need to invent a new "Type" value for the container variable.

Many thanks for developing this addition. I support its going ahead. I hope it won't take long.

Best wishes

Jonathan

Convention

codec. Could you replace this word with less technical English.

define a metadata framework to provide quantization properties. Maybe replace "provide" with "describe" or "record"?

irrecoverable. It's possible it still exists, but that is beside the point. Could we say "differ from the original unquantized data, which are not stored in the dataset and may no longer exist."

I suggest omitting the paragraph "Software can use ...". This isn't part of the purpose or description of the convention; it's an explanation of why you proposed this form of it. It's useful in making the proposal, but we don't usually preserve such reasoning in the conventions document.

lossy compression should be ordinary font, although it's a CF technical term. We use verbatim for the names and contents of attributes and variables.

If and when algorithms. Actually we would probably not do that, since we have a generous interpretation of backward compatibility, in order to minimise misinterpretations, in principle 9 of sect 1.2, "Because many datasets remain in use for a long time after production, it is desirable that metadata written according to previous versions of the convention should also be compliant with and have the same interpretation under later versions." Following this principle, we would never make family mandatory, but we could make quantize its default.

The final attribute. As well as "free-form", you could say "unstandardised".

Each variable that. I think this should be "each data variable". It might also be useful, for the sake of clarity, to state in the preamble that this convention is only for data variables, if that's the case.

"... must include at least two attributes to describe the lossy compression. First, all such data variables must have a lossy_compression attribute containing the name of the lossy compression variable describing the algorithm." That is to explain what "associated" means explicitly. Also I suggest omitting the sentence "This attribute is attached ..." because, again, that is an explanation of the design. I would run into the next para with "Second, all such variables must record ...", and end the para after "preserved by the algorithm". That makes this para apply generally. It leaves two more paras with the requirements specific to the algorithms.

This section briefly describes ... for BitRound. I think "thus" should be omitted before "bias-free". This word implies that BitRound is free of bias because the IEEE adopted it!

Conformance

I would say, "The value of algorithm must be one of the values permitted by thie section." I think that's sound, and it is less likely to have to be updated (one fewer thing to go wrong).

The value of implementation. I would omit this, because it can't be verified by a checker program (unless it's truly intelligent). If you think it's essential to include, I would say, "must be a string that concisely".

The value of lossy_compression must be the name of the lossy compression container variable which exists in the file.

"The value of lossy_compression_nsb must be in the range ..." and similarly for lossy_compression_nsd, because you've already required them to be integer attributes.

Thank you for these comments @JonathanGregory. I'll try to incorporate them in time for the 1.12 release.

@davidhassell and @JonathanGregory and @sethmcg I have addressed and merged the comments from Jonathan's recent review in #519. I hope I interpreted all the suggestions as intended. Please take another look and LMK what further changes are desired. (FYI I have also updated the reference implementation in NCO to adhere to this updated version).

Hi Charlie, I'm away for a week from tomorrow, but look forward to taking a good look when I get back.
Cheers,
David

Dear Charlie @czender

Many thanks for making changes following my comments. I hope they made sense to you.

You have some text explaining why it would not be a good idea to quantize grid metrics. That's new text, isn't it? I think this is useful background to include, although it's not strictly part of the convention. However, it would be helpful to clarify that this text is intended to explain why quantization is not allowed for coordinate variables, bounds variables, etc. (as indicated by Appendix A, where the quantization attribute is allowed only for data variables). As it stands, this text could be read to imply that it is allowed, although not advisable. Have I interpreted your intention correctly?

If so, I would suggest moving and modifying (in bold) a few sentences in the preamble, like this:

The CF conventions of this section define a metadata framework to record quantization properties alongside quantized floating-point data variables. The goals are twofold. First, to inform interested users how, and to what degree, the quantized data differ from the original unquantized data, which are not stored in the dataset and may no longer exist. Second, to provide the necessary provenance metadata for users to reproduce the data transformations on the same or other raw data. These conventions also allow users to better understand the precision that data producers expect from source models or measurements. Use of these conventions ensures that all quantized variables are clearly marked as such, and thus alerts users to cases where these guidelines have not been followed.

These conventions must not be used with data variables of integer type, or any other kind of CF variable. This is because fields that describe idealized or reference coordinate grids, or grid transformations, are often known to the highest precision possible. These fields can include spatial and temporal coordinate variables (e.g., latitude, longitude, time) and properties derived from these coordinates (e.g., area, volume). Degrading the precision of such grid properties may have unintended side effects on the accuracy of subsequent operations such regridding, interpolation, and conservation checks which should generally be performed with the highest precision possible.

Finally, you have the sentence, "In general, we recommend against quantizing any coordinate variable, bounds variable, cell_measures variables, and any variables employed in formula_terms." If my interpretation is correct, the convention isn't allowed for coordinate, bounds or cell measures, so we don't need to recommend against it. Therefore we can omit the sentence, except that some variables named by formula terms are data variables, for example the surface pressure field required by a atmosphere sigma coordinate. Do you intend to recommend against quantizing such fields? If so, that would require another recommendation in the conformance document.

I hadn't previously noted the point that it's only for floating-point data (which makes sense, of course). I think that ought to be a requirement in the conformance document.

Best wishes and thanks for your patience

Jonathan

@JonathanGregory Thanks for your additional suggestions. Responses interleaved...

You have some text explaining why it would not be a good idea to quantize grid metrics. That's new text, isn't it?

No, that text was in the original PR.

I think this is useful background to include, although it's not strictly part of the convention. However, it would be helpful to clarify that this text is intended to explain why quantization is not allowed for coordinate variables, bounds variables, etc. (as indicated by Appendix A, where the quantization attribute is allowed only for data variables). As it stands, this text could be read to imply that it is allowed, although not advisable. Have I interpreted your intention correctly?> As it stands, this text could be read to imply that it is allowed, although not advisable. Have I interpreted your intention correctly?

Yes, you have.

If so, I would suggest moving and modifying (in bold) a few sentences in the preamble, like this ...:

Agreed. Done.

Finally, you have the sentence, "In general, we recommend against quantizing any coordinate variable, bounds variable, cell_measures variables, and any variables employed in formula_terms." If my interpretation is correct, the convention isn't allowed for coordinate, bounds or cell measures, so we don't need to recommend against it. Therefore we can omit the sentence

Agreed. Done.

, except that some variables named by formula terms are data variables, for example the surface pressure field required by a atmosphere sigma coordinate. Do you intend to recommend against quantizing such fields? If so, that would require another recommendation in the conformance document.

Yes, I think that CF should recommend against quantizing all data variables that are in formula_terms since, as I understand it, formula_terms is only used to define grid-variables like vertical coordinates. I just now added this to the conformance document:

"Data variables that appear in formula_terms attributes should not be quantized"

I hadn't previously noted the point that it's only for floating-point data (which makes sense, of course). I think that ought to be a requirement in the conformance document.

I just added this to the conformance document:

"Only floating-point type data variables can be quantized."

The PR now contains all of the above changes. Let's keep this momentum going and resolve all the issues. More feedback welcome.

Charlie

Dear Charlie

Thanks very much for explanations and improvements. I think this is essentially fine! Hoping that you have patience, I would like to make the following minor suggestions:

• At the end of the paragraph beginning "These conventions must not be used with data variables of integer type, or any other kind of CF variable", I suggest "For the same reason, it is recommended not to quantize any data variable which is referenced by a formula_terms attribute of any variable." This is consistent with the recommendation in the conformance document, but it's not now stated in the conventions text otherwise. Unlike the coordinate variables etc., which can't be quantized because they are not data variables, formula terms can be data variables, so we need to say why it's not a good idea to quantize them.

• In the first requirement, I would say "Quantization container variables", because in a later requirement you refer to "quantization container variable" and if we're not consistent it may cause confusion. This phrase makes it clear we don't mean the quantized data variable.

• I would join the third and fourth requirements, thus: "Only floating-point type data variables can be quantized. Quantized variables must have and are identified by having a string-valued attribute named quantization." This is to make clear how to detect a "quantized variable", for anyone writing a checker program. The actual "mechanical" check is that the attribute should not be present on a variable which doesn't qualify to have it.

• To the new, second, recommendation, I would add "and therefore should not have a quantization attribute".

Seth was already happy and I hope that he still is. I expect we will hear soon from David.

Best wishes

Jonathan

Aloha @JonathanGregory,
Thank you for the additional suggestions. I agree they improve consistency, clarity, and completeness. I have implemented them verbatim.
Mahalo,
Charlie