Inclusion of additional cell_methods statistics for climate normal and anomaly products

[Roebeling]

Topic for discussion

At EUMETSAT we are developing a climate normals and anomaly service. The data are stored in CF compliant NetCDF files. In our files we provide several time-series statistics that, as far as we can judge, are not part of the statistics supported in CF Conventions version 1.13. It concerns the statistics

• “standard_error”
• “percentile“ (p05, p10, p25, p50, p75, p90, p95)
• “trend”

From our perspective, it would be useful if these statistics could be included in the cell_methods statistics listed in Appendix E (cell_methods). Is our observation correct, and would it be worthwhile to add these statistics to the next version of the cell_methods table?

Thank you!
202604_CF_Request_Cell_Methods_Statistics.pdf

[JonathanGregory]

@Roebeling

Thanks for your comment. Those three bullet points have different kinds of answer!

• There is a standard_name modifier of standard_error (section 3.4).
• Do you mean that you produce time-dependent or climatological fields of these percentiles e.g. (percentile, time, latitude, longitude) or (percentile, latitude, longitude)? Possibly in this case the percentile could be treated as a coordinate. I agree it's also analogous to cell_methods, especially median, which is a percentile (50%), but that might not be the best way to record it.
• A trend is a regarded as different physical quantity in CF vocabulary, with different units, and therefore needing a different standard name. We have hundreds of standard names for tendencies, beginning tendency_of, which is defined as indicating the derivative wrt time in the guidelines. I think that a trend is a particular way to calculate a tendency over a time-interval. To be specific about how the tendency is computed, a comment could be included in () in the cell_methods, or we could define a new cell_methods, but it would still need a standard name for a tendency.

Best wishes

Jonathan

[Roebeling]

Thanks for your ample reply, that clearifies already many of my questions.

Percentiles: For percentiles we provide climatological fields. In our case, for example, we provide for each month of the year the 10th percentile calculated from the monthly means of a climate normal period of 30 years (12 months, latitude, longitude).

These we now note as:
cell_methods = "time: mean within years time: percentile_10 over years";

Trend products: Thanks for indicating that trend fields have unique standard names, ie those beginning with tendency_of. I will indicate our data model that such standard names shall be used for trend fields. For consistency with other fields in the dataset, I would prefer to use a cell_methods description for these tendency_of products that specifies how the trend is calculated. Currently we do it like this: "time: mean within years; time: trend over years."

Standard error products: Using the modifier, format <standard_name> standard_error clarifies how to handle the standard names of these fields. I will indicate our data model that a modifier shall be used for the standard_error fields. But as above, we would like to provide a CF compliant description of the cell_method . Currently we do it like this: cell_methods = "time: mean within years time: standard error over years";

In conclusion, it is now fully clear how to treat the standard names of the statistics we consider. For the description of the manner how these are computed we would prefer using cell_methods descriptions where possible, and resort to using comments () where nessesary.

[JonathanGregory]

These three points are all non-trivial and worth considering carefully. They can all be treated with the current conventions, but possibly could be done better if we made enhancements.

I can see there's an argument for allowing standard_error as a cell method for the purpose you describe. Is it calculated like the standard_deviation over years, and divided by the square root of the number of years (perhaps corrected to allow for autocorrelation)? This is a well-defined statistical procedure, like other cell methods. In coming months, there's going to be a proposal, which @davidhassell is coordinating, about the description of uncertainties in general. That proposal is relevant to the standard_error modifier. It would be useful to hear David's perspective on your use-case (but he's away for a couple of weeks, so we won't hear from him immediately). Meanwhile, in the current conventions, you could put time: mean within years time: standard_deviation over years (divided by the square root of the number of years), for instance to indicate the interannual standard deviation of January means. Maybe you could use this in conjunction with the standard_error modifier.

A trend is an intensive quantity, like a mean. The default cell method for intensive quantities is point. That would seem surprising for a trend, wouldn't it, but it has some logic: you could say, I think, that the trend during a time interval is the best estimate of the time-derivative at any instant of time within that interval. That's the idea of numerical timestepping schemes, which have a formula for a time-derivative during a timestep, and the value thus obtained is assumed to apply exactly at the appropriate instant within the timestep. Despite this, its being counter-intuitive (even if my reasoning is correct) is an argument for allowing a cell method for trends, as an alternative to point. Since the standard names have "tendency", not "trend", it would be consistent to use tendency for the cell method, I would suggest e.g. time: tendency (OLS regression against time) if you want to record how the trend was calculated.

I don't think we ought to introduce the numerical percentage into the cell_methods; it would be better treated as a coordinate. With the current conventions, you could provide the percentage as a coordinate variable, for which we'd need a new standard name, but we can't describe it with cell_methods. Calculating a percentile from the variation over an axis is like calculating a mean or maximum or median, but unlike those cases the calculation depends also on a parameter (a median involves a parameter too viz. 50%, but that's implied). There is already a case where we use a parameter in a cell methods, that is when we calculate the mean over an area type in sect 7.3.3. Following that pattern, we could have something like time: percentile where P, where P is the name of a coordinate or scalar coordinate variable specifying the percentile. With this syntax you could have several percentiles in the same data variable. What do you think?

Jonathan

[davidhassell]

Hi @Roebeling and @JonathanGregory

standard_error

I'm fine with introducing a new cell method. The aforementioned uncertainty proposal suggests deprecating the use of the standard_error standard name modifier for the use of expressing uncertainty, and in general I'd be happy if we didn't introduce any more standard name modifiers! (Edit August 2026 - the uncertainty proposal is now available: cf-convention/cf-conventions#660)

The standard error of a sampling distribution is the standard deviation of a statistic calculated from that sample, so I don't think that the name standard_error is descriptive enough. The standard error described above (and in 202604_CF_Request_Cell_Methods_Statistics.pdf - i.e. the sample standard deviation divided by the root of the sample size) is the "standard error of the mean (of the sampling distribution)". So a well-defined cell method name could be standard_error_of_the_mean, I think. Other similar standard_error_of_* could be introduced if ever required.

tendency

If we didn't have any tendency_of_* standard names, I image this would be agreeable (as is already in use!): standard_name = eastward_wind and cell_methods = time: mean within years time tendency over years (comment on how tendency was calculated) ... but we do! and it would make intercomparison hard to have these two quite different representations of the same thing, so I think this is not the way to go.

I think I agree with Jonathan's analysis, but find the use of a "tendency" cell method confusing - it's not intuitive to me that the cell method does not indicate that a tendency of the physical quantity (itself tendency) has been carried out, and we'd have to restrict its use to particular standard names. For me "point" makes more sense, as it is implied anyway and doesn't require any special treatment - i.e. standard_name = tendency_of_eastward_wind and cell_methods = time: point (method of calculation).

percentile

I agree with Jonathan on the use of a coordinate or scalar coordinate variable.

[Roebeling]

Dear Johnathan, David,

Thanks so much for looking into this and desribing the way forward. I have now been going through your answers and tried to summarize the information to instruct our data providers how to treat standard_errors, percentiles, and trends. I still have a question, and that is about the approach proposed for the percentile variables. If I understand well, we need to register new standard_names for each pertentile variable that we want to include in our NetCDF files. I assume a bit similar as is done for tenency variables (tendency_of <standard_name>). Is that correct?

In my slide I summarized as follows

I hope I am on the right track, and am curious to read your reply.

Kind regards, Rob

[TomLav]

Hei Rob,

After re-reading @JonathanGregory's answer about percentiles, I wonder if he rather meant something like below: per is the coordinate variable that stores the values at which the percentiles are reported (in my example 10%, 25%, 50%, 75%, 90%). sic_percentiles are the actual percentile values.

(I deliberately left out the coordinate variables that do not directly involve the percentiles).

dimensions:
    climtime = 12 ;
    lat = 180 ;
    lon = 360 ; 
    per = 5 ;

variables:
    float per(per) ;
        per:standard_name = <new_standard_name> ;   // could it be 'percentile' ?

    float sic_percentiles(per, climtime, lat, lon) ;
        standard_name = "sea_ice_area_fraction" ;
        cell_methods = "area: mean where sea time: mean within years time: percentile where per" ;

data:
    percentiles = 10, 25, 50, 75, 90 ;

(edit: I edited the cell method)
The cell method tells that the sea-ice area fraction is a) computed for the sea portion of grid cells (it is the sea-ice concentration), then b) averaged within years (dimension climtime = 12 would define monthly climatology ranges) and finally c) percentile(s) are extracted.

[JonathanGregory]

Dear Rob @Roebeling

Sorry for slow reply. I have been on holiday.

Percentiles

@TomLav is correct about what I meant. He shows an example with multiple percentiles. The method could also be used for a single percentile e.g.

dimensions:
  per = 1;
variables:
  float per(per);
    per:standard_name = "percentile";
  float sic_percentile(per, climtime, lat, lon) ;
    sic:standard_name = "sea_ice_area_fraction" ;
    sic:cell_methods = "area: mean where sea time: percentile where per" ;
data:
    per = 50; // NB this is the variable named after "where" in cell_methods

or per could be a scalar coordinate variable, in which case there is no per dimension:

variables:
  float per;
    per:standard_name = "percentile";
  float sic_percentile(climtime, lat, lon) ;
    sic:standard_name = "sea_ice_area_fraction" ;
    sic:cell_methods = "area: mean where sea time: percentile where per" ;
    sic:coordinates = "per";
data:
    per = 50;

Either way, the special case of per = 50 is the same as the median:

variables:
  float sic_percentile(climtime, lat, lon) ;
    sic:standard_name = "sea_ice_area_fraction" ;
    sic:cell_methods = "area: mean where sea time: median" ;

Your use-case is of climatological percentiles. To indicate this, I think that @TomLav's example should have cell_methods="area: mean where sea time: mean within years time: percentile over years where per" i.e. including over years, like any other multiannual climatological statistic. Similarly, you could have cell_methods="area: mean where sea time: mean within years time: median over years".

I agree that the natural choice for the standard name would be percentile, whose canonical units would be 1. A value of 10 would indicate the 10th percentile, as in @TomLav's example. There's possibly scope for confusion about the units. It would be tempting to specify %, but in that case a value of 10 would mean the 0.1-percentile. The confusion could be avoided by not making the obvious choice. The standard name could instead be cumulative_distribution_function, for instance, having values between 0 and 1, with 0.5 meaning the 50th percentile i.e. the median.

Standard error

I agree with @davidhassell that standard_error_of_the_mean would be the right name for a new cell method. Do you have a definite use-case for this? It would be a straightforward addition to Appendix E.

Trends

The description of tendency standard names say "tendency_of_X means derivative of X with respect to time". I accept @davidhassell's argument that point is OK as the default cell method for a time-derivative, because it is an instantaneous quantity.

The trend (from regression against time) of X is an estimate of the time-mean tendency of X within the interval of interest, assuming that the tendency $\mbox{d}X/\mbox{d}t$ is constant, so that X varies linearly with time. If that assumption holds, a cell method of point is fine, because $\mbox{d}X/\mbox{d}t$ is the same at all points in time.

However, if there are temporal variations in X not due to the underlying constant trend, the instantaneous $\mbox{d}X/\mbox{d}t$ at any point in time is generally not equal to the underlying trend, suggesting that point wouldn't be the correct cell method. Moreover, the time-mean tendency of X is $(1/t) \int_0^t (\mbox{d}X(t')/\mbox{d}t')~\mbox{d}t' = (X(t)-X(0))/t$. That is, the time-mean of the time-derivative of X is equal to the change in X over the meaning period divided by the length of the meaning period. Since this includes any fluctuations in $X(0)$ and $X(t)$ that aren't due to the (constant) underlying tendency, mean would also not be a correct cell method of a trend calculated by regression over a time-interval.

The regression trend does not come from one instantaneous value (point) or two instantaneous values (mean); it's computed from all available values within the time-interval. Each trend value represents the variation within its entire cell, but the quantity is intensive with respect to time, not extensive like for the sum cell method. I accept David's point that a cell method of trend would duplicate the notion of tendency that's included in the standard name, but I still think we don't really have an appropriate cell method for this case.

Therefore I'd like to suggest a new cell method of variation, for which I'd propose the definition: "The quantity is intensive with respect to the specified dimension, and the data values represent the variation within the cells in some manner which may be specified by other metadata, such as the standard_name." You could use cell_methods="time: mean within years time: variation over years" with a tendency standard name to indicate an estimate of the underlying trend of annual means. Does that make sense, and would it meet the need?

Best wishes

Jonathan

[davidhassell]

Just to let you know that the aforementioned uncertainty proposal is now available: cf-convention/cf-conventions#660

[Roebeling]

Dear @davidhassell, This is very good news to read. Having so many more options to include uncertainty information is really helpfull for the works we are doing with climate data. At frist glimps I see that you already cover many aspects. With the developers of our climate data records we will look in more detail though, and see if these new conventions cover the applications we need these for.

Have a nice weekend, Rob

[davidhassell]

I hope that you find what you're looking for! but If you do have use cases that the proposal does not cater for, please do comment on the uncertainty issue and we'll see if we can incorporate them.

[JonathanGregory]

Dear Rob @Roebeling

@davidhassell and I have discussed my suggestion at the end of my last posting. As a result, I'd like to replace the name I suggested for a new cell method. Instead, I suggest we introduce a cell method of intensive with the definition I gave: "The quantity is intensive with respect to the specified dimension, and the data values represent the variation within the cells in some manner which may be specified by other metadata, such as the standard_name. If the variation is represented by a method which can be identified by a particular cell_methods keyword e.g. point or mean, it is recommended to use that specific keyword, because it is more informative, instead of intensive."

If you think that the intensive cell method would be suitable for your use-case, because it's more general than point (which is a confusing way to describe a trend), then I'll propose it. At the same time, I could propose an extensive cell_method, with a corresponding definition, for the same sort of reason, that sum might sound unsuitable for some kinds of extensive cell methods. Perhaps it would also be useful to say something in Appendix E or the main text about the meanings of "intensive" and "extensive" e.g. from IUPAC quoted by wikipedia, "An intensive ... quantity is one whose magnitude is independent of the size of the system. An extensive ... quantity is one whose magnitude is additive for subsystems."

Best wishes

Jonathan

[Roebeling]

Dear @JonathanGregory , thanks for looking into this. I see the point that using variation would be confusing in this case, as trend is not a variation variable in statistical definitions. Using intensive sounds more plausible to me. How would thsi chnage the cell_method description of the example you used in your earlier answer (https://github.com/orgs/cf-convention/discussions/471#discussioncomment-17672872)

Is it correct that your previous suggestion:
cell_methods="time: mean within years time: variation over years"

then would changes to:
cell_methods="time: mean within years time: intensive over years"

If so, working with intensive would be clear and fine to me.

Have a nice weekend, Rob

[JonathanGregory]

Dear Rob

cell_methods="time: mean within years time: intensive over years"

Yes, exactly. I'm glad that it makes sense to you. In that case, I will propose the intensive and extensive cell methods in a conventions issue, and we can discuss it further there.

The same proposal could include standard_error_of_the_mean if you have a definite use case for it - do you?

What do you think about my suggestion for percentiles? That's a more complicated one. There's no hurry (as far as I'm concerned)!

Happy weekend to you as well

Jonathan

[Roebeling]

Dear @JonathanGregory

Thanks for the percentile explaination. Following your suggestion to the comment of @TomLav the idea became clear to me. Meanwhile I already experimented a bit, and now implememted the percentile part as you suggest in your first option ie

Example: Metadata of percentile variables

variables:
float per;
:standard_name = "percentile";
:units = "%";
float sea_ice_area_fraction_percentile(climtime, per, latitude, longitude);
:standard_name = "sea_ice_area_fraction";
:cell_methods = "time: mean within years time: percentile over years where per";
:coordinates = "per latitude longitude";
data:
per = 5,50,95;

Cheers, Rob

[JonathanGregory]

Dear Rob

OK, that is fine. My other examples were for the case of a single percentile.

However, I am concerned about the units of % for percentile, because % units are defined by UDUNITS to mean 0.01. So you're specifying percentiles of 0.05, 0.50 and 0.95, all less than the 1st percentile. This is not what you want. The units of percentile should be 1, I believe. Because % is very tempting, I wondered (above) if we should call the quantity cumulative_distribution_function, which you'd expect to lie in the range 0–1, rather than percentile. The canonical unit for cumulative_distribution_function would be 1, and you could then use 5,50,95 with units="%" to mean 0.05, 0.50, 0.95 in the CDF, which is what you intend, isn't it.

Best wishes

Jonathan

[taylor13]

I for one can't quite get comfortable with the the method being called "intensive". To my mind "intensive" doesn't care how much of something you have as long as any additional substance has the same properties as the original bit. For a trend, this would not be the case. Suppose you have a time interval with a trend x and a second interval where the time-series is identical to that in the first time interval (so again a trend x). If you then connect the two intervals and compute the trend, you probably get something about half of the trend in each of the individual intervals (i.e., nearer x/2).

How should I look at this to make sense of characterizing the trend as an "intensive" property?

[JonathanGregory]

Dear Karl

I agree that "intensive" and "extensive" are not self-explanatory words, and therefore not ideal for CF vocab. We do already use them in App E—we always have as far as I remember—because they're scientific terminology e.g. described by Wikipedia. We could certainly include definitions like Wikipedia has, but I'm sure you know all this. Can you think of better words to use which characterise this distinction?

I agree with your interpretation of "intensive", and I understand your example, but I think there's an obvious counter-example. Suppose you have a timeseries with a fairly constant trend in it, and divide it in two. The trend in each of the two shorter series will be about the same as the trend in the longer series. This fits the definition of "intensive". It's like the mean. If you have an intensive quantity with no trend, you can estimate the mean from any section, but if there's random variability, the estimate will be less accurate if you chop up the dataset. I think that's the sort of situation in which your example would most likely occur.

Best wishes

Jonathan

[taylor13]

Could we simply use the generic term "property" or "statistic" to point out which dimension(s) the standard name is referring to? so a "tendency" representative of a time-interval (and interpretable as a "trend" over that time interval) would have cell_methods: "time: property". (or "time: statistic"). Of course maybe you don't need a cell_methods at all in this case, since trend presumably implies "over time".

We would encourage further explanation of the "property" (or "statistic") by asking the data writers to include some explanatory text in parentheses: e.g., for a variable like tendency_of_air_temperature, we might have cell_methods="time: mean within years time: statistic (calculated as a least squares fit from samples) over years"

[larsbarring]

Dear Jonathan, Karl, David,

On the cell_methods question, I think Karl's concern is the decisive one, and I'd like to add a structural observation and a suggestion.

Every current cell_methods keyword names an operation applied to a cell — mean, sum, maximum, variance — whereas intensive would name a property of the quantity, which in CF is otherwise carried by the standard name and the units. Appendix E's existing use of "intensive" and "extensive" is descriptive, explaining why point and sum are the defaults they are; promoting a descriptor to a keyword seems to me a step of a different kind.

There is also a difficulty with the proposed Appendix E wording. If we quote IUPAC's "magnitude is independent of the size of the system", the motivating case does not satisfy it: for two equal blocks of identical linear series, each of slope x, the regression slope of the concatenation tends to x/4 (exactly 5/21 for 0,1,2,3,0,1,2,3 at unit steps, and 0.2499992 for two blocks of 0…499). Jonathan's counter-example partitions a series where Karl's concatenates two, and for a genuinely intensive quantity those operations are inverses of one another -- that they disagree here seems to me to settle the question, rather than to leave it a matter of which example is better chosen.

I would suggest that the property actually at stake is not intensivity but aggregability. For mean, sum, maximum and variance, a user wanting coarser cells can recompute the statistic from the finer-cell values together with the cell measures. A regression trend is non-aggregable: no combination of finer-cell trends and cell measures reproduces it, and it must be refitted from the source data. This is the operationally consequential fact -- it tells software what it may and may not do when collapsing or regridding -- and it is what neither point nor intensive conveys. It also explains the intuition behind both positions in the discussion, since aggregability and intensivity coincide for the established CF cell methods and part company here.

Whatever keyword is adopted, the estimator would need recording as well, since there are many alternatives for calculating a trend -- OLS, non-parametric (e.g. Theil–Sen), and many more, that give materially different values for the same cell; and it may be worth noting that a trend is only well defined along an ordered coordinate, so time: trend makes sense where area: trend would not.

Best wishes,
Lars

[JonathanGregory]

Dear Karl, Lars, Rob et al.

In view of your discomfort, I agree that intensive and extensive are not suitable, so I withdraw that suggestion. They aren't methods, as Lars says.

Your comments make me realise that we're using those words in a more general way than the physics and chemisty definitions involving properties of matter. Can we think of better words to use to explain the idea? I think the distinction is well defined: If a quantity is "extensive" with respect to a dimension, it must tend to zero as the size of the cell does (in that dimension), whereas if it is "intensive" a non-zero value is meaningful for a cell of zero extent. Temperature and air pressure are intensive in time, so their default cell method is point, meaning an instantaneous value, like an thermometer or barometer reading, whereas precipitation amount is extensive, and the default is sum, like a rain gauge reading.

A tendency, interpreted as a time-derivative, is an intensive value in time, in this sense. A linear trend over time is an estimate of a tendency. A trend is mostly useful as a statistic when it's reasonable to regard the variable as being the sum of an underlying constant rate of change (= the tendency), some random noise and maybe some periodic variation. If this is true, and you know the time-derivative at a sufficient temporal resolution to resolve all the random wiggles, over a long enough time to cover many complete periods of variation, the time-mean of the tendency over the finer cells would be a good estimate of the underlying tendency. By that argument, it would be correct to put time: mean over years in Rob's case, where the tendency is the rate of change over years. Although that's not as surprising as point, it might be confusing. What do you think?

One reason for its not being obvious is that in practice we don't have data of such length and high temporal resolution that it can be done that way. Instead we take time-means over intervals to reduce the noise and then calculate the trend. I appreciate Lars's point that a trend is not "aggregatable", meaning you can't get it from the finer cells i.e. the annual means alone. I think the reason is that the linear trend of X is not a statistic of X alone; it's a multivariate statistic that depends on the simultaneous variation of X and t; the annual values of X aren't enough to calculate it because you need to know what years (t) they come from.

"Trend" couldn't be a cell method because it isn't a statistic solely of the variation of the data variable within the cells, like all the others are. To compute dX/dt you need t as well as X. I think that's the fundamental reason why dX/dt must have a different standard name from X, rather being regarded as a statistic of X, and it's consistent with a related reason, that its units can't be derived from the units of X alone. When we use a trend to estimate a tendency, perhaps the tendency can't be regarded as intensive in time, because you can't calculate it for intervals of zero extent. Equally, it doesn't tend to zero as the number of years does from which it is calculated; it just gets less accurate.

If that is correct, maybe we need some generic cell method keyword to indicate that it depends on the variation of X and the extent of the cell, but it's not extensive and can't be evaluated at a point within the cell. Perhaps the word distribution would be a reasonable keyword, in a mathematical sense, which I don't understand properly? Then Rob would put time: distribution over years.

There's another problem which I had overlooked, regarding "within years". It's natural to describe Rob's use-case as a climatological statistic: first you mean within years e.g. monthly means, then calculate the trend over years. However, the quantity meaned within years is not the tendency_of_X, but X itself. Therefore it seems wrong to describe the first operation by a cell method of the same variable. Is there a neat way out of this problem, or am I hallucinating?

I guess we could make some exception for distribution over, that the within method describes a precursor quantity, in some way that relates to the definition of the standard name. In Rob's case, we'd say, in the description of the standard name tendency_ofX, that if a quantity with this standard name has cell methods time-dimension: method1 within years time-dimension: distribution over years it means that the tendency was computed over years after applying method1 to X (not its tendency) within years.

I agree with Lars that it may be useful or necessary to record in more detail how the trend was computed. Since this operation creates a new variable, with a new standard name, I think this extra information should be stored in some other attrbute than the cell_methods, such as the long_name or the history.

Best wishes

Jonathan