climatology startYear and endYear

[xylar]

If someone tries to set startYear or endYear (more likely the latter) out of range of the available data, endYear should be updated to be the last year actually available. This is most important in the title of the plots and (presumably) the name of the plots.

This should be easy to fix using the actual start and end date of the data set once it's loaded in.

[xylar]

@mark-petersen pointed this out during today's ACME Ice-Ocean telecon.

[xylar]

Addressed by #150

[xylar]

#150 didn't fix this issue for the MOC calculation.

[xylar]

MOC is going to be tougher because we don't want to open the data set if there's already a climatology saved, but we don't know what the range will be for sure until we do. In general, we should think about what it means to cache a climatology that is incomplete compared with the range requested, and how to update it, similarly to how we update the time series.

[milenaveneziani]

But if someone tries to set yearStart, yearEnd out of the data range, then there shouldn't exist a previously cached climatology file, therefore it goes on opening the data set as it should and it will realize that the dates should be different. Maybe I am missing something.

[xylar]

But it prints the wrong bounds in in the figure title and uses the wrong bounds in the figure name.

[xylar]

So there are 2 scenarios. First, you run for the first time with endYear=50 but the simulation only got to year 2 so far (or you only copied results up to year 2, or whatever). It would be good if the figure said the end year was 2, not 50. It might also be good if the climatology file name had end year of 2, not 50 and the same for the plot files.

The second scenario is that you re-run the analysis. The simulation has now got to year 3. but you still have endYear=50. If we use 2 as the end year before but we check for an end year of 50, we find the file doesn't exist. If, instead, we used and end year of 50 in the file name, we find it already exists and load it. It's only valid for 2 years, but we have no way of knowing that. We also have no way of knowing what the real end year of the simulation was because we never loaded the data set. When we get to plotting the figures, what do we use as the end year in both the title and the file name?!

So a possible solution would be to have the climatology be cached in a smart way. It knows what the start and end dates were that were used to create it. Somehow, we check to see what the start and end dates would be if we actually load the data set. (This could be done by loading only the Time variable from the data set, but that could end up being just as costly as loading the data set. So we might just have to trust the file dates with a safety buffer, like usual.) If the dates of the climatology are different from those of the actual data set, we load the part of the data set we haven't already used and average it into the climatology (multiplying by the old time span, adding in the new fields, dividing by the new time span). Then we update the cached climatology and also store the real start and end dates of the data set, and we plot the results.

Can you think of a simpler alternative?

[xylar]

Okay, so a simpler approach would be to cache the real start and end dates and also the start and end file dates (which might be different) along with the climatology. The name of the cache file would use the intended climatology bounds, not the actual ones. If the start and/or end file dates are not right for the first and last file in the data set, we just recompute the climatology and re-cache it. This way, the plots have the real bounds in time (both in the title and the name of the file) but the cache file doesn't.

[milenaveneziani]

sorry, I didn't have time to think about this in the last hour. I will get back to you later today.

[milenaveneziani]

Hi @xylar. I guess I am confused because, looking at how, say, ocean_modelvsobs works now, this is what I think it would happen in the scenarios you envisioned:

1. say we have 2 years of data but we request climos for years 1-50. This is the first time around, no climo file has been cached, so we load the data using startDate=0001-01-01 and endDate=0050-12-31.
2. in ocean_modelvsobs (and the sea-ice correspondent) I see that now yearStart and yearEnd are re-set using ds.Time. So, for this example they would become 1 and 2, respectively, which means the climos are computed over those 2 years and the filenames and titles reflect this ‘change’ from the initial intention.
3. I think this is fine, as the initial intention was probably just a mistake or simply due to the fact that the user didn’t know there weren’t 50 years of data to compute the requested climos. So instead of an exit error, one gets some kind of result, a shorter climatology, and this should be reflected in the filename.

I really think there should be consistency between the filename years and the actual years over which the climo was computed.
If we do that, then there is no problem in the second scenario as well, when we now have 3 years of data and we still request climos over years 1-50. The climo file does not exist with those bounds, we re-load the data, the actual years are reset to 1-3, the climos are re-created and re-cached using those years.

Having said that, perhaps I misunderstood how things now work in modelvsobs.. Apologies if I did.