[BUG/ISSUE] HISTORY diagnostics only saves 23 hours of data to diagnostic files (except for Budget)

[ktravis213]

Hi support team,
I am running version 12.6.3 at 0.25x0.3125 resolution for the China grid. I need hourly time series, which I output with the HISTORY.rc diagnostic. I can only run two days at a time at this resolution on my computing system. For some reason, the budget diagnostic outputs all 24 hours in a day to the same file, but the other diagnostics (species concentration, prod/loss) output 23 hours and put the 24th hour in a new file. I am not sure which is correct and how to fix this.
Thank you!

[lizziel]

Hi Katie, could you provide your HISTORY.rc file? You can drag and drop it into a comment box if you add extension .txt. Thanks!

[yantosca]

Hi Katie, thanks for bringing this to our attention. @msulprizio and I just looked at some timeseries output that she had handy for another project.

Long story short: What happens is that at 0Z on the last day of your run, a file is created with one timestamp (e.g. 2018-07-01, 00Z). But then when you start the next run stage in your, the existing file gets overwritten with data from e.g. 2018-07-01 00 to 2018-07-01 23z.

We think the solution may be simple. In 12.8.1, we can add a test to see if the file that is trying to be created already exists, and if it does, to use netCDF append mode instead of netCDF write mode.

In the meantime, there are a couple of things you can do as a workaround.

1. Add a line into your run script to rename the diagnostic files that are created at the last timestep of your run, e.g.

mv GEOSChem.SpeciesConc.20180701_0000z.nc4 GEOSChem.SpeciesConc.20180701_0000z.1.nc4

then you can combine those files with the new files that are created at the end of the month.

2. Or even better, at the end of the run, rename the OutputDir folder, and then create a new blank OutputDir for the next run. This will avoid having files from a previous run being overwritten. Then you can either merge the files with nco or cdo, or read them into Python with xr.open_mfdataset,

Anyway, thanks again and stay tuned for the fix!

[yantosca]

Also -- to answer your question about why the budget diagnostics were not affected, that is probably because you archived the budget diagnostics as time-averaged. The bug above only affects instantaneous collections.

[yantosca]

I have pushed commit 28109a7, which is now currently on the bugfix/NcWrite branch. This will go into 12.8.1 once that version is ready (but you can check out this update if you need it).

I did a couple of test runs where I ran the transport tracer simulation for 1 day at a time and then saved out instantaneous 3-hourly output. I then checked to see if the file for the 2nd day had the first timestep (which was written out at the end of the previous day's run) preserved. Indeed it is!

netcdf GEOSChem.SpeciesConc.20160102_0000z {
dimensions:
        time = UNLIMITED ; // (8 currently)
        lev = 72 ;
        ilev = 73 ;
. . .
data:

 time = "2016-01-02", "2016-01-02 03", "2016-01-02 06", "2016-01-02 09", 
    "2016-01-02 12", "2016-01-02 15", "2016-01-02 18", "2016-01-02 21" ;

Without the fix, the list of timestamps would have started with 2016-01-02 03 (for 3-hourly output) instead of 2016-01-02 00, and there would have been 7 timesteps instead of the correct value of 8.

[yantosca]

Commit 28109a7 has now been merged up to version 12.8.1

[yantosca]

It was discovered that the prior commit 28109a7 had a bug that was caused by a missing Container%Set_Inst = .FALSE. statement. This error was causing instantaneous files to always use the reference time at the start of the simulation. This would try to append to existing files that should have been appended to (e..g the initial restart file).

This behavior is now corrected in commit 8003e32.

NOTE: We may add a further update to decide which instantaneous files should be appended to and which should not.

[yantosca]

Also note, a further commit 1a160e1 was made in order to make sure that instantaneous collections where frequency=duration are written to disk with the proper reference time.

Also, we removed "UTC" from the time:units" string, as this sometimes causes ncdump -t not to display the proper time values.

[yantosca]

Also added further commit 81c7126, which adds extra updates so that instantaneous collections with multiple time points are now archived properly.

Also, we now throw an error if we try to append to an instantaneous netCDF file with more than one time point. This is now in 12.8.1.

[yantosca]

After further consideration, we have decided to remove the need for appending to instantaneous file collection files due to the danger of clobbering data. This was updated in commit 94d78a7.

Instantaneous collection with more than one time point per file

• First file timestamp: 1st day of simulation @ 01:00Z
• Llast file timestamp: last day of simulation @ 00:00Z.
• No appending needed.

For example, a 2-day run with hourly archiving saved to daily files creates this output:

GEOSChem.SpeciesConc.20160101_0010z.nc4
GEOSChem.SpeciesConc.20160102_0000z.nc4
GEOSChem.SpeciesConc.20160102_0010z.nc4
GEOSChem.SpeciesConc.20160103_0000z.nc4

Instantaneous collection with only one time point per file

• The file name uses the date/time when the data is written to disk.
• No appending needed.

For example, a multi-day run with hourly archiving saved to hourly files now creates this output:

GEOSChem.SpeciesConc.20160101_0100z.nc4
GEOSChem.SpeciesConc.20160101_0200z.nc4
GEOSChem.SpeciesConc.20160101_0300z.nc4
...
GEOSChem.SpeciesConc.20160101_2000z.nc4
GEOSChem.SpeciesConc.20160101_2100z.nc4
GEOSChem.SpeciesConc.20160101_2200z.nc4
GEOSChem.SpeciesConc.20160101_2300z.nc4
GEOSChem.SpeciesConc.20160102_0000z.nc4
GEOSChem.SpeciesConc.20160102_0100z.nc4
... etc ...