Ocean *_write_on_startup = .True. option produces faulty xtime for first output

[pwolfram]

The first time step output using *_write_on_startup = .True. for the model and analysis members produces a faulty xtime entry of 0000-01-15_00:00:00. This is true for mpaso.hist.0001*nc as well as for AM output, e.g., mpaso.hist.am.lagrPartTrack.*nc.

A comparison of source code for the E3SM MPAS-O driver vs MPAS-O forward mode suggests that the mpas clock may not be initialized properly.

In E3SM mpas-ocean/driver/ocn_comp_mct.F:

849       ! Set MPAS Log module instance
 850       mpas_log_info => domain % logInfo
 851       if (debugOn) call mpas_log_write("=== Beginning ocn_run_mct ===")
 852 
 853       call mpas_pool_get_config(domain_ptr % configs, 'config_restart_timestamp_name', config_restart_timestamp_name)
 854       call mpas_pool_get_config(domain_ptr % configs, 'config_sw_absorption_type', config_sw_absorption_type)
 855 
 856       ! Setup log information.
 857       call shr_file_getLogUnit (shrlogunit)
 858       call shr_file_getLogLevel(shrloglev)
 859       call shr_file_setLogUnit (ocnLogUnit)
 860 
 861       timeStep = mpas_get_clock_timestep(domain_ptr % clock, ierr=ierr) 
 862       call mpas_get_timeInterval(timeStep, dt=dt)
 863       call mpas_reset_clock_alarm(domain_ptr % clock, coupleAlarmID, ierr=ierr)
 864 
 865       ! Import state from coupler
 866       call ocn_import_mct(x2o_o, ierr)
 867 
 868       ! Ensures MPAS AM write/compute startup steps are performed
 869       call ocn_analysis_compute_startup(domain_ptr, ierr)    
 870 
 871       ! Handle writing initial state
 872       call mpas_pool_get_config(domain_ptr % configs, 'config_write_output_on_startup', config_write_output_on_startup)
 873       if (config_write_output_on_startup) then
 874          call mpas_stream_mgr_write(domain_ptr % streamManager, 'output', forceWriteNow=.true., ierr=ierr)
 875 
 876          ! Reset config to false, so we don't write the state every coupling interval.
 877          config_write_output_on_startup = .false.
 878       end if 

In MPAS-O src/core_ocean/mode_forward/mpas_ocn_forward_mode.F:

499       ! Eventually, dt should be domain specific
500       timeStep = mpas_get_clock_timestep(domain % clock, ierr=ierr)
501       call mpas_get_timeInterval(timeStep, dt=dt)
502 
503       currTime = mpas_get_clock_time(domain % clock, MPAS_NOW, ierr)
504       call mpas_get_time(curr_time=currTime, dateTimeString=timeStamp, ierr=ierr)
505 #ifdef MPAS_DEBUG
506       call mpas_log_write('Initial time ' // trim(timeStamp))
507 #endif
508 
509       call ocn_analysis_compute_startup(domain, err)
510 
511       if (config_write_output_on_startup) then
512           call mpas_timer_start('io_write')
513           call mpas_stream_mgr_write(domain % streamManager, 'output', forceWriteNow=.true., ierr=ierr)
514           call mpas_timer_stop('io_write')
515       endif

In particular, lines like

503       currTime = mpas_get_clock_time(domain % clock, MPAS_NOW, ierr)
504       call mpas_get_time(curr_time=currTime, dateTimeString=timeStamp, ierr=ierr)

may be missing from the E3SM MPAS-O driver code.

cc @maltrud, @jonbob, @bradyrx

[vanroekel]

@pwolfram I don't think this is a bug in the E3SM driver. This is a requirement due to the coupler sequencing. In all E3SM simulations, the ocean is not run until after the first coupling interval and only the clock is advanced a coupling timestep (ocean). So I don't think this is a bug, but @jonbob should confirm.

[jonbob]

@pwolfram - can you provide details about the run you saw this in? @vanroekel is correct that this is known to have problems in coupled runs, which is why it is off by default in E3SM. But xtime should not be off by 15 days, more like a coupling timestep. Can you point me to your run and output?

[bradyrx]

@jonbob -- An example run is here:

/lustre/scratch3/turquoise/rileybrady/ACME/cases/GMPAS-IAF.T62_oEC60to30v3.physicalSensors03/run

This is a 60to30 G-case with physics only. It was run for a full month without particles, and then particles were turned on starting at Feb 1. See the xtime output on mpaso.hist.am.lagrPartTrack.0001-02-01_00.00.00.nc where it injects Jan. 15th as the first date with write_on_startup as true.

This is common across multiple cases, but most apparent with the February startup. I haven't tried doing write_on_startup for any other output, but we assume this is a common issue across output and not just for particles.

[pwolfram]

Thank you @vanroekel, @jonbob. I should note that for the case at /lustre/scratch3/turquoise/rileybrady/ACME/cases/GMPAS-IAF.T62_oEC60to30v3.physicalSensors03/run that mpaso.hist.0001-02-01_00000.nc has the same issue with an xtime of 0000-01-15_00:00:00 for config_write_output_on_startup = .true..

The fundamental issue is that the initialization of the MPAS-O clock in E3SM does not appear to be correct. @bradyrx observed that the string of 0000-01-15_00:00:00 appears to be output for a variety of restart times, so I'm guessing this initial value is something that is arbitrarily assumed and not updated as it should be once E3SM is initialized.

[pwolfram]

This issue came up again today-- just a reminder to everyone that we shouldn't forget this issue.

[xylar]

I tried grepping for "0000-01-15" in both the MPAS and E3SM codes (since this seems unlikely to come out of come calculation -- we start with year 0001 and a time average over January should give us 01-15_12:00:00 as the mean time.

E3SM didn't turn up anything. What I could find were:
https://github.com/MPAS-Dev/MPAS-Model/blob/master/src/core_ocean/shared/mpas_ocn_tracer_surface_restoring.F#L264-L265
and again in BGC:
https://github.com/MPAS-Dev/MPAS-Model/blob/master/src/core_ocean/shared/mpas_ocn_tracer_ecosys.F#L1765-L1766

It's unclear to me why these dates are hard coded (perhaps inherited from MPAS-Seaice?) or why this date was chosen. It's even more unclear to me why this date would influence the time written out with config_write_output_on_startup = .true.. But I thought I'd share this just in case it helps someone figure out what's going wrong.

[xylar]

@mark-petersen, this is the issue I was referring to today that I think might also be related to alarms not working as expected on startup (either from init or from restart).