Bug in coszen time interpolation in shr code

[alperaltuntas]

When running with the JRA data streams, the coszen time interpolation algorithm in shr code appears to have a bug resulting in large flux values being passed from the coupler to components. We discovered this issue when running the G compset (both w/ POP and MOM6) and JRA-forced data atm. Note that the JRA SWDN datastream makes use of coszen time interpolation algorithm with an offset of 5400 sec, i.e., 1.5 hours. This issue doesn't occur with the CORE2 data streams, probably because CORE2 is daily average, whereas JRA is 3-hourly.

The following is the x2oacc_Foxx_swnet field passed from the coupler to the ocean component at the very first coupling timestep with a band of large values (with a max of ~6k W m-2)

My initial troubleshooting efforts suggest this has to to with the time averaging of the coszen term used in normalization. See the relevant line here:

cime/src/share/streams/shr_strdata_mod.F90

Line 1067 in e1ae39f

SDAT%avs(n)%rAttr(:,i) = SDAT%avFLB(n)%rAttr(:,i)*cosz(i)/tavCosz(i)

In the above line, cosz(i)/tavCosz(i) term leads to these large values because, i think, there is a discrepancy between how the cosz and tavCosz terms are computed. The time value passed to the subroutine that computes cosz and the time values passed to the subroutine that computes tavCosz don't seem to agree. But I'll let someone who is more familiar with this part of the code further diagnose and come up with a fix.

One final comment that further puzzles me: When the offset is -5400, as it should be, this issue occurs only at the first two timesteps. When the offset is 0, however, the issue occurs routinely at every third timestep.

To reproduce the issue:

• check out CESM 2.2.1
• run out-of-the-box GIAF_JRA: create_newcase --res TL319_g17 --compset GIAF_JRA --case g.e22.GJRA.TL319_g17.sw.001 --run-unsupported

[rljacob]

Alerting @jonbob

[jonbob]

Thanks @rljacob - I suspect that hits us as well

[jedwards4b]

@ekluzek I don't see an obvious cause for the problem here - you are one of the authors of this code - can you have a look?

[jedwards4b]

BTW test SMS_D.TL319_g17.GIAF_JRA.cheyenne_intel will reproduce the issue.

[ekluzek]

So offset is zero for all of the CLM forcing files. From @alperaltuntas caution this is especially disconcerting. Looks like CPLHIST files should be fine, and only CLM forcing and CORE forcing that are incorrect.

[swensosc]

is this related to #3380?

[ekluzek]

@alperaltuntas confirms that the fix in ESCOMP/CDEPS#123 for #3380 fixes this issue.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[rljacob]

Fixed for E3SM in E3SM-Project/E3SM#4589
Fixed for CESM in ESCOMP/CDEPS#123