Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix several bugs in GFSv16 forecast-only cold-start mode #1177

Closed
KateFriedman-NOAA opened this issue Dec 13, 2022 · 2 comments · Fixed by #1208
Closed

Fix several bugs in GFSv16 forecast-only cold-start mode #1177

KateFriedman-NOAA opened this issue Dec 13, 2022 · 2 comments · Fixed by #1208
Assignees
Labels
bug Something isn't working

Comments

@KateFriedman-NOAA
Copy link
Member

User reported bug in GFSv16 forecast-only cold-start mode.

Expected behavior

The f000 output from the forecast job should be produced in forecast-only mode when running with cold-start ICs (a.k.a. lower resolution).

Current behavior

When running forecast-only mode with cold-start ICs the f000 files are not produced by the forecast model in cycles after the initial cycle.

Machines affected

All.

To Reproduce

Run lower resolution / cold-start forecast-only mode for more than one cycle.

Detailed Description

Upon investigation it is an issue with the IAU_OFFSET variable being mismatched with the DOIAU setting. The DOIAU variable will be set to no in the forecast job when the forecast script does not see prior cycle warm-start inputs but the IAU_OFFSET variable does not get set to 0 to match. With IAU_OFFSET=3 the forecast model doesn't start producing output until 3hr (or 6hr if using 6hr frequency). This causes a ripple effect in the post, postsnd, and archive jobs. The post f000 files aren't produced and thus the postsnd and archive jobs fail.

Possible Implementation

Need to check for forecast-only mode and cold-start earlier in the job so IAU_OFFSET=0 can be set and passed for the forecast model. The following if-block in config.base[.emc.dyn] can be expanded to handle this for now:

if [[ "$SDATE" = "$CDATE" && $EXP_WARM_START = ".false." ]] || [[ "$DOIAU" = "NO" ]] ; then # Cold starting or IAU off

...should be changed to:

if [[ "$SDATE" = "$CDATE" && $EXP_WARM_START = ".false." ]] || [[ "$DOIAU" = "NO" ]] || [[ "$MODE" = "forecast-only" && $EXP_WARM_START = ".false." ]] ; then # Cold starting or IAU off

Looking at the system, the MODE variable was added to the setup scripts but didn't make it into the config.

Add export MODE="@MODE@" into config.base.emc.dyn for use by updated if-block conditions (shown above). Also change "free" to "forecast-only" in setup_expt_fcstonly.py to be consistent with develop branch implementation. The develop branch does not have this issue since it has the expanded check in config.base.

Will correct, test, and commit into the dev/gfs.v16 branch.

@KateFriedman-NOAA KateFriedman-NOAA added the bug Something isn't working label Dec 13, 2022
@KateFriedman-NOAA KateFriedman-NOAA self-assigned this Dec 13, 2022
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Dec 13, 2022
- Change "free" to "forecast-only" for MODE setting in
setup_expt_fcstonly.py.

Refs NOAA-EMC#1177
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Dec 13, 2022
In config.base.emc.dyn (only developer version):
- add MODE setting (set from setup scripts)
- update IAU if-block to also check for forecast-only and cold-start
modes to turn off IAU (corrects bug with f000)

Refs NOAA-EMC#1177
@KateFriedman-NOAA
Copy link
Member Author

Testing of GFSv16 forecast-only mode on WCOSS2 with UFS-UTILS-generated C192 cold-start ICs and bug fix now produces f000 outputs for cycles after the initial one:
First (SCDATE) cycle:

kate.friedman@dlogin05:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmodeff> ls -ltr gfs.20220102/00/atmos/*f000*
-rw-r--r-- 1 kate.friedman emc 227464587 Dec 13 19:26 gfs.20220102/00/atmos/gfs.t00z.master.grb2f000
-rw-r--r-- 1 kate.friedman emc   9527281 Dec 13 19:26 gfs.20220102/00/atmos/gfs.t00z.sfluxgrbf000.grib2
-rw-r--r-- 1 kate.friedman emc 498964418 Dec 13 19:26 gfs.20220102/00/atmos/gfs.t00z.atmf000.nc
-rw-r--r-- 1 kate.friedman emc  71086554 Dec 13 19:26 gfs.20220102/00/atmos/gfs.t00z.sfcf000.nc
-rw-r--r-- 1 kate.friedman emc        71 Dec 13 19:26 gfs.20220102/00/atmos/gfs.t00z.logf000.txt
-rw-r--r-- 1 kate.friedman emc     31092 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2.1p00.f000.idx
-rw-r--r-- 1 kate.friedman emc  37750432 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2.1p00.f000
-rw-r--r-- 1 kate.friedman emc     31349 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2.0p50.f000.idx
-rw-r--r-- 1 kate.friedman emc 123744380 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2.0p50.f000
-rw-r--r-- 1 kate.friedman emc     31790 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2.0p25.f000.idx
-rw-r--r-- 1 kate.friedman emc 395450516 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2.0p25.f000
-rw-r--r-- 1 kate.friedman emc 176381279 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2b.0p25.f000
-rw-r--r-- 1 kate.friedman emc      2574 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.sfluxgrbf000.grib2.idx
-rw-r--r-- 1 kate.friedman emc     15987 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2b.1p00.f000.idx
-rw-r--r-- 1 kate.friedman emc  16814218 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2b.1p00.f000
-rw-r--r-- 1 kate.friedman emc     16153 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2b.0p50.f000.idx
-rw-r--r-- 1 kate.friedman emc  55202182 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2b.0p50.f000
-rw-r--r-- 1 kate.friedman emc     16339 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.pgrb2b.0p25.f000.idx
-rw-r--r-- 1 kate.friedman emc    235908 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.master.grb2if000
-rw-r--r-- 1 kate.friedman emc      1066 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.special.grb2if000
-rw-r--r-- 1 kate.friedman emc   1117276 Dec 13 19:38 gfs.20220102/00/atmos/gfs.t00z.special.grb2f000

Next cycle:

kate.friedman@dlogin05:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmodeff> ls -ltr gfs.20220102/06/atmos/*f000*                                                             
-rw-r--r-- 1 kate.friedman emc   9454719 Dec 13 19:38 gfs.20220102/06/atmos/gfs.t06z.sfluxgrbf000.grib2
-rw-r--r-- 1 kate.friedman emc 228758836 Dec 13 19:38 gfs.20220102/06/atmos/gfs.t06z.master.grb2f000
-rw-r--r-- 1 kate.friedman emc 499264009 Dec 13 19:38 gfs.20220102/06/atmos/gfs.t06z.atmf000.nc
-rw-r--r-- 1 kate.friedman emc  70988694 Dec 13 19:38 gfs.20220102/06/atmos/gfs.t06z.sfcf000.nc
-rw-r--r-- 1 kate.friedman emc        71 Dec 13 19:38 gfs.20220102/06/atmos/gfs.t06z.logf000.txt
-rw-r--r-- 1 kate.friedman emc     31092 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2.1p00.f000.idx
-rw-r--r-- 1 kate.friedman emc  37926414 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2.1p00.f000
-rw-r--r-- 1 kate.friedman emc     31355 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2.0p50.f000.idx
-rw-r--r-- 1 kate.friedman emc 124429847 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2.0p50.f000
-rw-r--r-- 1 kate.friedman emc     31790 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2.0p25.f000.idx
-rw-r--r-- 1 kate.friedman emc 397911195 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2.0p25.f000
-rw-r--r-- 1 kate.friedman emc      2573 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.sfluxgrbf000.grib2.idx
-rw-r--r-- 1 kate.friedman emc     15986 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2b.1p00.f000.idx
-rw-r--r-- 1 kate.friedman emc  16864551 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2b.1p00.f000
-rw-r--r-- 1 kate.friedman emc     16153 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2b.0p50.f000.idx
-rw-r--r-- 1 kate.friedman emc  55379190 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2b.0p50.f000
-rw-r--r-- 1 kate.friedman emc     16337 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2b.0p25.f000.idx
-rw-r--r-- 1 kate.friedman emc 177187520 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.pgrb2b.0p25.f000
-rw-r--r-- 1 kate.friedman emc    235880 Dec 13 19:45 gfs.20220102/06/atmos/gfs.t06z.master.grb2if000
-rw-r--r-- 1 kate.friedman emc      1066 Dec 13 19:46 gfs.20220102/06/atmos/gfs.t06z.special.grb2if000
-rw-r--r-- 1 kate.friedman emc   1121767 Dec 13 19:46 gfs.20220102/06/atmos/gfs.t06z.special.grb2f000

@KateFriedman-NOAA KateFriedman-NOAA changed the title Missing f000 outputs in GFSv16 forecast-only cold-start mode Fix f000 and archive bugs in GFSv16 forecast-only cold-start mode Jan 4, 2023
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jan 4, 2023
- In hpssarch_gen.sh add an if-block to check if cycled MODE
for the pgb*anl files.

Refs NOAA-EMC#1177
@KateFriedman-NOAA KateFriedman-NOAA changed the title Fix f000 and archive bugs in GFSv16 forecast-only cold-start mode Fix several bugs in GFSv16 forecast-only cold-start mode Jan 4, 2023
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jan 4, 2023
- Replaced tabs with spaces.

Refs NOAA-EMC#1177
@KateFriedman-NOAA KateFriedman-NOAA linked a pull request Jan 5, 2023 that will close this issue
9 tasks
@KateFriedman-NOAA
Copy link
Member Author

Resolved by #1208.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant