NNODES_RUN_FCST in task_run_fcst

[SarahLu-NOAA]

Hi, UFS/SRW experts -

I created a new grid (12 km over NE US) at Derecho with SRW v2.2 public release. The new grid is specified in task_make_grid section in config.yaml. I also specify WRTCMP related parameters in task_run_fcst section. The SRW experiments with the new grid went well. I then add this new grid to PREDEF_GRID_NAME by updating valid_param_vals.yaml and predef_grid_params.yaml. The parameters removed from config.yaml and added to predef_grid_params.yaml are listed at the end of the msg. The experiment is halted at the fcst step. FV3LAM_wflow.log shows:

Submission of run_fcst_mem000 failed! qsub: directive error: -l select={{ task_run_fcst.NNODES_RUN_FCST // 1 }}:mpiprocs=128:ncpus=128

var_defns.sh in the failed SRW run shows
NNODES_RUN_FCST='{{ (PE_MEMBER01 + PPN_RUN_FCST - 1) // PPN_RUN_FCST }}'
var_defns.sh in the previous SRW run (when new grid is specified in config.yaml) shows
NNODES_RUN_FCST='1'

My quick fix is to hardcode NNODES_RUN_FCST to 1 in config.yaml and the SRW exp now runs ok.
My question is: why these changes related to new grid affects how NNODES_RUN_FCST is setup in the workflow?

Thanks -Sarah

GRID_GEN_METHOD: ESGgrid
ESGgrid_LON_CTR: -78.
ESGgrid_LAT_CTR: 44.
ESGgrid_DELX: 12000.0
ESGgrid_DELY: 12000.0
ESGgrid_NX: 200
ESGgrid_NY: 140
ESGgrid_WIDE_HALO_WIDTH: 6
ESGgrid_PAZI: 0.0
DT_ATMOS: 60
LAYOUT_X: 4
LAYOUT_Y: 4
BLOCKSIZE: 32
QUILTING:
WRTCMP_write_groups: 1
WRTCMP_write_tasks_per_group: '{{ LAYOUT_Y }}'
WRTCMP_output_grid: lambert_conformal
WRTCMP_cen_lon: '{{ task_make_grid.ESGgrid_LON_CTR }}'
WRTCMP_cen_lat: '{{ task_make_grid.ESGgrid_LAT_CTR }}'
WRTCMP_lon_lwr_left: -91.6565
WRTCMP_lat_lwr_left: 35.8696
WRTCMP_stdlat1: '{{ task_make_grid.ESGgrid_LAT_CTR }}'
WRTCMP_stdlat2: '{{ task_make_grid.ESGgrid_LAT_CTR }}'
WRTCMP_nx: 198
WRTCMP_ny: 138
WRTCMP_dx: 12000.0
WRTCMP_dy: 12000.0

[gspetro-NOAA]

Hi @SarahLu-NOAA ,

Just wanted to let you know that I am working on getting a subject matter expert (SME) to assist with your question. Hopefully we can get you an answer by early next week, but in the meantime, I didn't want to leave you hanging.

Best,
Gillian Petro

[MichaelLueken]

Hello @SarahLu-NOAA -

I was able to replicate what you are seeing with your new grid, but using the RRFS_CONUS_25km grid instead. I set this grid as follows:

"RRFS_CONUS_25km":
  GRID_GEN_METHOD: "ESGgrid"
  ESGgrid_LON_CTR: -97.5
  ESGgrid_LAT_CTR: 38.5
  ESGgrid_DELX: 25000.0
  ESGgrid_DELY: 25000.0
  ESGgrid_NX: 219
  ESGgrid_NY: 131
  ESGgrid_PAZI: 0.0
  ESGgrid_WIDE_HALO_WIDTH: 6
  DT_ATMOS: 150
  LAYOUT_X: 5
  LAYOUT_Y: 2
  BLOCKSIZE: 40
  QUILTING:
    WRTCMP_write_groups: 1
    WRTCMP_write_tasks_per_group: '{{ LAYOUT_Y }}'
    WRTCMP_output_grid: "lambert_conformal"
    WRTCMP_cen_lon: '{{ task_make_grid.ESGgrid_LON_CTR }}'
    WRTCMP_cen_lat: '{{ task_make_grid.ESGgrid_LAT_CTR }}'
    WRTCMP_stdlat1: '{{ task_make_grid.ESGgrid_LAT_CTR }}'
    WRTCMP_stdlat2: '{{ task_make_grid.ESGgrid_LAT_CTR }}'
    WRTCMP_nx: 217
    WRTCMP_ny: 128
    WRTCMP_lon_lwr_left: -122.719528
    WRTCMP_lat_lwr_left: 21.138123
    WRTCMP_dx: 25000.0
    WRTCMP_dy: 25000.0

Upon doing so, the var_defns.sh file is showing:

NNODES_RUN_FCST='{{ (PE_MEMBER01 + PPN_RUN_FCST - 1) // PPN_RUN_FCST }}'

and the test failed with qsub: directive error: -l select={{ task_run_fcst.NNODES_RUN_FCST // 1 }}:mpiprocs=128:ncpus=128.

I was able to correct this behavior by setting WRTCMP_write_tasks_per_group: 2.

In order to correct this, you will need to set WRTCMP_write_tasks_per_group: 4 in your ush/predef_grid_params.yaml file.

Please let me know if you continue to encounter this issue even with this modification.

[SarahLu-NOAA]

Thanks, @MichaelLueken
Are you suggesting me to hard code WRTCMP_write_tasks_per_group to 4 (it was set to '{{ LAYOUT_Y }}')?
My SRW jobs at Derecho are in the queue for 12+ hours. Not sure when the exp with your suggested fix will be ran.
I'm still puzzled why moving these grid-related and quilting-related parameters from config.yaml to predef_grid_params.yaml will affect NNODES_RUN_FCST.

[MichaelLueken]

Hi @SarahLu-NOAA,

Yes, you will need to hard code WRTCMP_write_tasks_per_group to 4 in ush/predef_grid_params.yaml. Doing so should let your experiment run.

I've been seeing long queue times on Derecho this week as well. It could be due to final updates to AMS presentations, so hopefully the queue times will alleviate at the end of the month (though, that doesn't help in the interim).

The issue is that the ush/predef_grid_params.yaml file is expecting hard coded values for the listed parameters, not YAML {{...}} entries. Now, I can't speak on why this is the case, but NNODES_RUN_FCST is affected by PE_MEMBER01, which itself is affected by WRTCMP_write_tasks_per_group, and if this parameter is set using {{ LAYOUT_Y }}, then the value of PE_MEMBER01 is never calculated, leading to a bad NNODES_RUN_FCST.

[SarahLu-NOAA]

@MichaelLueken , it's interesting to learn how WRTCMP_write_tasks_per_group affects PE_MEMBER01 and thus NNODES_RUN_FCST. Your suggested fix is implemented and I just need to wait for the queue to be cleared to check the outcome. Many thanks.

@gspetro-NOAA and @MichaelLueken
Before I close this string, I have a question about scientific support. I have one SRW v2.2 run using the NE US 12km grid. The IC and LBC are from GFS, and it's 84 hour simulation init from 2023-06-01 00z. I set CCPP_PHYS_SUITE to FV3_GFS_v17_p8. The experiment is completed without any issue but the results are not realistic (the entire NE US become super cold). I understand that GFSv17_p8 is not supported by EPIC and the namelist variables are likely not set up properly. How do I sort this out ? Is it appropriate to post this at Discussions? Thanks.

[gspetro-NOAA]

Hi @SarahLu-NOAA ,

It's fine to post as a separate Discussion, but you might get more physics folks with eyes on it (and the appropriate expertise) if you post in the CCPP Discussions. @mkavulich do you have a suggestion for what might be best here? Should she post in the ccpp-physics discussions?

[mkavulich]

I agree the CCPP-physics discussions are probably best for this question. I do know that only limited testing was done with the FV3_GFS_v17_p8 suite for this release (no scientific testing that I am aware of). Without knowing more about the results, it may be just be bad results or bad namelist settings, but it could also be a configuration failure (wrong files being staged by the workflow) or a legitimate bug.

A side note related to the original question in this discussion: this should probably be marked as a bug and opened as an issue in the SRW repo: the workflow should be failing at the generation step if these jinja templates aren't properly getting filled in with required variables, it shouldn't be continuing to the run step.

[SarahLu-NOAA]

Thanks, @mkavulich and @gspetro-NOAA for your input.
I could post my question on how to configure FV3_GFS_v17_p8 at ccpp-physics discussions. The UFS website indicates that GFS_v17_p8 is not supported for this release. Can I post an issue at the UFS repo to report my results and seek help on workflow configuration?

[gspetro-NOAA]

@mkavulich @MichaelLueken FYI I added an Issue #1006 to document the lack of failure on experiment generation.