Error make_lbcs and make_ics when using a custom grid

[rgsmith27]

I'm running the SRW on MSU-HPC and I'm attempting to use a custom 13km subconus grid. When the model gets to the make_ics and make_lbcs steps, each crash with an error. The error text from the TMP files is as follows:

PET30.ESMF_LogFile:20250703 094609.534 ERROR PET30 ESMF_FieldRegrid.F90:4442 checkGrid Invalid argument - some types of regridding (e.g. bilinear) are not supported on Grids that contain a DE of width 1.
PET30.ESMF_LogFile:20250703 094609.534 ERROR PET30 ESMF_FieldRegrid.F90:3229 b_or_p_GridToMesh Invalid argument - Internal subroutine call returned Error
PET30.ESMF_LogFile:20250703 094609.534 ERROR PET30 ESMF_FieldRegrid.F90:1387 getMeshWithNodesOnFieldLoc Invalid argument - Internal subroutine call returned Error
PET30.ESMF_LogFile:20250703 094609.534 ERROR PET30 ESMF_FieldRegrid.F90:1011 ESMF_FieldRegridStoreNX Invalid argument - Internal subroutine call returned Error
PET30.ESMF_LogFile:20250703 094609.534 ERROR PET30 atmosphere.F90:382 Invalid argument - Passing error in return code

From what I've seen online, this issue is caused by the model attempting to divide the grid among too many tasks, leading to it attempting to divide the grid to less than one grid cell.

The grid size I'm using is 140 by 70, and I'm using a 2 by 1 layout size. I have set the write groups and write tasks per group each to one in attempt to solve this error. Is there a way to further reduce the division of the grid? Is my grid simply too small? Help would be appreciated.

[gspetro-NOAA]

Hi @rgsmith27 !

A lot of folks are out today ahead of the 4th of July holiday, so there might be a delay getting you help. That said, it would be helpful to see your custom grid parameters so that we can give you more concrete guidance. And when you talk about layout size, which variables specifically are you talking about? One thing you might try is adjusting the LAYOUT_X and LAYOUT_Y variables if you haven't yet. Can't guarantee that it's the issue here, but it's something to try while you're waiting for others to respond. There is some documentation on the computational parameters here if you haven't seen it already, and you can also look at values in predef_grid_params.yaml to get ideas from similar size or similar resolution grids.

Sorry I'm not of more help on this one. @mkavulich do you have any suggestions? If not, I think @MichaelLueken may have some suggestions when he gets back on Monday.

Best,
Gillian

[mkavulich]

If the problem is over-decomposition in chgres_cube, you'll need to reduce the number of MPI tasks allocated to the make_*cs tasks (which is independent from the tasks allocated for the run_fcst step). The built-in method for changing this depends on what version of SRW you're using, since this has changed a few times over the past year. In the current version (since #1204 was merged on April 15), adding the following section to your config.yaml file should suffice:

task_make_lbcs:
  chgres_cube:
    execution:
      batchargs:
        nodes: 1
        tasks_per_node: 12

You can also reduce tasks_per_node further but I assume this will be low enough.

For older versions of the code (such as the 3.0 release), it's a bit harder, so let me know if you need help with that. There's also manually modifying the FV3LAM_wflow.xml file, but I wouldn't recommend this unless you're comfortable with how Rocoto works.

[rgsmith27]

This issue was solved by increasing the grid size by a bit and playing around with the variables mentioned above.