Failed on generating ICS and BCS for predefined grid in 3km

[htan2013]

Dear all experts,

I tried to generate ICs and BCs for a predefined grid (GSL_HAFSV0.A_3km). But the log files for ICs and BCs show that they didn't finish.
I know that if it is finished, it should show something like:

========================================================================
Initial condition, surface, and zeroth hour lateral boundary condition
files (in NetCDF format) for FV3 generated successfully!!!

But no such info was seen for Make_ICS_3km and Make_LBCS_3km. I have attached log files for Make_ICS_13km, Make_LBCS_13km, Make_ICS_3km, and Make_LBCS_3km. I have successfully finished the 13km simulation.

I am wondering what files I should check to fix this. Is this something related to processors number since the NX and NY increased for different grids?

Many thanks in advance,
Haochen

Make_ICS_13km.log
Make_LBCS_13km.log
Make_ICS_3km.log
Make_LBCS_3km.log

[mkavulich]

@htan2013 Sorry you're having this issue. Is there a chance your test is running out of wallclock time? The lack of any errors indicates the process may have been cut off. Aside from that, it may have been a memory-related segfault that didn't log properly. Either way, increasing the number of processors would be a good step forward to try to solve this.

As a side note, the GSL_HAFSV0 domains are untested, and there has been some discussion of removing them. I wouldn't recommend using them unless you need them for a specific reason.

[htan2013]

@htan2013 Sorry you're having this issue. Is there a chance your test is running out of wallclock time? The lack of any errors indicates the process may have been cut off. Aside from that, it may have been a memory-related segfault that didn't log properly. Either way, increasing the number of processors would be a good step forward to try to solve this.

As a side note, the GSL_HAFSV0 domains are untested, and there has been some discussion of removing them. I wouldn't recommend using them unless you need them for a specific reason.

Hi Kavulich,

Thanks for your reply!
I have added processors from 96 to 240 but still stopped. I don't quite understand why 3km doesn't work but 13 worked for generating ICs and BCs. I do need to use the GSL_HAFSV0 grid for some purposes.

[htan2013]

@htan2013 Sorry you're having this issue. Is there a chance your test is running out of wallclock time? The lack of any errors indicates the process may have been cut off. Aside from that, it may have been a memory-related segfault that didn't log properly. Either way, increasing the number of processors would be a good step forward to try to solve this.
As a side note, the GSL_HAFSV0 domains are untested, and there has been some discussion of removing them. I wouldn't recommend using them unless you need them for a specific reason.

Hi Kavulich,

Thanks for your reply! I have added processors from 96 to 240 but still stopped. I don't quite understand why 3km doesn't work but 13 worked for generating ICs and BCs. I do need to use the GSL_HAFSV0 grid for some purposes.

I have also tried customized grid with 3km (2595 x 1467 points). A similar thing happen when I make ICs. It stopped at some point.

[natalie-perlin]

Haochen,
What @mkavulich referred to was the wallclock time requested for the jobs, WTIME_MAKE_LBCS and WTIME_MAKE_ICS .
They are currently set to 30 min ( WTIME_MAKE_ICS=00:30:00, WTIME_MAKE_LBCS=00:30:00) for both 13-km grids and for 3-km grids. The 3-km grid job cuts short as it runs out of job allocation time.

[htan2013]

Haochen, What @mkavulich referred to was the wallclock time requested for the jobs, WTIME_MAKE_LBCS and WTIME_MAKE_ICS . They are currently set to 30 min ( WTIME_MAKE_ICS=00:30:00, WTIME_MAKE_LBCS=00:30:00) for both 13-km grids and for 3-km grids. The 3-km grid job cuts short as it runs out of job allocation time.

Hi Natalie,
@natalie-perlin
Same error (or hanging) on making ICs. I have added the following options in config.yaml:

task_make_ics:
MAKE_ICS_TN: "make_ics"
NNODES_MAKE_ICS: 4
PPN_MAKE_ICS: 24
WTIME_MAKE_ICS: 06:00:00
OMP_NUM_THREADS_MAKE_ICS: 4
OMP_STACKSIZE_MAKE_ICS: "2048m"

I have attached Make_ICS.log and PET00.ESMF.log here.

Thanks again for your help,
Haochen
Make_ICS.log
PET00.ESMF_LogFile.txt
config.yaml (2).txt

[natalie-perlin]

Haochen,
The WTIME_MAKE_LBCS=00:30:00 , as in the previous case. You could overwrite the default WTIME_MAKE_LBCS by setting it explicitly in your config.yaml, e.g.:

task_make_lbcs:
WTIME_MAKE_LBCS: 06:00:00

[htan2013]

Haochen, The WTIME_MAKE_LBCS=00:30:00 , as in the previous case. You could overwrite the default WTIME_MAKE_LBCS by setting it explicitly in your config.yaml, e.g.:

task_make_lbcs: WTIME_MAKE_LBCS: 06:00:00

Hi Natalie,

I did make WTIME_MAKE_ICS=06:00:00 in the config.yaml. Didn't change anything.

[natalie-perlin]

Just to make sure - do you mean that both WTIME_MAKE_ICS and WTIME_MAKE_LBCS are set to 06:00:00?

As JeffBeck mentions further below, there is a known memory issue with the UFS_UTILS.

[JeffBeck-NOAA]

@htan2013, what hash of UFS_UTILS are you using? There was a known memory issue with how chgres_cube was handling regridding of the 3-D wind field (where your 3-km chgres_cube runs hang) for large domains at high resolutions. Please see ufs-community/UFS_UTILS#766 PR in UFS_UTILS and the associated issue. The current HEAD of ufs-srweather-app does not pull a new enough version of UFS_UTILS to contain this fix. Please try pulling the latest UFS_UTILS code and let me know if this solves your problem. @mkavulich, @natalie-perlin, we should update the hash of UFS_UTILS in the Externals.cfg file to the HEAD of that repo.

[natalie-perlin]

@JeffBeck-NOAA - thank you for the tip on the memory fix PR#766. It mentioned the "branch at 220ca48".
When use this hash 220ca48 in place of the current one in SRW develop branch Externals.cfs,
and run ./manage_externals/checkout_externals, it produces the error:

Checking out externals: aqm-utils, ufs_utils, ERROR:root:In repo "ufs_utils": reference "220ca48" does not appear to be a valid tag, branch or hash! Please verify the reference name (e.g. spelling), is available from: https://github.com/ufs-community/UFS_UTILS 

ERROR: In repo "ufs_utils": reference "220ca48" does not appear to be a valid tag, branch or hash! Please verify the reference name (e.g. spelling), is available from: https://github.com/ufs-community/UFS_UTILS 

Is there anything to be done differently?

[MichaelLueken]

@natalie-perlin I believe that the hash you should use to bring in the memory fix from that PR is 2d1490f, rather than ufs-community/UFS_UTILS@220ca48.

Looking at the commit history for the authoritative repo, I don't see a January 31 commit with the hash 220ca48, so I suspect that this hash is from the forked branch that the PR was created using.

[natalie-perlin]

@MichaelLueken - thank you, that worked, was able to check out the 2d1490f. Testing it it now.

[natalie-perlin]

Haochen,

The memory fix for the UFS_UTILS did solve the issue. Please do the following to update it in your code, considering your SRW= :

1. Clean-up the earlier build of the code:

   cd ${SRW}
   ./devclean.sh -a --sub-modules

2. Change the hash in Externals.cfg, and check out the submodules:

sed -i -e "s|ca9bed8|2d1490f|" Externals.cfg
 ./manage_externals/checkout_externals

3. Update some modulefiles before building the code,

  cd ./modulefiles

Change the line 17 in `build_cheyenne_intel.lua` to the following:
prepend_path("MODULEPATH","/glade/work/epicufsrt/contrib/hpc-stack/intel2022.1/modulefiles/stack")
Build:

    cd ${SRW}
    ./devbuild.sh --platform=cheyenne --compiler=intel

4. Configure your case similarly to your earlier attempts, with the changes to the files in ./ush/ directory, i.e., config.yaml, predef_grid_params.yaml, valid_param_vals.yaml
   I've tested running your case on cheyenne using rocoto workflow manager, by setting
   WORKFLOW_MANAGER: rocoto
   The following steps would be:

module use ${SRW}/modulefiles
 module load wflow_cheyenne
 conda activate regional_workflow
 cd ${SRW}/ush
 ./generate_FV3LAM_wflow.py

... and then proceed with your workflow tasks.
Please see a log file make_ics.log attached. The next task, make_lbcs, successful but the log is larger; the forecast job have been submitted successfully after all the preprocessing tasks.

[make_ics_2020030612.log](https://github.com/ufs-community/ufs-srweather-app/files/11434361/m
make_ics_2020030612.log
ake_ics_2020030612.log)

[htan2013]

@natalie-perlin
Thanks, Natalie! I will test it now!

[natalie-perlin]

@htan2013 - please note that the wallclock for the forecast job needs to be increased. It is currently set to 2 hours, and the forecast job is killed early due to reaching time limit after ~36h out of 48 hour-forecast.