Restart files

[mkbenneh]

I need some assistance using the restart files to extend my model run.

[MichaelLueken]

Hello @mkbenneh. The use of restart files in the SRW App is for allowing the ability to run the AQM with warmstart capability. Are you trying to run a long forecast (FCST_LEN_HRS>=100)? If you are, I would recommend looking at the long_fcst WE2E configuration in the SRW. If you are wanting to try and run a warmstart experiment, I'd recommend looking at the AQM sample configuration available in the ush directory.

[mkbenneh]

Hello @MichaelLueken,

I tried increasing my resources, like increasing NCORES_PER_NODE:, NNODES_RUN_FCST:, the simulation stops. Here is the path to my working directory, "/work2/noaa/uswrp-hu/mbenneh/UFS-SRW/expt_dirs/dust". I tried different methods but it seems to persist if you try to increase computing resources.

[MichaelLueken]

Hello @mkbenneh,

I can't access your directories on Orion. Please provide the log files for the failed task. However, if you have already tried increasing the computing resources and it continues to stop, I don't know what else to try.

[mkbenneh]

Hello @MichaelLueken,

It doesn't necessarily fail, so it doesn't provide a log file for the run_fcst_mem000, however, I am including my config file and the workflow log file here for your perusal.

FV3LAM_wflow.txt
config.yaml (3).txt

I will work on my orion access restrictions too.

[MichaelLueken]

Hello @mkbenneh,

The issue is with your setting of NCORES_PER_NODE. This value should not be touched for any Tier-1 platform, and can not be increased. On Orion, each node has 40 cores. This is set and can't be changed. Setting NCORES_PER_NODE: 60 will cause your run_fcst task to not get submitted due to oversubscription of the cores per node on the machine.

Please set NCORES_PER_NODE: 40. If you are encountering memory issues, you will want to try reducing the PPN_RUN_FCST value in task_run_fcst. The PPN_RUN_FCST sets the number of processors per node to be used to run the forecast. The lower this value, the more nodes that you will use to run your run_fcst task and the more memory available per node. While reducing PPN_RUN_FCST, you will want the workflow to calculate the appropriate NNODES_RUN_FCST for you, so I would also recommend removing NNODES_RUN_FCST: 6 from your config.yaml file.

[mkbenneh]

Hello @MichaelLueken,

Thank you for the explanation and recommendation.