[BUG/ISSUE] Allocation error in RRTMG nested-grid simualation

[bsuther]

Describe the bug

Include a clear and concise description of the bug or issue that you have encountered.

Shortly after starting the run I am recieveing the following error:

================================================================
GEOS-CHEM ERROR: Allocation error in array: ASMCMCL
STOP at alloc_err.f
================================================================

To Reproduce

I am attempting to run RRTMG on a nested grid over north america.
Here are the steps I did to try to set up the run

1. I created a geosfp 4x5 RRTMG global directory using the unit tester and renamed it from
   geosfp_4x5_RRTMG to geosfp_025x03125_RRTMG_na.

2. I made the following modifications to input.geos
   a) changed the grid menu to

%%% GRID MENU %%%       :
Grid resolution         : 0.25x0.3125
Longitude min/max       : -130.0 -60.0
Latitude  min/max       :   9.75  60.0
Half-sized polar boxes? : F
Number of levels        : 47
Nested grid simulation? : T
Buffer zone (N S E W )  :  3  3  3  3

b) changed the timestep menu to

%%% TIMESTEP MENU %%%   :
 Tran/conv timestep [sec]: 300 

Note: I Made this modification because I was getting the following error:

GEOS-Chem ERROR: Transport timestep exceeds max:     600 >     300
 -> at Check_Time_Steps (in module GeosCore/input_mod.F)

3. I replaced the HEMCO_Config.rc file which is created with the RRTMG unit tester template with the template from the geosfp 025x03125 na tropchem simulation. I then modified it to turn the offline biogenicvoc off and megan on

--> OFFLINE_BIOGENICVOC    :       false
...
108     MEGAN                  : on    ISOP/ACET/PRPE/C2H4/ALD2/EOH/SOAP/SOAS
...
109     MEGAN_Mono             : on    MTPA/MTPO/LIMO/SESQ

And I changed to GC_BCs root to point toward the run where I created the boundary conditions

((GC_BCs
* BC_ /gpfs_common/shar02/meskhidze/bsuther/GEOSchem/rundirs/geosfp_2x25_RRTMG/Output/GEOSChem.BoundaryConditions.$YYYY$MM$DD_$HH$MNz.nc4 SpeciesBC_?ADV?  1980-2019/1-12/1-31/0-23 RFY xyz 1 * - 1 1
   )))GC_BCs

Compilation commands

1. make -j4 build RRTMG=y

Run commands

1. I start my run using the following script:

#!/bin/tcsh
#BSUB -W 10080
#BSUB -n 12
#BSUB -R span[hosts=1]
#BSUB -J GeosMP_na
#BSUB -o out.%J
#BSUB -q meskhidze

source ~/Enviroment.sh

cd /share/meskhidze/bsuther/GEOSchem/rundirs/geosfp_025x03125_RRTMG_na
./geos >> GC.log

Expected behavior

I was hoping to get similar results to what I got using the RRTMG global run but on the finer grid over north america

Error messages

This is the error message I am getting in the standard output log:

===============================================================================
GEOS-CHEM ERROR: Allocation error in array: ASMCMCL
STOP at alloc_err.f
===============================================================================

I am getting this backtrace

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x2b568d5ad2ef in ???
#1  0xc2700f in __hco_driver_mod_MOD_hco_final
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/HEMCO/Core/hco_driver_mod.F90:314
#2  0x70cee1 in __hcoi_gc_main_mod_MOD_hcoi_gc_final
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosCore/hcoi_gc_main_mod.F90:1009
#3  0x7c63aa in __emissions_mod_MOD_emissions_final
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosCore/emissions_mod.F90:474
#4  0x598011 in cleanup_
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosCore/cleanup.F:302
#5  0xca7173 in __error_mod_MOD_geos_chem_stop
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosUtil/error_mod.F:672
#6  0xca7490 in __error_mod_MOD_error_stop
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosUtil/error_mod.F:581
#7  0xca7559 in __error_mod_MOD_alloc_err
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosUtil/error_mod.F:757
#8  0x4f4909 in init_mcica_clouds
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosCore/rrtmg_rad_transfer_mod.F:2685
#9  0x4f4909 in __rrtmg_rad_transfer_mod_MOD_init_rrtmg_rad_transfer
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosCore/rrtmg_rad_transfer_mod.F:2146
#10  0x56215a in geos_chem
        at /gpfs_common/share02/meskhidze/bsuther/GEOSchem/Code.12.6.1/GeosCore/main.F:555
#11  0x403266 in ???
#12  0x2b568d599444 in ???
#13  0x403296 in ???
#14  0xffffffffffffffff in ???

Required information

Please include the following:

• GEOS-Chem version you are using [12.6.1]
• Compiler version that you are using [gcc-9.2.0]
• netCDF and netCDF-Fortran library version [netcdf/4.6.3-gcc4.8.5 ]
• Computational environment [cluster]
• Are you using "out of the box" code, or have you made modifications? Out of the box.

Input and log files to attach

input&logfiles.zip

It never created a HEMCO.log file, but all the others are in the zipped folder.

Additional context

Include any other context about the problem here.

[yantosca]

Thanks for writing. To find the location where the error occurred I did:

grep -in ASMCMCL */*.F*

which retuns a lot of output, but in particular these are the lines of interest:

GeosCore/rrtmg_rad_transfer_mod.F:2683:      ALLOCATE( ASMCMCL( NGPTSW, State_Grid%NX, State_Grid%NY,
GeosCore/rrtmg_rad_transfer_mod.F:2685:      IF ( AS /= 0 ) CALL ALLOC_ERR( 'ASMCMCL' )

What is happening is that your simulation is running out of memory, and cannot allocate the RRTMG array ASMCMCL. You are probably hitting the maximum memory allocated for the job in the particular queue of your scheduling system.

It looks like your queue system is LSF. You can try to run in a queue that has more memory than what you are currently using. I would ask your IT staff if they can suggest which queue on your system has the most memory.

The RRTMG code is third-party, and as such, it takes up a lot of memory. We are not aware of users who have tried to use RRTMG with a nested simulation.

[bsuther]

Thank you very much.

[yantosca]

Sure. Let us know if you still have problems.