[BUG/ISSUE] Issue with nested CO2 simulation (custom grid) boundary conditions

[tscarpelli]

Description of the problem

Hello support team! I am a new GEOS-Chem user and I am trying to run a 1-week nested CO2 simulation at 0.25 x 0.3125 using GEOS-Chem v13.2.1 over a custom region in Europe. The code dies soon after it starts time stepping with a Segmentation Fault error (invalid memory reference).

Description of troubleshooting performed

I believe the problem is associated with how I specify boundary conditions because when I remove my GC_BCs edits from my HEMCO_Config.rc my edited code finishes successfully. When I set up the nested simulation, the HEMCO_Config.rc file did not have the option to toggle GC_BCs true/false in the Extension Switches Section so I added it there and I also specified my boundary conditions from a global simulation (4x5 CO2 simulation) in the BASE EMISSION section. When I remove both of these edits the code runs successfully.

I followed the directions on the Wiki regarding building with Debug on and the error that resulted is below. If I don't run with Debug on it gives the same error as below but at line 2471 in hco_utilities_gc_mod.F90. When I investigated this it looks like there is an issue when it tries to manipulate Ptr3D but I'm not familiar enough with the source code to investigate more than that.

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x146efa0d3d21 in ???
#1  0x146efa0d2ef5 in ???
#2  0x146ef9d5b20f in ???
#3  0x5597c149ad28 in __hco_utilities_gc_mod_MOD_get_boundary_conditions
        at /exports/csce/datastore/geos/users/tscarpel/gc_geosfp_CO2_nested_custom_plain_bug/CodeDir/src/GEOS-Chem/GeosCore/hco_utilities_gc_mod.F90:2463
#4  0x5597c148a70c in __hco_interface_gc_mod_MOD_hcoi_gc_run
        at /exports/csce/datastore/geos/users/tscarpel/gc_geosfp_CO2_nested_custom_plain_bug/CodeDir/src/GEOS-Chem/GeosCore/hco_interface_gc_mod.F90:1051
#5  0x5597c139bd37 in __emissions_mod_MOD_emissions_run
        at /exports/csce/datastore/geos/users/tscarpel/gc_geosfp_CO2_nested_custom_plain_bug/CodeDir/src/GEOS-Chem/GeosCore/emissions_mod.F90:203
#6  0x5597c12ee52f in geos_chem
        at /exports/csce/datastore/geos/users/tscarpel/gc_geosfp_CO2_nested_custom_plain_bug/CodeDir/src/GEOS-Chem/Interfaces/GCClassic/main.F90:961
#7  0x5597c12f3011 in main
        at /exports/csce/datastore/geos/users/tscarpel/gc_geosfp_CO2_nested_custom_plain_bug/CodeDir/src/GEOS-Chem/Interfaces/GCClassic/main.F90:32 ​

GEOS-Chem version

v13.2.1

Description of modifications

For input.geos: I adjusted the time frame and specified a custom grid. I also toggled all emissions to False except Fossil Fuel for CO2 SIM MENU.
For HEMCO_Config.rc: I specified boundary conditions, turned off all non-fossil fuel emissions. Toggled CO_COPROD to False and GFED to off. I also changed the GC_RESTART entry so that it read CYS rather than EFYO for SPC_ because it was giving me errors.

Log files

• Run logs (if applicable):
  HEMCO.log.txt.txt
  GC.log.txt.txt

• Run script (if applicable):
  HEMCO_Config.rc.txt.txt
  input.geos.txt.txt

[tscarpelli]

Just to add in case it's relevant: I've also noticed that for the set-up I describe above if I add my own emissions files in HEMCO_Config.rc the added emissions are not read in during the simulation.

[msulprizio]

Hi @tscarpelli. Thanks for writing. Try changing you BC_ entry in HEMCO_Config.rc from hour=0-23 to *:

#==============================================================================
# --- GEOS-Chem boundary condition file ---
#==============================================================================
(((GC_BCs
* BC_ /exports/csce/datastore/geos/users/tscarpel/my_gc_data/Global_BCs_4x5/01012018-16012018/3hourly/GEOSChem.BoundaryConditions.$YYYY$MM$DD_$HH$MNz.nc4 SpeciesBC_?ADV?  2010-2019/1-12/1-31/* C xyz 1 * - 1 1
)))GC_BCs

The asterisk/wildcard character tells HEMCO to read all hours for that day at the start of the day. For BCs, HEMCO expects daily files with 3-hourly data (i.e. 8 time stamps). The appropriate 3-hourly timestamp is selected and applied to the species concentration array in the GEOS-Chem routine Get_Boundary_Conditions (in GeosCore/hco_utilities_gc_mod.F90). The way you you currently have it with hour specified as 0-23, HEMCO only reads in one hour at a time but GEOS-Chem is expecting 8 time slots which may explain the error you are seeing.

[tscarpelli]

Thanks @msulprizio for the response. I still get the same error after making your suggested edit.

[yantosca]

Hi @tscarpelli, thanks for writing. A seg fault is an invalid memory reference, which can happen if you try to reference a pointer that is NULL. It can also happen if you are running out of available memory. How much memory did you allot for your simulation? You might try increasing the amount of memory in your run script (if you are using SLURM or some other scheduler) and then see if that gets past the error.

Also, if you have the gdb debugger installed on your system, and if a core file was dumped, then you can do:

gdb gcclassic core.XXXXX

where XXXXX is the suffix of the core file. That should take you to the point of failure. If you type "where" then it should give you info about the error. If it says "no stack" then configure GEOS-Chem with -DCMAKE_BUILD_TYPE=Debug, recompile it, and then type:

gdb gcclassic

and then when the debugger opens, type "run". That should get you to the point of the error.

[tscarpelli]

Thanks @yantosca. I'm not using SLURM or a scheduler (and I have ulimit -s unlimited in my .bashrc as suggested in the wiki). I did compile GEOS-Chem as directed for debugging (see below, as suggested in #788) and our system does have gdb but there doesn't appear to be a core file.

Given that I was also unable to make edits to the emission inventories specified in the HEMCO_Config.rc, for now I will switch to an older version of the model that other people in my group are also using and we know runs successfully with my desired edits. I will close out this issue but may try the update to v13 again later when I'm more familiar with GEOS-Chem. Thanks for the help!

[yantosca]

Thanks @tscarpelli. When you are ready to migrate to v13, let us know if you have any questions. We also have a couple of intro videos about v13 on our Youtube channel.