[BUG/ISSUE] FlexGrid simulation error in GEOS-Chem 12.7.0

[yantosca]

I am submitting this issue on behalf of Nan Lin (nlin3 [at] Central.UH.EDU)

I met an error in running GCv12.7.0 FlexGrid. Don't know if you have any suggestions.
First, I use MERRA2 2x2.5 global simulation to generate the boundary file "GEOSChem.BoundaryConditions*.nc4". But when I simulate the nested grid, it shows that the files cannot be found. I know this is a very common mistake and usually means there is something wrong with the file path or time. But this time it is a little different. The log file is as follows:

---> DATE: 2014/10/08  UTC: 21:10  X-HRS:     21.166666
GET_BOUNDARY_CONDITIONS: Found All BCs at 2014/10/08 21:10
---> DATE: 2014/10/08  UTC: 21:20  X-HRS:     21.333334
GET_BOUNDARY_CONDITIONS: Found All BCs at 2014/10/08 21:20
     - DO_STRAT_CHEM: Linearized strat chemistry at 2014/10/08 21:20
     - LINOZ_CHEM3: Doing LINOZ
---> DATE: 2014/10/08  UTC: 21:30  X-HRS:     21.500000
GET_BOUNDARY_CONDITIONS: Found All BCs at 2014/10/08 21:30
---> DATE: 2014/10/08  UTC: 21:40  X-HRS:     21.666666
GET_BOUNDARY_CONDITIONS: Found All BCs at 2014/10/08 21:40
     - DO_STRAT_CHEM: Linearized strat chemistry at 2014/10/08 21:40
     - LINOZ_CHEM3: Doing LINOZ
---> DATE: 2014/10/08  UTC: 21:50  X-HRS:     21.833334
GET_BOUNDARY_CONDITIONS: Found All BCs at 2014/10/08 21:50
     - Creating file for StateMet; reference = 20141008 210000
        with filename = OutputDir/GEOSChem.StateMet.20141008_2100z.nc4
     - Creating file for SpeciesConc; reference = 20141008 210000
        with filename = OutputDir/GEOSChem.SpeciesConc.20141008_2100z.nc4
---> DATE: 2014/10/08  UTC: 22:00  X-HRS:     22.000000
HEMCO: Opening /project/data/ExtData/HEMCO/SAMPLE_BCs/v2019-12/tropchem/GEOSChem.BoundaryConditions.20141008_0000z.nc4
 Error in HCOIO_DATAREAD called from HEMCO ReadList_Fill: BC_NO
 Error in ReadList_Fill (5) called from HEMCO ReadList_Read
 Error in ReadList_Read called from hco_run
===============================================================================
GEOS-Chem ERROR: Error encountered in "HCO_Run"!
 -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90)

THIS ERROR ORIGINATED IN HEMCO!  Please check the HEMCO log file for
additional error messages!
===============================================================================

===============================================================================
GEOS-Chem ERROR: Error encountered in "HCOI_GC_Run"!
 -> at Emissions_Run (in module GeosCore/emissions_mod.F90)
===============================================================================

===============================================================================
GEOS-CHEM ERROR: Error encountered in "Emissions_Run"!
STOP at  -> at GEOS-Chem (in GeosCore/main.F)
===============================================================================
     - CLEANUP: deallocating arrays now...

As you can see, the model has started daily simulation, but the file cannot be found in the first day at UTC:22:00😰. I'm pretty sure I have the corresponding files and variables. HEMCO's log file is also attached:

--> LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
HEMCO ERROR: Cannot find field with valid time stamp in /project/data/ExtData/HEMCO/SAMPLE_BCs/v2019-12/tropchem/GEOSChem.BoundaryConditions.20141008_0000z.nc4 - Cannot get field BC_NO. Please check file name and time (incl. time range flag) in the config
ERROR LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90)
ERROR LOCATION: HCOIO_DataRead (hcoio_dataread_mod.F90)
ERROR LOCATION: ReadList_Fill (hco_readlist_mod.F90)
ERROR LOCATION: ReadList_Read (hco_readlist_mod.F90)
ERROR LOCATION: HCO_RUN (hco_driver_mod.F90)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
HEMCO ERROR: Error encountered in routine HCOIO_Read_Std!
ERROR LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90)
ERROR LOCATION: HCOIO_DataRead (hcoio_dataread_mod.F90)
ERROR LOCATION: ReadList_Fill (hco_readlist_mod.F90)
ERROR LOCATION: ReadList_Read (hco_readlist_mod.F90)
ERROR LOCATION: HCO_RUN (hco_driver_mod.F90)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

If you have any suggestions or ideas it will be very helpful for me!

Thanks,
Nan

[yantosca]

Thanks for writing. It seems like the model cannot find the file:

/project/data/ExtData/HEMCO/SAMPLE_BCs/v2019-12/tropchem/GEOSChem.BoundaryConditions.20141008_0000z.nc4

Have you made sure that this file exists? Perhaps your global simulation only created BC files for October 1-7, but not for October 8th. The quick solution would be to run the global simulation from October 7th to October 8th to create the BC files for October 8th.

@msulprizio: Do you think this might be a side-effect of the new way we are reading in the 3-hourly files? If so we should encourage nested-grid users to create boundary conditions for one day beyond the nested-simulation period.

[NanLinn]

Thank you for your reply. I double checked that these files do exist. My BoundaryConditions file covers October 06 - November 06.
When I run the nested grid simulation, if the start date is set to October 06, the error occurs before the daily simulation is started. The HEMCO.log is as follows:

HEMCO ERROR: Cannot find field with valid time stamp in /project/data/ExtData/HEMCO/SAMPLE_BCs/v2019-12/tropchem/GEOSChem.BoundaryConditions.20141006_0000z.nc4 - Cannot get field BC_NO. Please check file name and time (incl. time range flag) in the config

If I change the start data to October 08, the error will occur at 22:00 on the October 08. If I change the start date to October 07, it will report the error at 22:00 on the October 07 (first day of simulation).
I'm surprised the model can find boundary condition variables before UTC 22:00, but can not at UTC 22:00, it is very weird.

[yantosca]

Can you send us the input files and log files from your simulation, as described here:
http://wiki.geos-chem.org/Submitting_GEOS-Chem_support_requests#What_do_I_need_to_include_in_my_support_request.3F

That will help us better make an assessment. You can attach them (rename files so they have the .txt extension) and just drag & drop into the Github comment box.

[NanLinn]

GC_log.txt
HEMCO.log.txt
input.geos.txt
HISTORY.rc.txt
HEMCO_Config.rc.txt

Attached are those files. Thanks.

[NanLinn]

One more piece of information:
I used GCv12.6.3's nested run directory to read the above BountaryConditionnc4 files (The boundary files generated by GCv12.7.0 MERRA2 2*2.5 simulation). It can run successfully.

[yantosca]

I am wondering if it is a problem with your file

/project/data/ExtData/HEMCO/SAMPLE_BCs/v2019-12/tropchem/GEOSChem.BoundaryConditions.20141008_0000z.nc4

perhaps it is corrupted. Can you do

ncdump -cts  /project/data/ExtData/HEMCO/SAMPLE_BCs/v2019-12/tropchem/GEOSChem.BoundaryConditions.20141008_0000z.nc4 > log.txt

and attach the log.txt file to this issue? That will let us see if the file is OK.

[NanLinn]

log.txt
Here it is.

[yantosca]

Thanks. I wonder if this is an issue in HEMCO. It seems like it is failing on the 22nd hour. I wonder if it closed the file at UTC=21 but then is still trying to read it.

In 12.6.3, HEMCO was still trying to read data hourly. We have added several fixes to HEMCO in 12.7.0 and later. You might want to try updating to see if that fixes your problem.

[msulprizio]

I have just pushed a fix for this issue to the dev/12.7.2 branch (see commit 722a151). Essentially, it was an issue of the 3-hourly boundary condition files being read every hour by HEMCO. With this update, you will also need to change the line in HEMCO_Config from:

* BC_  $ROOT/SAMPLE_BCs/v2019-12/tropchem/GEOSChem.BoundaryConditions.$YYYY$MM$DD_$HH$MNz.nc4 SpeciesBC_?ADV?  1980-2019/1-12/1-31/0-23 RFY xyz 1 * - 1 1

to

* BC_  $ROOT/SAMPLE_BCs/v2019-12/tropchem/GEOSChem.BoundaryConditions.$YYYY$MM$DD_$HH$MNz.nc4 SpeciesBC_?ADV?  1980-2019/1-12/1-31/* RFY xyz 1 * - 1 1

Replacing hour 0-23 with * will tell HEMCO to read the BCs for all timesteps on a given day at hour 0. Then in hcoi_gc_main_mod.F90, we use new functions to determine if it's time to update the BCs. By default, it assumes BCs are 3-hourly but we may need to make that more flexible in the future to allow for other frequencies.

This will be included in the 12.7.2 release (hopefully next week), but you may use the fix in the dev/12.7.2 branch in the meantime.

[NanLinn]

It works!! Thank you both for your help!

[neyranaz]

Hi @msulprizio Based on your explanation here I wanted to check something with you.

I generated hourly boundary conditions on daily files, for two nested grid runs. In one, I included your correction suggested in this thread, and in the other one I didn't.For some reason, I didn't get the tilmestep error but I wanted to ask you if the fact that BCs are being read every hour instead of every day, might cause an issue in the species concentrations generated.

I've prepared a figure so that it can be easier to see my concern. You can find it here

My guess is that the * just tells when to open the file, but that shouldn't cause a difference in species concentrations. Is that correct?

Could this difference in HEMCO_Config.rc be related to an accumulation in DST, rooted here and seen here

[neyranaz]

We are using Version 12.7.0