[BUG/ISSUE] Boundary conditions not updating in nested grid simulations

[rgstevens]

Description of the problem

I have been running GEOS-Chem on a custom nested grid over eastern Canada, and it appears that the boundary conditions for dust tracers are not updating properly with time during the simulation. Nothing seems to be unusual about the boundary condition files that I am using (these were output from a 2x2.5 resolution simulation using the same source code). The concentrations of dust in the boundary condition files do change with time, but the concentrations of dust at the edges of the high-resolution domain do not change with time.

Other species, including NH3, NH4, NIT, NO, O3, SALA, SALC, SOAP, SO4, have clearly time-varying concentrations in the boundaries. So I think that this is just affecting the dust tracers.

It's not quite an apples-to-apples comparison, but I include below plots of the monthly mean surface concentrations of DST2 from the nested simulation, and the DST2 surface concentrations from the boundary conditions files for the first timestep of each month. There is a transient dust plume in the 2x2.5 simulation over the northern USA. In the 0.5x0.625 simulation, the high dust concentrations in the boundary conditions persist for the entire nine-month simulation.

Description of troubleshooting performed

I am aware of issue #269 , and my 2x2.5 simulations would suffer from this issue. However, I don't believe that this is connected, as this issue seems to specifically be affecting dust tracers, and not other tracers.

GEOS-Chem version

I am using GEOS-Chem version 12.9.1. I have not modified the source code at all.

Log files

input.geos.txt
slurm-17063305.out.txt

Software versions

• CMake version: cmake/3.18.4
• Compilers (Intel or GNU, and version): ifort 18.0.3
• NetCDF version: netcdf-fortran-mpi/4.5.2; netcdf-mpi/4.7.4

[yantosca]

Thanks for writing @rgstevens. Can you also attach your HEMCO_Config.rc and HEMCO.log files?

[rgstevens]

I can provide my HEMCO_Config.rc. Unfortunately, I no longer have my HEMCO.log for this simulation; it has since been overwritten. I can provide the HEMCO.log from a similar simulation that suffers from the same issue.
HEMCO_Config.rc.txt
HEMCO.log.txt

Thank you for looking into this!

[yantosca]

Just to let you know I am still looking into this. Had to do some other work on short notice. Stay tuned.

[yantosca]

One thing you can try is to re-run a short simulation with Verbose: 3 in the HEMCO_Config.rc file. Then search for SpeciesBC_DST1 in the HEMCO.log file. I wonder if the boundary condition is only being read in at the first timestep of the month for some reason.

[rgstevens]

I did as you suggested, and I looked through the HEMCO.log output. There doesn’t seem to be anything suspicious about the content related to speciesBC_DST1, as compared to any other species. However, I noticed that searching for “speciesBC” or “boundary” does not turn up any results after the first hour of the model simulation.

I think that I was mistaken before when I concluded that the dust tracers were the only ones affected. Looking through the standard output, I see the message:
“HEMCO: Opening /home/rstevens/GC/RunDirs/merra2_2x25_tropchem/OutputDir/GEOSChem.BoundaryConditions.20130501_0000z.nc4”
But this is the only instance of a Boundary Conditions file being read. Other files are read throughout the simulation, notably emissions and meteorology, but not the boundary conditions.

The concentrations of some other species do vary with time in the boundaries of the SpeciesConc output, but perhaps this is because other processes (chemistry, transport, etc.) occur after the concentrations are set based off of the boundary conditions, and before the diagnostics are saved. I have not looked at the order of operations in the model.

So my current suspicion is that the first boundary conditions file is read, the boundary conditions container is filled, and then never updated in my simulations. The Get_Boundary_Conditions just continually retrieves the same information from the container. I am currently running another short simulation with the debug output option turned on in input.geos, so I can see if the mins and maxes output for the boundary conditions are constant with time. I’m also searching the code for where the frequency of reading the boundary conditions files is set. I’ll post again when I have more information.

[rgstevens]

I can confirm now that the mins and maxs of each species for each day are identical, as listed in the standard output. They follow the same cycle of 8 time points each day. The same diurnal cycle repeats every day.

The HEMCO.log does list the boundary condition species in the Day List, though, so they should be updated each day.

However, when the day list is processed on the second day of model simulation, when it gets to the entries where the speciesBC variables should be (after TMPU_NEXTDAY, before XLAI00), it repeats the following ~180 times:

HEMCO: Entering tIDx_Assign (hco_tidx_mod.F90) ( 4)
HEMCO: Leaving tIDx_Assign (hco_tidx_mod.F90) ( 4)
HEMCO: Entering EmisList_Pass (hco_emislist_mod.F90) ( 4)
HEMCO: Entering EmisList_Add (HCO_EMISLL_MOD.F90) ( 5)
HEMCO: Leaving EmisList_Add (HCO_EMISLL_MOD.F90) ( 5)
HEMCO: Leaving EmisList_Pass (hco_emislist_mod.F90) ( 4)
HEMCO: Entering HCOIO_DataRead (hcoio_dataread_mod.F90) ( 4)
HEMCO: Leaving HCOIO_DataRead (hcoio_dataread_mod.F90) ( 4)

There are also ~180 speciesBC variables, so I think it likely that this is HEMCO skipping over each of the boundary conditions species.

I note that it doesn’t print a message saying that it is entering HCOIO_Read_Std for these variables. Since it skips past without printing an error message, I am guessing that the following IF statement in HCOIO_DataRead is True:

 IF ( Lct%Dct%Dta%CycleFlag == HCO_CFLAG_EXACT .and.                      &
         Lct%Dct%Dta%UpdtFlag  == HCO_UFLAG_ONCE  .and.                      &
         Lct%Dct%Dta%isTouched                          ) THEN

The message “No further attempts will be made to read file:” was already printed with regards to the HEMCO restart file, so doPrintWarning is already False, so no message regarding the boundary conditions file is printed.

In HEMCO_Config, I note that the line specifying the boundary conditions file is:

* BC_ /home/rstevens/GC/RunDirs/merra2_2x25_tropchem/OutputDir/GEOSChem.BoundaryConditions.$YYYY$MM$DD_$HH$MNz.nc4 SpeciesBC_?ADV?  1980-2020/1-12/1-31/* EY xyz 1 * - 1 1

If I understand correctly, the EY specifies exact, using the simulation year. So I am guessing that this corresponds to HCO_CFLAG_EXACT. I'm not yet clear on what sets HCO_UFLAG_ONCE. But it looks like I might be able to bypass this issue by changing the CRE flag. I will start looking for how HCO_UFLAG_ONCE is set, and try changing the CRE flag for the boundary conditions to R, and see if that resolves the issue.

[yantosca]

Thanks for the feedback. Have you tried GEOS-Chem 13.0.2? We added some fixes into HEMCO so that EY (and EFY) should keep reading if it's in the exact range. EFYO is used for the restart file, and only reads once.

[rgstevens]

The workaround of changing the CRE flag seems to have worked. The mins and maxs of the boundary conditions printed in the standard output now differ from day to day. A quick look at the output shows that the large peak in dust in the boundary does not persist after the first day of model simulation; the boundary concentrations of dust differ from day to day.

I have not yet tried 13.0.2. I would prefer to have a consistent model version for the global 2x2.5 simulations and the nested 0.5x0.625 simulations, for comparison, and I am also hoping not to re-run the global simulations yet. So I think that I will stick with the workaround as my solution for now.

It sounds like the fixes that you introduced should fix this issue, but I will do a quick naive test using 13.0.2 anyway, to make sure.

[rgstevens]

I ran a simulation using version 13.0.2, and I was surprised to see the behaviour repeated. If you look at the standard output, only the boundary conditions file for the first day is read:

HEMCO: Opening /home/rstevens/GC/RunDirs/merra2_2x25_tropchem/OutputDir/GEOSChem.BoundaryConditions.20130501_0000z.nc4

No similar statement for e.g. 20130502 ever appears.
Also, the sequence of mins and maxs of each species in the boundary conditions repeats exactly for each day.

I have attached the input.geos, HEMCO_Config.rc, and the standard output for this simulation. I'd attach the HEMCO.log, but with the verbose output, it's 278M.

I think that the workaround of changing the CRE flags will work for my purposes, so I wasn't planning on investigating this further myself. However, if there are simple tests that you'd like me to try, or if there is more information I can provide, I would be happy to help.

HEMCO_Config.rc_debug04.txt
input.geos_debug04.txt
slurm-19050889.out.txt

[msulprizio]

@rgstevens You mention that changing the time cycle flag to CRE works for you, in that the boundary conditions seem to update daily. Can you confirm in your GEOS-Chem log file that the correct BC files are being read for each day?

[rgstevens]

Yes, I can confirm that. I just double-checked the log file for that test. For each day, I see a statement like:

HEMCO: Opening /home/rstevens/GC/RunDirs/merra2_2x25_tropchem/OutputDir/GEOSChem.BoundaryConditions.20130502_0000z.nc4

in the log file. Each of these statement are preceded by other "HEMCO: Opening" statements, and above them is a statement like:

---> DATE: 2013/05/02  UTC: 00:00  X-HRS:     24.000000

The date in the boundary conditions file name matches the current simulation date in each instance, and a new file is read for every 24 hours of simulation time.

[msulprizio]

Thanks for confirming. I just checked the behavior of the time cycle flags in HEMCO/src/Core/hco_config_mod.F90. We recently update the comments to better reflect what happens for each flag:

                ! Set time cycling behaviour. Possible values are:
                ! - "C"   : cycling <-- DEFAULT
                ! - "CS"  : cycling, skip if not exist
                ! - "CY"  : cycling, always use simulation year
                ! - "CYS" : cycling, always use simulation yr, skip if not exist
                ! - "R"   : range
                ! - "RA"  : range, average outside
                ! - "RF"  : range, forced (error if not in range)
                ! - "RFY" : range, forced, always use simulation year
                ! - "RFY3 : range, forced, always use simulation year, 3-hourly
                ! - "RY"  : range, always use simulation year
                ! - "E"   : exact, read/query once
                ! - "EF"  : exact, forced (error if not exist), read/query once
                ! - "EFY" : exact, forced, always use sim year
                ! - "EFYO": exact, forced, always use sim year, read once
                ! - "EC"  : exact, read/query continuously (e.g. for ESMF interface)
                ! - "ECF" : exact, forced, read/query continuously
                ! - "EY"  : exact, always use simulation year, read/query once
                ! - "A"   : average
                ! - "I"   : interpolate
                ! - "ID"  : interpolate, discontinuous dataset

The code tells us that the EY flag is in fact the wrong flag to use for boundary conditions because it sets Dta%UpdtFlag = HCO_UFLAG_ONCE, meaning the data are only read once. I believe what we want to use is the EFY flag which does not hardcode the update flag to "once". We can fix this in GEOS-Chem 13.1.0.

[rgstevens]

Okay, that makes sense. Thank you for finding the source of the issue.

Given that this was a silent error, and EY is the default setting for boundary conditions in the HEMCO_Config.rc templates, it seems likely that there are other users affected by this who do not know that they are affected by this. Is there an efficient way to let other GEOS-Chem users who are running nested simulations know about this?

[msulprizio]

Yes, when we release 13.1.0, we will send an email to the entire GEOS-Chem community. In that email, we can also highlight the importance of this fix for nested-grid simulation users.

[msulprizio]

This is now resolved in commit b474612 (currently in dev) and will be included in GEOS-Chem 13.1.0.