Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of TOMAS in GCHP #1378

Closed
wants to merge 39 commits into from
Closed

Implementation of TOMAS in GCHP #1378

wants to merge 39 commits into from

Conversation

Jourdan-He
Copy link
Contributor

@Jourdan-He Jourdan-He commented Sep 15, 2022

This is the PR adding TOMAS simulations in GCHP. It supersedes #1261 originally from @BettyCroft. This PR should be merged along with:

I have test run the merge on a single node simulation, and it's completed successfully.

@Jourdan-He Jourdan-He added the category: Feature Request New feature or request label Sep 15, 2022
@Jourdan-He Jourdan-He added this to the 14.1.0 milestone Sep 15, 2022
@Jourdan-He Jourdan-He self-assigned this Sep 15, 2022
Jourdan-He and others added 2 commits September 21, 2022 10:20
run/shared/species_database.yml
- Following geoschem/geos-chem #1414, we have added the attribute
  "WD_CoarseAer: true" for SO4s and NITs species.  SO4s and NITs
  should be treated exactly like coarse-mode aerosols in wet scavenging.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@msulprizio msulprizio changed the base branch from dev to main October 18, 2022 16:51
This commit adds the reaction requested in geoschem/geos-chem #1235

KPP/custom/custom.eqn
KPP/fullchem/fullchem.eqn
- Added MO2 + NO3 = NO2 + CH2O + HO2 with constant rxn rate of 1.0d-12

KPP/fullchem/*.map
KPP/fullchem/gckpp*.F90
- Rebuilt with KPP 2.5.0

NOTE: We will bring this update in with KPP 2.5.0 and then rebuild the
mechanism with KPP 3.0.0 before releasing GEOS-Chem 14.1.0.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
KPP/custom/custom.eqn
KPP/fullchem/fullchem.eqn
- Apply the bug fix provided by @khbates in geoschem/geos-chem #1274,
  namely to change the Arrhenius A coefficient of the ETO = HO2 + CH2O
  rxn from 9.5d-13 to 9.5d+13

KPP/fullchem/gckpp_Rates.F90
- Rebuilt with KPP 2.5.0

NOTES:
(1) This commit builds on top of geoschem/geos-chem PR #1444, and
    thus must be merged on top of PR #1444.
(2) For now we have rebuilt the fullchem mechanism with KPP 2.5.0,
    but we will rebuild all KPP mechanisms prior to the 14.1.0 release.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
KPP/custom/custom.eqn
KPP/fullchem/fullchem.eqn
- HOBr + SO2 = SO4s + HBr is now HOBr + SO2 = SO4 + HBr
- HOCl + SO2 = SO4s + HCl is now HOCl + SO2 = SO4 + HCl
- Updated comments

KPP/fullchem/gckpp*.map
KPP/fullchem/gckpp*.F90
- Rebuilt with KPP 2.5.0

NOTES:
(1) We will rebuild all mechanisms with KPP 3.0.0 before the official
    release of GEOS-Chem 14.1.0.
(2) This commit is built on top of the commit 234c429 (Bug fix: change
    sign in the Arrhenius A coefficient of ETO = HO2 + 2CH2O), and should
    be merged afterwards.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
run/shared/species_database.yml
run/shared/species_database_hg.yml
- As suggested by @XueyingYu-1993, we are changing the molecular weight
  of methane from 16.05 g/mol to 16.04 g/mol in the species_database*yml
  files.  This fixes a roundoff error that was caused by adding the
  rounded value of 4H (4.04) with C (12.01). The actual value for MW_g
  of CH4 should be 16.0425 g/mol or 16.04 when rounded to 2 places.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
… feature/Hg0drydep+updateconfig

This brings @arifein's updates for Hg0 drydep up to dev/14.1.0,
stored in branch feature/Hg0drydep+updateconfig.

We will add some configuration file updates (requested by the Hg & POPs
working in subsequent commits to this branch.

Resolved conflicts in: GeosCore/drydep_mod.F90

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
run/GCClassic/geoschem_config.yml.templates/geoschem_config.yml.Hg
- As described in geoschem/geos-chem PR #1367, we now point to the
  CHEM_INPUTS/FAST_JX/v2022-10-Hg data folder for FAST-JX, as the
  FJX_j2j.dat file was modified.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
run/GCClssic/HEMCO_Config.rc.templates/HEMCO_Config.rc
- Removed BrOX_GC entries (Br*_GC and Br*_TOMCAT)
- Removed JBrO entries (RUNDIR_JBrO and RUNDIR_JNO2)
- Removed AERO_HG2_PARTITION fields
- Updated comments

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
run/GCClassic/createRunDir.sh
- Make sure that we copy README.md instead of README to the run
  directory.  This might have been clobbered by a Git merge.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
run/GCClassic/HEMCO_Config.rc.templates/HEMCO_Config.rc.Hg
- Add GMA_* switches to turn on AMEP/UNEM GMA data by default
- Set WHET switch to false
- Add entries to read AMEP/UNEP GMA data from the netCDF file
  $ROOT/MERCURY/v2022-10/GMA_emissions_Hg.0.25x0.25.2015.nc

These updates were tested with a 1-day Hg simulation, which ran
to completion.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@msulprizio
Copy link
Contributor

A fix was pushed to the HEMCO repository to resolve the TOMAS compile issue. All GCClassic simulations now pass compilation tests:

==============================================================================
GEOS-Chem Classic: Compilation Test Results

Using 8 OpenMP threads
Number of compilation tests: 8
==============================================================================

Compiliation tests:
------------------------------------------------------------------------------
GCClassic........................................Configure & Build......PASS
GCClassic with APM...............................Configure & Build......PASS
GCClassic with BPCH diagnostics..................Configure & Build......PASS
GCClassic with Hg (as a KPP mechanism)...........Configure & Build......PASS
GCClassic with Luo et al wetdep..................Configure & Build......PASS
GCClassic with RRTMG.............................Configure & Build......PASS
GCClassic with TOMAS15...........................Configure & Build......PASS
GCClassic with TOMAS40...........................Configure & Build......PASS

Summary of compilation test results:
------------------------------------------------------------------------------
Complilation tests passed:        8
Complilation tests failed:        0
Complilation tests not completed: 0

TOMAS simulations are still failing the GCClassic execution tests:

==============================================================================
GEOS-Chem Classic: Execution Test Results

Using 24 OpenMP threads
Number of execution tests: 48
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gc_025x03125_CH4_geosfp_47L_na...................Execute Simulation.....PASS
...
gc_4x5_fullchem_TOMAS15_merra2_47L...............Execute Simulation.....FAIL
gc_4x5_fullchem_TOMAS40_merra2_47L...............Execute Simulation.....FAIL
...
gc_4x5_TransportTracers_merra2_LuoWd.............Execute Simulation.....PASS
 
Summary of test results:
------------------------------------------------------------------------------
Execution tests passed: 46
Execution tests failed: 2
Execution tests not yet completed: 0

The issue is an error in hcox_tomas_jeagle_mod.F90.

HEMCO ERROR:  (A) Cannot find option Reduce SS cold water in extension          130
 --> LOCATION: GetExtOpt (hco_extlist_mod)
 
HEMCO ERROR: Error encountered in "HCOX_TOMAS_Jeagle_Init"!
 --> LOCATION:  -> at HCOX_INIT (in module HEMCO/Extensions/hcox_driver_mod.F90
===============================================================================
GEOS-Chem ERROR: Error encountered in "HCOX_Init"!
 -> at HCOI_GC_Init (in module GeosCore/hco_interface_gc_mod.F90)

THIS ERROR ORIGINATED IN HEMCO!  Please check the HEMCO log file for 
additional error messages!
===============================================================================


Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x2b912d26e3ff in ???
#1  0x2b912c411ee9 in sflush
        at ../.././libgfortran/io/unix.h:89
#2  0x638d15 in __hco_interface_gc_mod_MOD_hcoi_gc_init
        at /n/holyscratch01/jacob_lab/msulprizio/GC/rundirs/IntTest_GCC_14.1.0+TOMAS/CodeDir/src/GEOS-Chem/GeosCore/hco_interface_gc_mod.F90:664
#3  0x53f441 in __emissions_mod_MOD_emissions_init
        at /n/holyscratch01/jacob_lab/msulprizio/GC/rundirs/IntTest_GCC_14.1.0+TOMAS/CodeDir/src/GEOS-Chem/GeosCore/emissions_mod.F90:117
#4  0x4062d1 in geos_chem
        at /n/holyscratch01/jacob_lab/msulprizio/GC/rundirs/IntTest_GCC_14.1.0+TOMAS/CodeDir/src/GEOS-Chem/Interfaces/GCClassic/main.F90:732
#5  0x40bfd2 in main
        at /n/holyscratch01/jacob_lab/msulprizio/GC/rundirs/IntTest_GCC_14.1.0+TOMAS/CodeDir/src/GEOS-Chem/Interfaces/GCClassic/main.F90:32
srun: error: holy7c20407: task 0: Segmentation fault (core dumped)

I am working on a fix.

Also updated GCHP/HEMCO_Config.templates/HEMCO_Config.rc.fullchem to ensure
consistency with GCClassic version.

Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
Make sure we only compile for the default and TOMAS options. Also add a
fix to link to the correct gchp executable.

Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
(1) Put two ## in front of 'Tomas' in the collection list to avoid
    activating that collecton for benchmark simulations.
(2) Moved the 'Tomas' collection entries further down in HISTORY.rc so
    that collections are in alphabetical order.
(3) Remove TOMAS species from SpeciesConc collection or else benchmark
    simulations will crash because it can't find those species.
(4) Added code to setupConfigFiles.sh to activate the TOMAS collection
    and to add the TOMAS species to the SpeciesConc list for TOMAS simulations
    only.
(5) Added Tomas collection to GCClassic HISTORY.rc.fullchem

Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
@msulprizio
Copy link
Contributor

GCHP integration tests have now been updated to include TOMAS simulations. All GCHP compilation tests are passing:

GCHP: Compilation Test Results

Number of compilation tests: 3
==============================================================================
 
Compiliation tests:
------------------------------------------------------------------------------
GCHP.............................................Configure & Build......PASS
GCHP with TOMAS15................................Configure & Build......PASS
GCHP with TOMAS40................................Configure & Build......PASS
 
Summary of compilation test results:
------------------------------------------------------------------------------
Complilation tests passed:        3
Complilation tests failed:        0
Complilation tests not completed: 0

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All compilation tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The GCHP execution tests for TOMAS simulations are failing however:

==============================================================================
GCHP: Execution Test Results

Number of execution tests: 5
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gchp_fullchem_benchmark_merra2_c48...............Execute Simulation.....PASS
gchp_fullchem_standard_merra2_c24................Execute Simulation.....PASS
gchp_fullchem_TOMAS15_merra2_c24.................Execute Simulation.....FAIL
gchp_fullchem_TOMAS40_merra2_c24.................Execute Simulation.....FAIL
gchp_TransportTracers_geosfp_c24.................Execute Simulation.....PASS
 
Summary of execution test results:
------------------------------------------------------------------------------
Execution tests passed:        3
Execution tests failed:        2
Execution tests not completed: 0

Further investigation is needed.

@msulprizio
Copy link
Contributor

The error in the GCHP TOMAS simulations is:

======================
 Contents of  CollList
  
 Tomas
 DefaultCollection
  
 
GEOS-Chem ERROR [0000]: Metadata not found for State_Diag field ID: TOMASSOANUMBER. If the
 name in HISTORY.rc has species appended, make sure the species name is preceded by a sing
le underscore. Otherwise, check that the name is listed with all capitals in subroutine Ge
t_Metadata_State_Diag (Headers/state_diag_mod.F90).
 --> LOCATION:  -> at Get_Metadata_State_Diag (in Headers/state_diag_mod.F90)
 
GEOS-Chem ERROR [0000]: Metadata not found for TomasSOAnumber_bin15 in state DIAG
 --> LOCATION: Init_HistoryExportsList (gchp_historyexports_mod.F90)

@BettyCroft
Copy link
Contributor

Hi @msulprizio and @Jourdan-He; Thanks for all your help with these TOMAS updates! I'm about to start work on the TOMAS side of things to move the TOMAS arrays out of sulfate_mod.F90 and into state_chm_mod.F90. As I'm relatively new to this work flow - could you advise me on the steps to follow to make sure that I am modifying the correct version of the code?

@msulprizio
Copy link
Contributor

@BettyCroft My goal is to try to get the TOMAS simulation running in GCHP so we can at least merge these update into dev/14.1.0. It may make sense to address the remaining science issues (raised in #1511) in a future version to avoid delaying this PR further. In any case, I think it makes sense to start the work addressing the sulfur chemistry issues off of the updates in this branch (or once they're merged into dev/14.1.0). I can keep you posted here once I get the TOMAS simulation at least running in GCHP.

@BettyCroft
Copy link
Contributor

Thanks @msulprizio, I'll also tag @theloniuspunk here so that he has an update on these developments.

@msulprizio
Copy link
Contributor

msulprizio commented Nov 18, 2022

GCClassic integration tests are all passing except for TOMAS.

==============================================================================
GEOS-Chem Classic: Execution Test Results

Using 24 OpenMP threads
Number of execution tests: 48
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gc_025x03125_CH4_geosfp_47L_na...................Execute Simulation.....PASS
gc_025x03125_fullchem_geosfp_47L_na..............Execute Simulation.....PASS
gc_05x0625_CH4_merra2_47L_na.....................Execute Simulation.....PASS
gc_05x0625_fullchem_merra2_47L_na................Execute Simulation.....PASS
gc_2x25_aerosol_merra2...........................Execute Simulation.....PASS
gc_2x25_CH4_merra2...............................Execute Simulation.....PASS
gc_2x25_CO2_merra2...............................Execute Simulation.....PASS
gc_2x25_fullchem_merra2..........................Execute Simulation.....PASS
gc_2x25_Hg_merra2................................Execute Simulation.....PASS
gc_2x25_metals_merra2............................Execute Simulation.....PASS
gc_2x25_POPs_BaP_merra2..........................Execute Simulation.....PASS
gc_2x25_tagCH4_merra2............................Execute Simulation.....PASS
gc_2x25_tagCO_merra2.............................Execute Simulation.....PASS
gc_2x25_tagO3_merra2.............................Execute Simulation.....PASS
gc_2x25_TransportTracers_merra2..................Execute Simulation.....PASS
gc_2x25_TransportTracers_merra2_LuoWd............Execute Simulation.....PASS
gc_4x5_aerosol_geosfp............................Execute Simulation.....PASS
gc_4x5_aerosol_merra2............................Execute Simulation.....PASS
gc_4x5_CH4_geosfp................................Execute Simulation.....PASS
gc_4x5_CH4_merra2................................Execute Simulation.....PASS
gc_4x5_fullchem_aciduptake_merra2................Execute Simulation.....PASS
gc_4x5_fullchem_APM_merra2.......................Execute Simulation.....PASS
gc_4x5_fullchem_benchmark_merra2.................Execute Simulation.....PASS
gc_4x5_fullchem_complexSOA_merra2................Execute Simulation.....PASS
gc_4x5_fullchem_complexSOA_SVPOA_merra2..........Execute Simulation.....PASS
gc_4x5_fullchem_geosfp...........................Execute Simulation.....PASS
gc_4x5_fullchem_marinePOA_merra2.................Execute Simulation.....PASS
gc_4x5_fullchem_merra2...........................Execute Simulation.....PASS
gc_4x5_fullchem_merra2_47L.......................Execute Simulation.....PASS
gc_4x5_fullchem_merra2_LuoWd.....................Execute Simulation.....PASS
gc_4x5_fullchem_RRTMG_merra2.....................Execute Simulation.....PASS
gc_4x5_fullchem_TOMAS15_merra2_47L...............Execute Simulation.....FAIL   <-----
gc_4x5_fullchem_TOMAS40_merra2_47L...............Execute Simulation.....FAIL   <-----
gc_4x5_Hg_geosfp.................................Execute Simulation.....PASS
gc_4x5_Hg_merra2.................................Execute Simulation.....PASS
gc_4x5_metals_merra2.............................Execute Simulation.....PASS
gc_4x5_POPs_BaP_geosfp...........................Execute Simulation.....PASS
gc_4x5_POPs_BaP_merra2...........................Execute Simulation.....PASS
gc_4x5_tagCH4_geosfp.............................Execute Simulation.....PASS
gc_4x5_tagCH4_merra2.............................Execute Simulation.....PASS
gc_4x5_tagCO_geosfp..............................Execute Simulation.....PASS
gc_4x5_tagCO_merra2..............................Execute Simulation.....PASS
gc_4x5_tagO3_geosfp..............................Execute Simulation.....PASS
gc_4x5_tagO3_merra2..............................Execute Simulation.....PASS
gc_4x5_TransportTracers_geosfp...................Execute Simulation.....PASS
gc_4x5_TransportTracers_geosfp_LuoWd.............Execute Simulation.....PASS
gc_4x5_TransportTracers_merra2...................Execute Simulation.....PASS
gc_4x5_TransportTracers_merra2_LuoWd.............Execute Simulation.....PASS
 
Summary of test results:
------------------------------------------------------------------------------
Execution tests passed: 46
Execution tests failed: 2
Execution tests not yet completed: 0

The TOMAS simulations are failing with the following error in the GEOS-Chem log file:

At line 6340 of file /n/holyscratch01/jacob_lab/msulprizio/GC/rundirs/IntTest_GCC_14.1.0_TOMAS/CodeDir/src/GEOS-Chem/GeosCore/tomas_mod.F90
Fortran runtime error: Index '2' of dimension 4 of array 'state_diag%tomasmnfixcheck3mass' above upper bound of 1

@BettyCroft
Copy link
Contributor

Hi @msulprizio and @Jourdan-He - I have just submitted a pull request for the bug fix for TOMAS sulfate production rates. Not sure that I named it all that well - it is called Feature/tomaswork and #1569

@msulprizio msulprizio modified the milestones: 14.1.0, 14.2.0 Jan 5, 2023
@msulprizio msulprizio changed the base branch from dev/14.1.0 to main January 5, 2023 15:28
@lizziel lizziel added the topic: GCHP Related to GCHP only label Mar 23, 2023
msulprizio added a commit that referenced this pull request Apr 7, 2023
Resolved conflicts in:
	CHANGELOG.md
	GeosCore/sulfate_mod.F90
	Headers/state_diag_mod.F90
	run/GCHP/HEMCO_Config.rc.templates/HEMCO_Config.rc.fullchem
	run/GCHP/HISTORY.rc.templates/HISTORY.rc.fullchem
	test/GCClassic/intTestCreate.sh#	test/GCHP/intTestCompile_slurm.sh
	test/GCHP/intTestCreate.sh
	test/shared/commonFunctionsForTests.sh

Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
@msulprizio
Copy link
Contributor

This has now been merged into dev/14.2.0.

@msulprizio msulprizio closed this Apr 7, 2023
@msulprizio msulprizio deleted the feature/TOMAS branch April 7, 2023 13:54
@BettyCroft
Copy link
Contributor

Hi @msulprizio, Thank you for all your work with this PR.

msulprizio added a commit to geoschem/GCClassic that referenced this pull request Apr 7, 2023
For more details see

- geoschem/geos-chem#1378
- geoschem/HEMCO#164

Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
@theloniuspunk
Copy link

theloniuspunk commented Apr 7, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: Feature Request New feature or request topic: GCHP Related to GCHP only
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants