June 2023 operating system update on Cannon
The operating system on Cannon was replaced during the annual powerdown from June 5-8, 2023. As FAS Research Computing wrote:
As part of our June 5-8, 2023 MGHPCC Downtime, FASRC will be upgrading the cluster operating system from CentOS 7 to RockyLinux 8. Details as to why the transition is taking place are provided on the downtime page.
The current CentOS 7 operating system went out of support in 2023. If you want to know the nitty-gritty details, see this post. (TL;DR: RedHat Linux was acquired by IBM, and didn't want to support the open-source CentOS project any longer.)
FASRC has chosen to install RockyLinux 8.7 as the successor operating system to CentOS. RockyLinux is a stable fork of RedHat Linux and is bitwise identical with CentOS7.
This presentation is a good overview:
- Switching from CentOS 7 to Rocky 8: How the new operating system will affect FASRC clusters, Town Hall, 4/25/23
The key things that you should know are:
-
All of the software packages (e.g. compilers, netCDF versions, MPI versions, etc.) that you have been currently using under CentOS will no longer work under RockyLinux.
-
FASRC has built several software packages that we need for GEOS-Chem The GEOS-Chem Support Team has also created additional software packages for GEOS-Chem with Spack.
-
If you have built one or more Conda environments for Python packages, these will have to be rebuilt on RockyLinux. Please see our instructions below.
-
FASRC has also released a Singularity container with the prior CentOS 7 environment. This may be useful for you if you need to keep backwards compatibility with code that was compiled before the switchover.
DON'T PANIC! Rest easy, the local GEOS-Chem Support Team has been testing GEOS-Chem and GCHP with the new software modules in order to ensure that these will work under RockyLinux. Results will be posted below.
The GEOS-Chem Support Team has created a new branch named rocky-linux in the cannon-env repository. In this branch you will find updated environment files for GEOS-Chem Classic and GCHP for use with RockyLinux.
Environment files are scripts containing commands to load both FASRC-built software packages and locally-built software packages for RockyLinux. Additional settings, such as environment variables specifying library paths, are also defined in these environment files.
If you have not yet downloaded the cannon-env repository, use this command to create a fresh clone:
$ git clone https://github.com/Harvard-ACMG/cannon-envOr if you already have a clone of the cannon-env repository, then navigate to it
$ cd /path/to/cannon-env # /path/to/cannon-env is the absolute path of your clone of cannon-env
$ git fetch -p https://github.com/Harvard-ACMG/cannon-env
$ git checkout main
$ git pullThen navigate to the envs subdirectory:
$ cd envsand you will the following environment files:
| Environment file | Loads packages for | With these compilers | Software packages |
|---|---|---|---|
gcclassic.rocky+gnu10.env |
GEOS-Chem Classic | gcc, g++, gfortran 10.2.0 | FASRC + Spack |
gcclassic.rocky+gnu10.minimal.env |
GEOS-Chem Classic | gcc, g++. gfortran 10.2.0 | FASRC only |
gchp.rocky+gnu10.env |
GCHP | gcc, g++, gfortran 10.2.0 | FASRC + Spack |
gchp.rocky+gnu10.minimal.env |
GCHP | gcc, g++, gfortran 10.2.0 | FASRC only |
gcclassic.rocky+gnu12.env |
GEOS-Chem Classic | gcc, g++, gfortran 12.2.0 | FASRC + Spack |
gcclassic.rocky+gnu12.minimal.env |
GEOS-Chem Classic | gcc, g++, gfortran 12.2.0 | FASRC only |
gchp.rocky+gnu12.env |
GCHP | gcc, g++, gfortran 12.2.0 | FASRC + Spack |
gchp.rocky+gnu12.minimal.env |
GCHP | gcc, g++, gfortran 12.2.0 | FASRC only |
NOTES:
-
Use the
*rocky+gnu10.envor*rocky+gnu12.envenvironment files in interactive login sessions. These environment files will load the complete set of software packages into your Linux environment. -
Use the
*rocky+gnu10.minimal.envor*rocky+gnu12.minimal.envenvironment files in run scripts that will be submitted to Cannon queues. These environment files will only load the essential software packages needed to run GEOS-Chem Classic or GCHP, without any of the interactive utilities that have been built with Spack. -
We recommend that you copy or link these files to a convenient location (such as your home directory).
To apply the settings contained in an environment file, type
$ source <env-file-name>where <env-file-name> refers to one of the files listed above.
FASRC have built several core packages (aka modules) for RockyLinux. These include compilers, netCDF libraries, MPI libraries, and programming languages (and their dependencies). FASRC-built packages use this naming scheme:
<package-name>/<package-version>-fasrc<build-id>where
-
<package-name>is the package name -
<package-version>is the package version number (usually integersX.Y.Z) -
<build-id>is a 2-digit number (e.g.01,02,03). Most packages will only use01. But if there is a package that has been built with more than one compiler or MPI version, then numbers higher than01will be used to differentiate the builds.
These commands (which are present in all RockyLinux environment files) load packages built with the GNU Compiler Collection 10.2.0:
module purge # Unload all prior loaded modules
module load gcc/10.2.0-fasrc01 # GNU compilers v. 10.2.0
module load openmpi/4.1.0-fasrc01 # OpenMPI (needed for netCDF and GCHP)
module load netcdf-c/4.8.0-fasrc01 # netcdf-c library
module load netcdf-fortran/4.5.3-fasrc01 # netcdf-fortran library
module load flex/2.6.4-fasrc01 # Fast lexical analyzer (needed for KPP)
module load cmake/3.25.2-fasrc01 # CMake (needed to compile GEOS-Chem and HEMCO)Type module list to get a listing of modules that have been loaded:
Currently Loaded Modules:
1) gmp/6.2.1-fasrc01 4) gcc/10.2.0-fasrc01 7) szip/2.1.1-fasrc01 10) netcdf-fortran/4.5.3-fasrc01
2) mpfr/4.1.0-fasrc01 5) openmpi/4.1.0-fasrc01 8) hdf5/1.10.7-fasrc01 11) flex/2.6.4-fasrc01
3) mpc/1.2.1-fasrc01 6) zlib/1.2.11-fasrc01 9) netcdf-c/4.8.0-fasrc01 12) cmake/3.25.2-fasrc01These commands (which are present in the RockyLinux environment files) load modules built with the GNU Compiler Collection 12.2.0:
module purge # Unloads all prior loaded modules
module load gcc/12.2.0-fasrc01 # GNU compilers, v. 12.2.0
module load openmpi/4.1.4-fasrc01 # OpenMPI (needed for netCDF and GCHP)
module load netcdf-fortran/4.6.0-fasrc02 # netCDF-Fortran (loads netCDF-C and HDF5 modules)
module load flex/2.6.4-fasrc01 # Fast lexical analyzer (needed for KPP)
module load cmake/3.25.2-fasrc01 # CMake (needed to compile GEOS-Chem and HEMCO)Type module list to get a listing of all loaded modules:
Currently Loaded Modules:
1) gmp/6.2.1-fasrc01 4) gcc/12.2.0-fasrc01 7) szip/2.1.1-fasrc01 10) netcdf-fortran/4.6.0-fasrc02
2) mpfr/4.2.0-fasrc01 5) openmpi/4.1.4-fasrc01 8) hdf5/1.14.0-fasrc01 11) flex/2.6.4-fasrc01
3) mpc/1.3.1-fasrc01 6) zlib/1.2.13-fasrc01 9) netcdf-c/4.9.2-fasrc01 12) cmake/3.25.2-fasrc01These commands (which are present in the RockyLinux environment files) load modules built with the GNU Compiler Collection 12.2.0:
module purge # Unloads all prior loaded modules
module load gcc/12.2.0-fasrc01 # GNU compilers, v. 12.2.0
module load openmpi/4.1.4-fasrc01 # OpenMPI (needed for netCDF and GCHP)
module load netcdf-fortran/4.6.0-fasrc02 # netCDF-Fortran (loads netCDF-C and HDF5 modules)
module load flex/2.6.4-fasrc01 # Fast lexical analyzer (needed for KPP)
module load cmake/3.25.2-fasrc01 # CMake (needed to compile GEOS-Chem and HEMCO)Type module list to get a listing of all loaded modules:
### Core packages for GNU 12.2.0 compilers
These commands (which are present in the [RockyLinux environment files](https://github.com/Harvard-ACMG/knowledge-base/wiki/June-2023-operating-system-update-on-Cannon#environment-files-for-rockylinux)) load modules built with the GNU Compiler Collection 12.2.0:
```console
# Load FASRC-built modules
module load intel/23.0.0-fasrc01 # icc / i++ / gfortran
module load intelmpi/2021.8.0-fasrc01 # MPI
module load netcdf-fortran/4.6.0-fasrc03 # netCDF-Fortran
module load flex/2.6.4-fasrc01 # Flex lexer (needed for KPP)
module load cmake/3.25.2-fasrc01 # CMake (needed to compile)Type module list to get a listing of all loaded modules:
Currently Loaded Modules:
1) intel/23.0.0-fasrc01 4) szip/2.1.1-fasrc02 7) netcdf-fortran/4.6.0-fasrc03
2) intelmpi/2021.8.0-fasrc01 5) hdf5/1.14.0-fasrc02 8) flex/2.6.4-fasrc01
3) zlib/1.2.11-fasrc02 6) netcdf-c/4.9.2-fasrc02 9) cmake/3.25.2-fasrc01The following modules are often needed for post-processing and analysis:
module load IDL/8.7.2-fasrc01 # IDL language (needed for GAMAP)
module load R/4.2.2-fasrc01 # R language
module load matlab/R2022b-fasrc01 # matlab languageTo find other available FASRC-built packages, use this command:
$ module availTo find more information about a specific FASRC-built package, type:
$ module spider <package-name>where <package-name> is the name of the package you are interested in.
In addition to the FASRC-built software packages that we previously discussed, the GEOS-Chem Support Team has built the following local software packages. These will be loaded when you source one of the environment files listed above.
The Earth System Modeling Framework (ESMF) is a library required by GCHP. See the table below for the available ESMF versions for RockyLinux:
-
<ROOT>refers to the path/n/jacob_lab/Lab/seasfs01/Lab/RockyLinux.
| Package | Version | Compiler | Path | Needed for |
|---|---|---|---|---|
| ESMF | 8.1.1 | GNU 10.2.0 | <ROOT>/ESMF/ESMF_8_1_1/INSTALL_gfortran10_openmpi4 |
GCHP |
| ESMF | 8.1.1 | GNU 12.2.0 | <ROOT>/ESMF/ESMF_8_1_1/INSTALL_gfortran12_openmpi4 |
GCHP |
| ESMF | 8.4.2 | GNU 10.2.0 | <ROOT>/ESMF/ESMF_8_1_1/INSTALL_gnu10_openmpi4 |
GCHP |
| ESMF | 8.4.2 | GNU 12.2.0 | <ROOT>/ESMF/ESMF_8_1_1/INSTALL_gnu12_openmpi4 |
GCHP |
When you source one of the GCHP environment files for RockyLinux, several environment variables will be defined that will tell GCHP where to find ESMF.
We have also built several software packages using the Spack package manager. In general, GEOS-Chem and GCHP will only need to use FASRC core packages such as compilers, netCDF, and MPI libraries. The Spack-built packages provide useful tools for post-processing and analysis.
Spack-built packages can be loaded with the same module load commands as the FASRC core packages. Several relevant Spack-built packages will be loaded into your environment when you source [one of the environment files described above]
](https://github.com/Harvard-ACMG/knowledge-base/wiki/June-2023-operating-system-update-on-Cannon#environment-files-for-rockylinux).
If you type module list, you may see many Spack-built packages in the printout. Several of these are dependencies for other packages, and these can be ignored. The Spack-built packages that you will probably use most frequently with GEOS-Chem are:
| Package | Description | Load package for GNU 10.2.0 | Load package for GNU 12.2.0 |
|---|---|---|---|
bison |
Parser, needed by KPP. | module load bison/3.8.2-gcc-10.2.0 |
module load bison/3.8.2-gcc-12.2.0 |
cdo |
Climate Data Operators | spack load cdo%gcc@10.2.0 |
spack load cdo%gcc@12.2.0 |
cgdb |
User-friendly front-end for gdb | module load cgdb/0.7.1-gcc-10.2.0 |
module load cgdb/0.7.1-gcc-12.2.0 |
colordiff |
Colorizes output from diff
|
module load colordiff/1.0.21-gcc-10.2.0 |
module load colordiff/1.0.21-gcc-12.2.0 |
gdb |
The GNU debugger | module load gdb/8.2.19-gcc-10.2.0 |
module load gdb/8.2.19-gcc-10.2.0 |
nco |
The netCDF operators | module load nco/5.1.4-gcc-10.2.0 |
module load cdo/5.1.4-gcc-12.2.0 |
ncview |
netCDF file viewer | module load ncview/2.1.8-gcc-10.2.0 |
module load ncview/2.1.8-gcc-12.2.0 |
xterm |
X11 terminal window | spack load xterm%gcc@10.2.0 |
spack load xterm%gcc@10.2.0 |
Note that the Spack-built packages use this naming scheme:
<package-name>/<version>-<compiler-name>-<compiler-version>where:
-
<package-name>is the name of a Spack-built software package (e.g.cdo) -
<package-version>is the version number of a Spack-built software package -
<compiler-name>is the name of the compiler. This will always begcc, which refers to the GNU Compiler Collection (and not GEOS-Chem Classic). -
<compiler-version>is the version number of the compiler. This will be either10.2.0or12.2.0.
Also note, the Spack configuration files (compilers.yaml, modules.yaml, and packages.yaml) in our spack-env repository have been set up in order to ensure that packages that require netCDF (e.g. cdo, nco, ncview`) will use the FASRC-built netCDF modules. Otherwise, Spack start installing a new version of netCDF (with dependencies) that will cause package conflicts.
We have built the following Python environments for RockyLinux:
| Environment | Used with |
|---|---|
gcpy_env |
GCPy, the GEOS-Chem Python toolkit |
imi_env |
Integrated Methane Inversion |
We encourage users to test with their own Python scripts and to report any issues.
If you wish to use our Python environments for RockyLinux, remove the following lines at the end of your ~/.bashrc file. as shown below:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/n/home09/ryantosca/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/n/home09/ryantosca/miniconda3/etc/profile.d/conda.sh" ]; then
. "/n/home09/ryantosca/miniconda3/etc/profile.d/conda.sh"
else
export PATH="/n/home09/ryantosca/miniconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<Then do the following steps:
$ cd /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/bin
$ ./conda initThis will add the following lines to your ~/.bashrc file:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/etc/profile.d/conda.sh" ]; then
. "/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/etc/profile.d/conda.sh"
else
export PATH="/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/bin:$PATH"
fi
fi
unset __conda_setupThen log out of Cannon and log back in, and the changes should take effect.
Follow these instructions to recreate your current Conda environment on RockyLinux:
-
Before the switchover (June 5-8), activate your current Conda environment and export the package list to a YAML file:
$ conda activate my-env $ conda env export > my-environment.yml $ conda deactivate
-
Log into Cannon Then use the following commands:
$ conda create --name my-new-env $ conda activate my-new-env $ conda env update --name root --file my-environment.yml $ conda deactivate my-new-env
-
After the switchover to RockyLinux is finished, delete your original conda environment:
$ conda remove --name my-env --all
When Cannon was powered up on June 9, 2023, the RockyLinux test cluster ceased to exist. All Cannon nodes are now running RockyLinux. Therefore, we have removed all references to the RockyLinux test cluster on this page in order to avoid confusion.
Once the operating system switchover to RockyLinux 8.7 has been completed, you may encounter warning messages such as:
WARNING: POSSIBLE DNS SPOOFING DETECTED!and/or
The RSA host key for login.rc.fas.harvard.edu has changed error messages.After an update of nodes the SSH key fingerprint of a node may change. This will, in turn, cause an error when you next try to log into that node as your locally stored key will no longer match.
To clear the errors, follow these instructions on the FASRC docs site.
When you source the gcclassic.rocky+gnu*.env or gchp.rocky+gnu*.env environment files, you will see this warning printed to the screen.
The following have been reloaded with a version change:
1) netcdf-c/X.Y.Z-fasrc01 => netcdf-c/X.Y.Z-gcc-A.B.C
The following have been reloaded with a version change:
1) zlib/1.2.11-fasrc01 => zlib/X.Y.Z-gcc-A.B.Cwhere X.Y.Z and A.B.C are version numbers.
You may safely ignore this. The message arises because Spack assigns its own version number to externally-loaded modules (such as the FASRC netCDF and zlib modules). In reality, netcdf-c/X.Y.Z-gcc.A.B.C points to the FASRC module netcdf-c/X.Y.Z.fasrc01. But the Lmod module manager (which is used to load the modules into your Linux environment) thinks that these are two different modules and thus prints the warning messages.
After the switchover to RockyLinux happens, login nodes will restrict you to using 1 core and 4GB of memory. For more intensive applications, you can open an interactive job or submit a batch job on a computational node.
Here is some important information for those who need to use the R programming languge on RockyLinux.
Steve Wofsy wrote:
There are some challenges in using R on the upcoming Rocky 8 installation. Since there is not a consistent set of the required modules for geospatial analysis (GDAL, GEOS, PROJ, UNUNITS,...), it is basically impossible to install packages terra, raster, etc on one's own. We succeeded in getting the spack package manager to work, but there were still challenges in specifying paths to the libraries. So we have adopted the geospatial docker container that is available from git. (Anyone wanting access to the spackR library collection can ask me for instructions on how to use it and not disable the bash shell...). Thanks to Jason Wells for making this happen.
I have attached the instructions for running R from the container. The docker has transparent access to the cannon file system. Also, there is a hack included to allow you to run jobs on the cluster from the container. The relevant files are in
/n/holylfs04/LABS/wofsy_lab/Everyoneand can be accessed by anyone on the cluster. The documentation for setting up this framework and using it is developing, but this text file makes it easy:
/n/holylfs04/LABS/wofsy_lab/Everyone/rocky_docker_run_jobs_export.txt
The GEOS-Chem Support Team has run several GEOS-Chem and GCHP simulations in order to determine:
- Can GEOS-Chem Classic and GCHP run with the new compilers/software modules for RockyLinux?
- Can GCHP run across multiple nodes with the new compilers/software modules for RockyLinux?
- What are the magnitude of differences in GEOS-Chem/GCHP runs done on CentOS 7 vs. RockyLinux, with the same compiler?
- What are the magnitude of differences in GEOS-Chem/GCHP runs done on RockyLinux, but with different compilers?
In order to answer the above questions, the GEOS-Chem Support Team has run the following 1-month fullchem_benchmark simulations on both the RockyLinux 8.7 and CentOS 7 operating systems, with GNU 10.2.0 and GNU 12.2.0 compilers. GEOS-Chem and GCHP were checked out at the tag 14.2.0-alpha.11.
| Run | OS | Model | Res. | Compiler | Cores (Nodes) | Notes |
|---|---|---|---|---|---|---|
| 1 | CentOS | GC Classic | 4x5 | GNU 10 | 48 (1) | Represents "status quo" on Cannon |
| 2 | RockyLinux | GC Classic | 4x5 | GNU 10 | 48 (1) | To check against Run 1 for numerical drift |
| 3 | RockyLinux | GC Classic | 4x5 | GNU 12 | 48 (1) | To check against Runs 1 & 2 for numerical drift |
| 4 | CentOS | GCHP | c48 | GNU 10 | 96 (2) | Represents "status quo" on Cannon |
| 5 | RockyLinux | GCHP | c48 | GNU 10 | 96 (2) | To check against Run 4 for numerical drift |
| 6 | RockyLinux | GCHP | c48 | GNU 12 | 96 (2) | To check against Runs 4 & 5 for numerical drift |
We summarize the results of our tests below:
-
Brief description: Compares GC Classic runs on CentOS 7 and RockyLinux 8.7 using the same compiler version.
-
Results: Very small differences in most species, as shown by the mass table and selected printouts. OH increased by 0.11%. Likely attributed to the differences in numerical libraries used by the compilers between CentOS and RockyLinux
Total atmospheric mass of species: Run 2 vs Run1
-
Takeaways:
- Brief description: Compares GC Classic runs on CentOS 7 and RockyLinux 8.7 using different compiler versions.
-
Results: Because Run 3 is identical to Run 2, the comparison of Run 3 vs. Run 1 gives identical results to Run 2 vs. Run 1. This is evident in the table of OH metrics shown below:
############################################################################### ### OH Metrics ### ### Left column Right column: ### Ref = GCC_centos_gnu10 (Run 1) Ref = GCC_centos_gnu10 (Run 1) ### Dev = GCC_rocky_gnu10 (Run 2) Dev = GCC_rocky_gnu12 (Run 3) ############################################################################### ------------------------------------------------------------ Global mass-weighted OH concentration [10^5 molec cm^-3] ------------------------------------------------------------ Run 1 : 13.07896632314 Run 1 : 13.07896632314 Run 2 : 13.09341931491 Run 3 : 13.09341931491 Abs diff : 0.01445299178 Abs diff : 0.01445299178 % diff : 0.110506 % diff : 0.110506 ------------------------------------------------------------ CH3CCl3 (aka MCF) lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Run 1 : 4.753380 Run 1 : 4.753380 Run 2 : 4.746909 Run 3 : 4.746909 Abs diff : -0.006471 Abs diff : -0.006471 % diff : -0.136133 % diff : -0.136133 ------------------------------------------------------------ CH4 lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Run 1 : 8.019566 Run 1 : 8.019566 Run 2 : 8.008516 Run 3 : 8.008516 Abs diff : -0.011050 Abs diff : -0.011050 % diff : -0.137782 % diff : -0.137782
- Takeaways: Compiling GEOS-Chem Classic with the GNU 10.2.0 compilers or the GNU 12.2.0 yields identical numerical results. The difference in the OH metrics are small (approx 0.1% or 0.001 absolute). This may be attributed to the difference in onboard libraries used by the compilers between CentOS and RockyLinux.
- Brief description: Compares GC Classic runs using 2 different compiler versions on RockyLinux.
-
Results: 100% identical
################################################################################ ### Benchmark summary table ### ### ### ### Ref = GCC_rocky_gnu10 ### ### Dev = GCC_rocky_gnu12 ### ################################################################################ ------------------------------------------------------------------------------- AerosolMass: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- Aerosols: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- Emissions: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- JValues: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- Metrics: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- SpeciesConc: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- StateMet: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10
- Takeaways: Using RockyLinux 8.7 but Switching GNU compiler versions does not affect simulation results. This is likely due to the fact that both GNU 10.2.0 and GNU 12.2.0 compilers use the same underlying system libraries.
-
Brief description: Compares GCHP runs on CentOS 7 and RockyLinux 8.7 using the same compiler.
-
Results: OH metrics change by +/- 0.001% or less.
############################################################################### ### OH Metrics ### Ref = GCHP_centos_gnu10 ### Dev = GCHP_rocky_gnu10 ############################################################################### ------------------------------------------------------------ Global mass-weighted OH concentration [10^5 molec cm^-3] ------------------------------------------------------------ Ref : 13.16007998549 Dev : 13.16022032913 Abs diff : 0.00014034365 % diff : 0.001066 ------------------------------------------------------------ CH3CCl3 (aka MCF) lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Ref : 4.717519 Dev : 4.717485 Abs diff : -0.000034 % diff : -0.000722 ------------------------------------------------------------ CH4 lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Ref : 7.958901 Dev : 7.958847 Abs diff : -0.000054 % diff : -0.000684
-
The Dev/Ref ratios for O3 varies by +/- 0.01% at the surface.. Other species show similar behavior.
-
The Dev/Ref ratios for O3 varies by +/- 0.003% in zonal means. Other species show similar behavior.
-
Takeaways: Using the same compiler on different operating systems causes very small differences in GCHP output. This is probably attributed to differences in software libraries between CentOS 7 and RockyLinux 8.1.1. One note: The ESMF version used on RockyLinux (8.1.1) is newer than that used on CentOS 7. But the differences observed do not indicate any systematic bias.
- Brief description: Compares GCHP runs on CentOS 7 and RockyLinux 8.7 using different compiler versions.
-
Results: Because Run 6 is identical to Run 5, then the comparison of Run 6 vs. Run 4 is also identical to Run 5 vs. Run 4.
############################################################################### ### OH Metrics ### ### Left column: Right column: ### Ref = GCHP_centos_gnu10 (Run 4) Ref = GCHP_centos_gnu10 (Run 4) ### Dev = GCHP_rocky_gnu10 (Run 5) Dev = GCHP_rocky_gnu12 (Run 6) ############################################################################### ------------------------------------------------------------ Global mass-weighted OH concentration [10^5 molec cm^-3] ------------------------------------------------------------ Run 4 : 13.16007998549 Run 4 : 13.16007998549 Run 5 : 13.16022032913 Run 6 : 13.16022032913 Abs diff : 0.00014034365 Abs diff : 0.00014034365 % diff : 0.001066 % diff : 0.001066 ------------------------------------------------------------ CH3CCl3 (aka MCF) lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Ref : 4.717519 Run 4 : 4.717519 Dev : 4.717485 Run 6 : 4.717485 Abs diff : -0.000034 Abs diff : -0.000034 % diff : -0.000722 % diff : -0.000722 ------------------------------------------------------------ CH4 lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Ref : 7.958901 Run 4 : 7.958901 Dev : 7.958847 Run 6 : 7.958847 Abs diff : -0.000054 Abs diff : -0.000054 % diff : -0.000684 % diff : -0.000684
- Takeaways: Runs 5 and 6 are identical. OH changes by +0.001%, which is negligble. This is a smaller amount of change as seen in GEOS-Chem Classic, however.
- Brief description: Compares GCHP runs using 2 different compiler versions on RockyLinux.
-
Results: 100% identical
################################################################################ ### Benchmark summary table ### ### ### ### Ref = GCHP_rocky_gnu10 ### ### Dev = GCHP_rocky_gnu12 ### ################################################################################ ------------------------------------------------------------------------------- AerosolMass: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- Aerosols: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- Emissions: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- JValues: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- Metrics: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- SpeciesConc: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- StateMet: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10
- Takeaways: Using RockyLinux 8.7 but Switching GNU compiler versions does not affect simulation results. This is likely due to the fact that both GNU 10.2.0 and GNU 12.2.0 compilers use the same underlying system libraries.