June 2023 operating system update on Cannon
The operating system on Cannon is slated to be replaced during the annual powerdown from June 5-8, 2023. Asin FAS Research Computing wrote:
As part of our June 5-8, 2023 MGHPCC Downtime, FASRC will be upgrading the cluster operating system from CentOS 7 to RockyLinux 8. Details as to why the transition is taking place are provided on the downtime page.
The current CentOS 7 operating system is going out of support in 2023. If you want to know the nitty-gritty details, see this post. (TL;DR: RedHat Linux was acquired by IBM, and didn't want to support the open-source CentOS project any longer.)
FASRC has chosen to install RockyLinux 8.7 as the successor operating system to CentOS. RockyLinux is a stable fork of RedHat Linux and is bitwise identical with CentOS7.
This presentation is a good overview:
- Switching from CentOS 7 to Rocky 8: How the new operating system will affect FASRC clusters, Town Hall, 4/25/23
The key things that you should know are:
-
All of the software packages (e.g. compilers, netCDF versions, MPI versions, etc.) that you have been currently using under CentOS will no longer work under RockyLinux.
-
FASRC has built several software packages that we need for GEOS-Chem The GEOS-Chem Support Team has also created additional software packages for GEOS-Chem with Spack.
-
If you have built one or more Conda environments for Python packages, these will have to be rebuilt on RockyLinux. Please see our instructions below.
-
FASRC has also released a Singularity container with the prior CentOS 7 environment. This may be useful for you if you need to keep backwards compatibility with code that was compiled before the switchover.
DON'T PANIC! Rest easy, the local GEOS-Chem Support Team has been testing GEOS-Chem and GCHP with the new software modules in order to ensure that these will work under RockyLinux. Results will be posted below.
YES! FASRC has created a RockyLinux test cluster (rockylogin.rc.fas.harvard.edu) that you can use for testing. See below for detailed usage instructions.
The GEOS-Chem Support Team has created a new branch named rocky-linux in the cannon-env repository. In this branch you will find updated environment files for GEOS-Chem Classic and GCHP for use with RockyLinux.
Environment files are scripts containing commands to load both FASRC-built software packages and locally-built software packages for RockyLinux. Additional settings, such as environment variables specifying library paths, are also defined in these environment files.
If you have not yet downloaded the cannon-env repository, use this command to create a fresh clone:
$ git clone -b rocky-linux https://github.com/Harvard-ACMG/cannon-envOr if you already have a clone of the cannon-env repository, then navigate to it
$ cd /path/to/cannon-env # /path/to/cannon-env is the absolute path of your clone of cannon-env
$ git fetch -p https://github.com/Harvard-ACMG/cannon-env
$ git checkout rocky-linux Then navigate to the envs subdirectory:
$ cd envsand you will the following environment files:
| Environment file | Loads packages for | With these compilers | Software packages |
|---|---|---|---|
gcclassic.rocky+gnu10.env |
GEOS-Chem Classic | gcc, g++, gfortran 10.2.0 | FASRC + Spack |
gcclassic.rocky+gnu10.minimal.env |
GEOS-Chem Classic | gcc, g++. gfortran 10.2.0 | FASRC only |
gchp.rocky+gnu10.env |
GCHP | gcc, g++, gfortran 10.2.0 | FASRC + Spack |
gchp.rocky+gnu10.minimal.env |
GCHP | gcc, g++, gfortran 10.2.0 | FASRC only |
gcclassic.rocky+gnu12.env |
GEOS-Chem Classic | gcc, g++, gfortran 12.2.0 | FASRC + Spack |
gcclassic.rocky+gnu12.minimal.env |
GEOS-Chem Classic | gcc, g++, gfortran 12.2.0 | FASRC only |
gchp.rocky+gnu12.env |
GCHP | gcc, g++, gfortran 12.2.0 | FASRC + Spack |
gchp.rocky+gnu12.minimal.env |
GCHP | gcc, g++, gfortran 12.2.0 | FASRC only |
NOTES:
-
Use the
*rocky+gnu10.envor*rocky+gnu12.envenvironment files in interactive login sessions. These environment files will load the complete set of software packages into your Linux environment. -
Use the
*rocky+gnu10.minimal.envor*rocky+gnu12.minimal.envenvironment files in run scripts that will be submitted to Cannon queues. These environment files will only load the essential software packages needed to run GEOS-Chem Classic or GCHP, without any of the interactive utilities that have been built with Spack. -
We recommend that you copy or link these files to a convenient location (such as your home directory).
To apply the settings contained in an environment file, type
$ source <env-file-name>where <env-file-name> refers to one of the files listed above.
NOTE: You must first log into the RockyLinux test cluster (as described below) in order to use the software packages listed below. Once the switchover occurs in June 2023, then these software packages can be used on all Cannon login and computational nodes.
FASRC have built several core packages (aka modules) for RockyLinux. These include compilers, netCDF libraries, MPI libraries, and programming languages (and their dependencies). FASRC-built packages use this naming scheme:
<package-name>/<package-version>-fasrc<build-id>where
-
<package-name>is the package name -
<package-version>is the package version number (usually integersX.Y.Z) -
<build-id>is a 2-digit number (e.g.01,02,03). Most packages will only use01. But if there is a package that has been built with more than one compiler or MPI version, then numbers higher than01will be used to differentiate the builds.
These commands (which are present in all RockyLinux environment files) load packages built with the GNU Compiler Collection 10.2.0:
module purge # Unload all prior loaded modules
module load gcc/10.2.0-fasrc01 # GNU compilers v. 10.2.0
module load openmpi/4.1.0-fasrc01 # OpenMPI (needed for netCDF and GCHP)
module load netcdf-c/4.8.0-fasrc01 # netcdf-c library
module load netcdf-fortran/4.5.3-fasrc01 # netcdf-fortran library
module load flex/2.6.4-fasrc01 # Fast lexical analyzer (needed for KPP)
module load cmake/3.25.2-fasrc01 # CMake (needed to compile GEOS-Chem and HEMCO)Type module list to get a listing of modules that have been loaded:
Currently Loaded Modules:
1) gmp/6.2.1-fasrc01 4) gcc/10.2.0-fasrc01 7) szip/2.1.1-fasrc01 10) netcdf-fortran/4.5.3-fasrc01
2) mpfr/4.1.0-fasrc01 5) openmpi/4.1.0-fasrc01 8) hdf5/1.10.7-fasrc01 11) flex/2.6.4-fasrc01
3) mpc/1.2.1-fasrc01 6) zlib/1.2.11-fasrc01 9) netcdf-c/4.8.0-fasrc01 12) cmake/3.25.2-fasrc01These commands (which are present in the RockyLinux environment files) load modules built with the GNU Compiler Collection 12.2.0:
module purge # Unloads all prior loaded modules
module load gcc/12.2.0-fasrc01 # GNU compilers, v. 12.2.0
module load openmpi/4.1.4-fasrc01 # OpenMPI (needed for netCDF and GCHP)
module load netcdf-fortran/4.6.0-fasrc02 # netCDF-Fortran (loads netCDF-C and HDF5 modules)
module load flex/2.6.4-fasrc01 # Fast lexical analyzer (needed for KPP)
module load cmake/3.25.2-fasrc01 # CMake (needed to compile GEOS-Chem and HEMCO)Type module list to get a listing of all loaded modules:
Currently Loaded Modules:
1) gmp/6.2.1-fasrc01 4) gcc/12.2.0-fasrc01 7) szip/2.1.1-fasrc01 10) netcdf-fortran/4.6.0-fasrc02
2) mpfr/4.2.0-fasrc01 5) openmpi/4.1.4-fasrc01 8) hdf5/1.14.0-fasrc01 11) flex/2.6.4-fasrc01
3) mpc/1.3.1-fasrc01 6) zlib/1.2.13-fasrc01 9) netcdf-c/4.9.2-fasrc01 12) cmake/3.25.2-fasrc01The following modules are often needed for post-processing and analysis:
module load IDL/8.7.2-fasrc01 # IDL language (needed for GAMAP)
module load R/4.2.2-fasrc01 # R language
module load matlab/R2022b-fasrc01 # matlab languageTo find other available FASRC-built packages, use this command:
$ module availTo find more information about a specific FASRC-built package, type:
$ module spider <package-name>where <package-name> is the name of the package you are interested in.
NOTE: You must first log into the RockyLinux test cluster (as described below) in order to use the software packages listed below. Once the switchover occurs in June 2023, then these software packages can be used on all Cannon login and computational nodes.
In addition to the FASRC-built software packages that we previously discussed, the GEOS-Chem Support Team has built the following local software packages. These will be loaded when you source one of the environment files listed above.
The Earth System Modeling Framework (ESMF) is a library required by GCHP. See the table below for the available ESMF versions for RockyLinux:
-
<ROOT>refers to the path/n/jacob_lab/Lab/seasfs01/Lab/RockyLinux.
| Package | Version | Compiler | Path | Needed for |
|---|---|---|---|---|
| ESMF | 8.1.1 | GNU 10.2.0 | <ROOT>/ESMF/ESMF_8_1_1/INSTALL_gfortran10_openmpi4 |
GCHP |
| ESMF | 8.1.1 | GNU 12.2.0 | <ROOT>/ESMF/ESMF_8_1_1/INSTALL_gfortran12_openmpi4 |
GCHP |
When you source either of the GCHP environment files for RockyLinux (gchp.rocky+gnu10.env or gchp.rocky+gnu12.env), several environment variables will be defined that will tell GCHP where to find ESMF.
We have also built several software packages using the Spack package manager. In general, GEOS-Chem and GCHP will only need to use FASRC core packages such as compilers, netCDF, and MPI libraries. The Spack-built packages provide useful tools for post-processing and analysis.
Spack-built packages can be loaded with the same module load commands as the FASRC core packages. Several relevant Spack-built packages will be loaded into your environment when you source [one of the environment files described above]
](https://github.com/Harvard-ACMG/knowledge-base/wiki/June-2023-operating-system-update-on-Cannon#environment-files-for-rockylinux).
If you type module list, you may see many Spack-built packages in the printout. Several of these are dependencies for other packages, and these can be ignored. The Spack-built packages that you will probably use most frequently with GEOS-Chem are:
| Package | Description | Load package for GNU 10.2.0 | Load package for GNU 12.2.0 |
|---|---|---|---|
bison |
Parser, needed by KPP. | module load bison/3.8.2-gcc-10.2.0 |
module load bison/3.8.2-gcc-12.2.0 |
cdo |
Climate Data Operators | spack load cdo%gcc@10.2.0 |
spack load cdo%gcc@12.2.0 |
cgdb |
User-friendly front-end for gdb | module load cgdb/0.7.1-gcc-10.2.0 |
module load cgdb/0.7.1-gcc-12.2.0 |
colordiff |
Colorizes output from diff
|
module load colordiff/1.0.21-gcc-10.2.0 |
module load colordiff/1.0.21-gcc-12.2.0 |
gdb |
The GNU debugger | module load gdb/8.2.19-gcc-10.2.0 |
module load gdb/8.2.19-gcc-10.2.0 |
nco |
The netCDF operators | module load nco/5.1.4-gcc-10.2.0 |
module load cdo/5.1.4-gcc-12.2.0 |
ncview |
netCDF file viewer | module load ncview/2.1.8-gcc-10.2.0 |
module load ncview/2.1.8-gcc-12.2.0 |
Note that the Spack-built packages use this naming scheme:
<package-name>/<version>-<compiler-name>-<compiler-version>where:
-
<package-name>is the name of a Spack-built software package (e.g.cdo) -
<package-version>is the version number of a Spack-built software package -
<compiler-name>is the name of the compiler. This will always begcc, which refers to the GNU Compiler Collection (and not GEOS-Chem Classic). -
<compiler-version>is the version number of the compiler. This will be either10.2.0or12.2.0.
Also note, the Spack configuration files (compilers.yaml, modules.yaml, and packages.yaml) in our spack-env repository have been set up in order to ensure that packages that require netCDF (e.g. cdo, nco, ncview`) will use the FASRC-built netCDF modules. Otherwise, Spack start installing a new version of netCDF (with dependencies) that will cause package conflicts.
NOTE: You must first log into the RockyLinux test cluster (as described below) in order to use the Python environments listed below. Once the switchover occurs in June 2023, then these Python environments can be used on all Cannon login and computational nodes.
We have built the following Python environments for RockyLinux:
| Environment | Used with |
|---|---|
gcpy_env |
GCPy, the GEOS-Chem Python toolkit |
imi_env |
Integrated Methane Inversion |
We encourage users to test with their own Python scripts and to report any issues.
If you wish to use or test our Python environments on the RockyLinux test cluster, first comment out the following lines at the end of the bashrc file. as shown below:
## >>> conda initialize >>>
## !! Contents within this block are managed by 'conda init' !!
#__conda_setup="$('/n/home09/ryantosca/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
#if [ $? -eq 0 ]; then
# eval "$__conda_setup"
#else
# if [ -f "/n/home09/ryantosca/miniconda3/etc/profile.d/conda.sh" ]; then
# . "/n/home09/ryantosca/miniconda3/etc/profile.d/conda.sh"
# else
# export PATH="/n/home09/ryantosca/miniconda3/bin:$PATH"
# fi
#fi
#unset __conda_setup
## <<< conda initialize <<<then add these lines:
### ROCKY TESTING
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/etc/profile.d/conda.sh" ]; then
. "/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/etc/profile.d/conda.sh"
else
export PATH="/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/python/miniconda/bin:$PATH"
fi
fi
unset __conda_setupIf you wish to resume using existing Conda environments on CentOS, then comment out the lines for RockyLinux and uncomment the lines for CentOS.
Follow these instructions to recreate your current Conda environment on RockyLinux: `
-
Before the switchover (June 5-8), activate your current Conda environment and export the package list to a YAML file:
$ conda activate my-env $ conda env export > my-environment.yml $ conda deactivate
-
Log into the the RockyLinux test cluster. Then use the following commands:
$ conda create --name my-new-env $ conda activate my-new-env $ conda env update --name root --file my-environment.yml $ conda deactivate my-new-env
-
After the switchover to RockyLinux is finished, delete your original conda environment:
$ conda remove --name my-env --all
NOTE: The RockyLinux test cluster will cease to exist after the Cannon shutdown is complete. Then all nodes will be running RockyLinux.
Follow these steps if you wish to use the RockyLinux test cluster to test your software.
-
Open a new terminal window from your PC or Mac. DO NOT LOG IN FROM ANOTHER CANNON NODE (this will cause module conflicts).
-
SSH into the RockyLogin test cluster. Replace
USERNAMEwith your Cannon username.$ ssh -YA -o StrictHostKeyChecking=accept-new USERNAME@rockylogin.rc.fas.harvard.eduThe
-o StrictHostKeyChecking=accept-newis needed in order to prevent you from typing "yes" each time you use SSH. This is because the RockyLogin cluster is using a slightly different IP address than the regular Cannon cluster which causes SSH key validation issues. -
Once you have logged in, use this command to start an interactive session on the
rockypartition.$ salloc --x11=all -c 8 -N 1 --mem=32000 -t 0-08:00 -p rocky --constraint=intel && ssh -CY $SLURM_JOB_NODELISTThis ensures that you will be placed onto an Intel node (w/ Cascade Lake CPUs) running RockyLinux. As it so happens, the login nodes on
rockylogin.fas.rc.harvard.eduuse AMD CPUs, which are unsuitable for GEOS-Chem and GCHP. -
Source one of the environment files listed above, such as:
$ source /path/to/gcclassic.rocky+gnu10.env # Load GCClassic environment file for Rocky Linux and GNU 10.2.0 compilersIf you wish to use GCHP, make sure you use one of the GCHP environment files above.
At this point you should be able to build and run GEOS-Chem, GCHP, and other software.
For more information about the RockyLinux test cluster, see the FASRC RockyLinux transition guide.
The GEOS-Chem Support Team has run several GEOS-Chem and GCHP simulations in order to determine:
- Can GEOS-Chem Classic and GCHP run with the new compilers/software modules for RockyLinux?
- Can GCHP run across multiple nodes with the new compilers/software modules for RockyLinux?
- What are the magnitude of differences in GEOS-Chem/GCHP runs done on CentOS 7 vs. RockyLinux, with the same compiler?
- What are the magnitude of differences in GEOS-Chem/GCHP runs done on RockyLinux, but with different compilers?
In order to answer the above questions, the GEOS-Chem Support Team has run the following 1-month fullchem_benchmark simulations on both the RockyLinux 8.7 and CentOS 7 operating systems, with GNU 10.2.0 and GNU 12.2.0 compilers. GEOS-Chem and GCHP were checked out at the tag 14.2.0-alpha.11.
| Run | OS | Model | Res. | Compiler | Cores (Nodes) | Notes |
|---|---|---|---|---|---|---|
| 1 | CentOS | GC Classic | 4x5 | GNU 10 | 48 (1) | Represents "status quo" on Cannon |
| 2 | RockyLinux | GC Classic | 4x5 | GNU 10 | 48 (1) | To check against Run 1 for numerical drift |
| 3 | RockyLinux | GC Classic | 4x5 | GNU 12 | 48 (1) | To check against Runs 1 & 2 for numerical drift |
| 4 | CentOS | GCHP | c48 | GNU 10 | 96 (2) | Represents "status quo" on Cannon |
| 5 | RockyLinux | GCHP | c48 | GNU 10 | 96 (2) | To check against Run 4 for numerical drift |
| 6 | RockyLinux | GCHP | c48 | GNU 12 | 96 (2) | To check against Runs 4 & 5 for numerical drift |
We summarize the results of our tests below:
-
Brief description: Compares GC Classic runs on CentOS 7 and RockyLinux 8.7 using the same compiler version.
-
Results: Very small differences in most species, as shown by the mass table and selected printouts. OH increased by 0.11%. Likely attributed to the differences in numerical libraries used by the compilers between CentOS and RockyLinux
Total atmospheric mass of species: Run 2 vs Run1
-
Takeaways:
- Brief description: Compares GC Classic runs on CentOS 7 and RockyLinux 8.7 using different compiler versions.
-
Results: Because Run 3 is identical to Run 2, the comparison of Run 3 vs. Run 1 gives identical results to Run 2 vs. Run 1. This is evident in the table of OH metrics shown below:
############################################################################### ### OH Metrics ### ### Left column Right column: ### Ref = GCC_centos_gnu10 (Run 1) Ref = GCC_centos_gnu10 (Run 1) ### Dev = GCC_rocky_gnu10 (Run 2) Dev = GCC_rocky_gnu12 (Run 3) ############################################################################### ------------------------------------------------------------ Global mass-weighted OH concentration [10^5 molec cm^-3] ------------------------------------------------------------ Run 1 : 13.07896632314 Run 1 : 13.07896632314 Run 2 : 13.09341931491 Run 3 : 13.09341931491 Abs diff : 0.01445299178 Abs diff : 0.01445299178 % diff : 0.110506 % diff : 0.110506 ------------------------------------------------------------ CH3CCl3 (aka MCF) lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Run 1 : 4.753380 Run 1 : 4.753380 Run 2 : 4.746909 Run 3 : 4.746909 Abs diff : -0.006471 Abs diff : -0.006471 % diff : -0.136133 % diff : -0.136133 ------------------------------------------------------------ CH4 lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Run 1 : 8.019566 Run 1 : 8.019566 Run 2 : 8.008516 Run 3 : 8.008516 Abs diff : -0.011050 Abs diff : -0.011050 % diff : -0.137782 % diff : -0.137782
- Takeaways: Compiling GEOS-Chem Classic with the GNU 10.2.0 compilers or the GNU 12.2.0 yields identical numerical results. The difference in the OH metrics are small (approx 0.1% or 0.001 absolute). This may be attributed to the difference in onboard libraries used by the compilers between CentOS and RockyLinux.
- Brief description: Compares GC Classic runs using 2 different compiler versions on RockyLinux.
-
Results: 100% identical
################################################################################ ### Benchmark summary table ### ### ### ### Ref = GCC_rocky_gnu10 ### ### Dev = GCC_rocky_gnu12 ### ################################################################################ ------------------------------------------------------------------------------- AerosolMass: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- Aerosols: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- Emissions: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- JValues: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- Metrics: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- SpeciesConc: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10 ------------------------------------------------------------------------------- StateMet: GCC_rocky_gnu12 is identical to GCC_rocky_gnu10
- Takeaways: Using RockyLinux 8.7 but Switching GNU compiler versions does not affect simulation results. This is likely due to the fact that both GNU 10.2.0 and GNU 12.2.0 compilers use the same underlying system libraries.
-
Brief description: Compares GCHP runs on CentOS 7 and RockyLinux 8.7 using the same compiler.
-
Results: OH metrics change by +/- 0.001% or less.
############################################################################### ### OH Metrics ### Ref = GCHP_centos_gnu10 ### Dev = GCHP_rocky_gnu10 ############################################################################### ------------------------------------------------------------ Global mass-weighted OH concentration [10^5 molec cm^-3] ------------------------------------------------------------ Ref : 13.16007998549 Dev : 13.16022032913 Abs diff : 0.00014034365 % diff : 0.001066 ------------------------------------------------------------ CH3CCl3 (aka MCF) lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Ref : 4.717519 Dev : 4.717485 Abs diff : -0.000034 % diff : -0.000722 ------------------------------------------------------------ CH4 lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Ref : 7.958901 Dev : 7.958847 Abs diff : -0.000054 % diff : -0.000684
-
The Dev/Ref ratios for O3 varies by +/- 0.01% at the surface.. Other species show similar behavior.
-
The Dev/Ref ratios for O3 varies by +/- 0.003% in zonal means. Other species show similar behavior.
-
Takeaways: Using the same compiler on different operating systems causes very small differences in GCHP output. This is probably attributed to differences in software libraries between CentOS 7 and RockyLinux 8.1.1. One note: The ESMF version used on RockyLinux (8.1.1) is newer than that used on CentOS 7. But the differences observed do not indicate any systematic bias.
- Brief description: Compares GCHP runs on CentOS 7 and RockyLinux 8.7 using different compiler versions.
-
Results: Because Run 6 is identical to Run 5, then the comparison of Run 6 vs. Run 4 is also identical to Run 5 vs. Run 4.
############################################################################### ### OH Metrics ### ### Left column: Right column: ### Ref = GCHP_centos_gnu10 (Run 4) Ref = GCHP_centos_gnu10 (Run 4) ### Dev = GCHP_rocky_gnu10 (Run 5) Dev = GCHP_rocky_gnu12 (Run 6) ############################################################################### ------------------------------------------------------------ Global mass-weighted OH concentration [10^5 molec cm^-3] ------------------------------------------------------------ Run 4 : 13.16007998549 Run 4 : 13.16007998549 Run 5 : 13.16022032913 Run 6 : 13.16022032913 Abs diff : 0.00014034365 Abs diff : 0.00014034365 % diff : 0.001066 % diff : 0.001066 ------------------------------------------------------------ CH3CCl3 (aka MCF) lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Ref : 4.717519 Run 4 : 4.717519 Dev : 4.717485 Run 6 : 4.717485 Abs diff : -0.000034 Abs diff : -0.000034 % diff : -0.000722 % diff : -0.000722 ------------------------------------------------------------ CH4 lifetime w/r/t tropospheric OH [years] ------------------------------------------------------------ Ref : 7.958901 Run 4 : 7.958901 Dev : 7.958847 Run 6 : 7.958847 Abs diff : -0.000054 Abs diff : -0.000054 % diff : -0.000684 % diff : -0.000684
- Takeaways: Runs 5 and 6 are identical. OH changes by +0.001%, which is negligble. This is a smaller amount of change as seen in GEOS-Chem Classic, however.
- Brief description: Compares GCHP runs using 2 different compiler versions on RockyLinux.
-
Results: 100% identical
################################################################################ ### Benchmark summary table ### ### ### ### Ref = GCHP_rocky_gnu10 ### ### Dev = GCHP_rocky_gnu12 ### ################################################################################ ------------------------------------------------------------------------------- AerosolMass: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- Aerosols: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- Emissions: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- JValues: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- Metrics: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- SpeciesConc: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10 ------------------------------------------------------------------------------- StateMet: GCHP_rocky_gnu12 is identical to GCHP_rocky_gnu10
- Takeaways: Using RockyLinux 8.7 but Switching GNU compiler versions does not affect simulation results. This is likely due to the fact that both GNU 10.2.0 and GNU 12.2.0 compilers use the same underlying system libraries.
Follow these steps to run the Integrated Methane Inversion workflow with RockyLinux.
-
Make sure that you have edited your
.bashrcfile to point to the RockyLinux Python environments. -
Log on to the RockyLinux test cluster. as described above.
-
Download the Integrated Methane Inversion from Bob Yantosca's forked repository. We recommend downloading to a folder into the one of the scratch spaces, such as
/n/holyscratch01/jacob_lab/$USERor/n/holylfs05/LABS/jacob_lab/$USER(where$USERis your login name).$ git clone -b feature/rocky-linux-envs https://github.com/yantosca/integrated_methane_inversion.git $ cd integrated_methane_inversion
-
Download GEOS-Chem Classic version 14.0.2. This is the current model version that is compatible with the IMI. (NOTE: The
config.ymllisted below is set up for AWS; we will replace it with a version that is customized for Cannon.)$ git clone -b 14.0.2 --recurse-submodules https://github.com/geoschem/GCClassic.git $ ls config.yml docs/ envs/ GCClassic/ LICENSE.md README.md resources/ run_imi.sh* setup_imi.sh* src/
-
Copy a GEOS-Chem environment file for Harvard-Cannon (from the
feature/rocky-linux-envsbranch) to theintegrated_methane_inversionroot folder. We'll edit this in the following steps.$ cp envs/Harvard-Cannon/gcclassic.rocky+gnu10.env . $ ls config.yml envs/ gcclassic.rocky+gnu10.env LICENSE.md resources/ setup_imi.sh* docs/ GCClassic/ imi_output.log README.md run_imi.sh* src/
-
Check out the
devbranch in theintegrated_methane_inversionfolder. We need to do this to take advantage of some fixes that were made after the previous IMI release. Switching branches will not clobber the environment files that we just copied.$ git checkout dev $ git branch * dev feature/rocky-linux-envs
-
Copy a the IMI configuration file for Harvard-Cannon (from the
devbranch) tointegrated_methane_inversionroot folder. We'll edit this in the following steps.$ cp envs/Harvard-Cannon/config.harvard-cannon.yml . $ ls config.harvard-cannon.yml docs/ GCClassic/ imi_output.log README.md run_imi.sh* src/ config.yml envs/ gcclassic.rocky+gnu10.env LICENSE.md resources/ setup_imi.sh*
-
Make the following edits to
config.harvard-cannon.yml. Replace/path/to/integrated_methane_inversionwith the path to your clone of the IMI repository.## IMI preview DoPreview: true DOFSThreshold: 0 . . . ## Path for IMI runs and output OutputPath: "/path/to/integrated_methane_inversion" . . . ## Environment files ## See envs/README to create the Conda environment specified below NCOEnv: "/path/to/integrated_methane_inversion/gcclassic.rocky+gnu10.env" GEOSChemEnv: "/path/to/integrated_methane_inversion/gcclassic.rocky+gnu10.env" CondaEnv: "imi_env"
-
Edit the GEOS-Chem template run script (
integrated_methane_inversion/src/geoschem_run_scripts/ch4_run.template) so that it will submit jobs to the RockyLinux test partition.#!/bin/bash ##SBATCH -c 8 ##SBATCH -N 1 ##SBATCH -t 0-06:00 ##SBATCH --mem=32000 ##SBATCH --mail-type=END ##SBATCH -p rocky <=========== Add this line here!!
-
Run the IMI:
$ ./run_imi.sh config.harvard-cannon.yml > imi_output.log 2>&1
After following the above steps, you should see a preview_diagnostics.txt file with output similar to this:
##Found 505 observations in the region of interest
##approximate cost = $0.25 for on-demand instance
## = $0.08 for spot instance
##Total prior emissions in region of interest = 0.012884588180591391 Tg/y
##k = [1.25903 1.25903 1.25903 1.25903 1.25903 1.25903 1.25903 1.25903 1.25903
1.25903 1.25903 1.25903 1.25903 1.25903 1.25903 1.25903 1.25903 1.25903
1.25903 1.25903] kg-1 m2 s
##a = [4.66e-03 9.00e-04 0.00e+00 1.00e-05 7.00e-05 0.00e+00 0.00e+00 2.00e-05
1.40e-04 0.00e+00 0.00e+00 0.00e+00 0.00e+00 0.00e+00 2.00e-05 0.00e+00
0.00e+00 0.00e+00 4.00e-05 0.00e+00]
expectedDOFS: 0.00587For other options, please see the IMI documentation at https://imi.readthedocs.org.
Once the operating system switchover to RockyLinux 8.7 has been completed, you may encounter warning messages such as:
WARNING: POSSIBLE DNS SPOOFING DETECTED!and/or
The RSA host key for login.rc.fas.harvard.edu has changed error messages.After an update of nodes the SSH key fingerprint of a node may change. This will, in turn, cause an error when you next try to log into that node as your locally stored key will no longer match.
To clear the errors, follow these instructions on the FASRC docs site.
After the switchover to RockyLinux happens, login nodes will restrict you to using 1 core and 4GB of memory. For more intensive applications, you can open an interactive job or submit a batch job on a computational node.