Skip to content

Building

Jump to bottom
dwfncar edited this page Oct 9, 2026 · 5 revisions

Building

CheMPAS-A has build procedures for Ubuntu, macOS with LLVM flang, and Derecho. All three use the top-level MPAS Makefile and the same revision-pinned MUSICA dependency closure.

For a contained Linux/AMD64 build that supports the three bounded demonstration runs, see Docker. The native procedures below remain the path for development, alternate cases, data preparation, and production workflows.

Environment Make target Compiler family
Ubuntu 24.04 gfortran GCC/gfortran with Open MPI
macOS llvm Apple clang/clang++ and Homebrew LLVM flang
Derecho CPU gfortran GCC 14.3.0 with Cray MPICH (NCAR mpicc, mpicxx, and mpif90 wrappers)

Do not reuse .mod or object files across compiler families. Build PIO, MUSICA-Fortran, NetCDF-Fortran, and CheMPAS-A with one compatible toolchain, and use separate installation prefixes for Ubuntu, macOS, and Derecho.

Required MUSICA source closure

The CheMPAS-A Makefile rejects a package with the wrong MUSICA version, MUSICA or MIEM source revision, or without MIEM enabled. The MICM, MechanismConfiguration, and TUV-x revisions are checked only by the pkg-config metadata tests below. The required closure is:

Component Required revision
MUSICA-Fortran 1403e3d22717bc87f3bf9d0aa591caf039c92bbc (0.16.5)
MICM bb57684a2047f0e58f30b199366294af879e8597
MIEM 9fdf14a189262eecb677862d877ab72b06c95e21
MechanismConfiguration 82c159ae6d74934318ffd6c405a45c2159065b12
TUV-x bbf7dd9a144fa0f0294b3779f3f993818638e20c

Obtain the source once in each platform workspace:

export MUSICA_REV=1403e3d22717bc87f3bf9d0aa591caf039c92bbc

git clone --branch feature/miem-scalability-fortran --single-branch \
  https://github.com/NCAR/MUSICA.git MUSICA-CheMPAS-A
git -C MUSICA-CheMPAS-A checkout --detach "$MUSICA_REV"
test "$(git -C MUSICA-CheMPAS-A rev-parse HEAD)" = "$MUSICA_REV"

The platform sections below build static MUSICA libraries with the Fortran interface, MICM, MIEM, and TUV-x enabled and CARMA, MIAM, and fmt disabled. FETCHCONTENT_TRY_FIND_PACKAGE_MODE=NEVER prevents a same-named system package from replacing a pinned component.

Ubuntu 24.04 with GCC and Open MPI

The tested MVP environment used GNU Fortran 15.2.0, Open MPI 5.0.10, NetCDF-C 4.10.1, NetCDF-Fortran 4.6.3, PnetCDF 1.14.1, and PIO 2.6.9. Create one conda environment for the compiler, MPI wrappers, and I/O stack:

conda create -n chempas-a -c conda-forge \
  gcc=15.2 gxx=15.2 gfortran=15.2 openmpi=5.0.10 \
  libnetcdf=4.10.1 netcdf-fortran=4.6.3 libpnetcdf=1.14.1 \
  cmake pkg-config make git
conda activate chempas-a

export CHEMPAS_DEPS="$HOME/software/chempas-a/ubuntu-gnu"
export NETCDF="$CONDA_PREFIX"
export NETCDFF="$CONDA_PREFIX"
export PNETCDF="$CONDA_PREFIX"
export PIO="$CHEMPAS_DEPS/pio-2.6.9"
export MUSICA_PREFIX="$CHEMPAS_DEPS/musica-$MUSICA_REV"
export CHEMPAS_MAKE_TARGET=gfortran

Build PIO with the same MPI wrappers:

git clone --depth 1 --branch pio2_6_9 \
  https://github.com/NCAR/ParallelIO.git ParallelIO-2.6.9
cmake -S ParallelIO-2.6.9 -B ParallelIO-2.6.9/build-chempas \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$PIO" \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_Fortran_COMPILER=mpifort \
  -DPIO_ENABLE_TIMING=OFF \
  -DPIO_ENABLE_TESTS=OFF
cmake --build ParallelIO-2.6.9/build-chempas --parallel 8
cmake --install ParallelIO-2.6.9/build-chempas

Build MUSICA:

cmake -S MUSICA-CheMPAS-A -B MUSICA-CheMPAS-A/build-ubuntu \
  -DFETCHCONTENT_TRY_FIND_PACKAGE_MODE=NEVER \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$MUSICA_PREFIX" \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpifort \
  -DBUILD_SHARED_LIBS=OFF \
  -DBUILD_TESTING=OFF \
  -DMUSICA_BUILD_FORTRAN_INTERFACE=ON \
  -DMUSICA_ENABLE_MPI=ON \
  -DMUSICA_ENABLE_MICM=ON \
  -DMUSICA_ENABLE_MIEM=ON \
  -DMUSICA_ENABLE_TUVX=ON \
  -DMUSICA_ENABLE_CARMA=OFF \
  -DMUSICA_ENABLE_MIAM=OFF \
  -DMUSICA_USE_FMT=OFF
cmake --build MUSICA-CheMPAS-A/build-ubuntu --parallel 8
cmake --install MUSICA-CheMPAS-A/build-ubuntu

macOS with LLVM flang

Install the C/C++ compilers, flang, Open MPI, NetCDF-C, and build tools:

brew install llvm flang open-mpi netcdf cmake pkg-config autoconf automake libtool

export CHEMPAS_DEPS="$HOME/software/chempas-a/macos-llvm"
export NETCDF="$(brew --prefix netcdf)"
export NETCDFF="$CHEMPAS_DEPS/netcdf-fortran-4.6.2"
export PNETCDF="$CHEMPAS_DEPS/pnetcdf-1.14.1"
export PIO="$CHEMPAS_DEPS/pio-2.6.9"
export MUSICA_PREFIX="$CHEMPAS_DEPS/musica-$MUSICA_REV"
export CHEMPAS_MAKE_TARGET=llvm
export OMPI_CC=clang
export OMPI_CXX=clang++
export OMPI_FC=flang

Homebrew's Open MPI and NetCDF-Fortran modules are normally produced by gfortran and cannot be consumed as Fortran modules by flang. Build a static NetCDF-Fortran 4.6.2 with flang against Homebrew NetCDF-C:

curl -L https://github.com/Unidata/netcdf-fortran/archive/refs/tags/v4.6.2.tar.gz \
  -o netcdf-fortran-4.6.2.tar.gz
tar -xf netcdf-fortran-4.6.2.tar.gz
mkdir -p netcdf-fortran-build
cd netcdf-fortran-build
CC=clang FC=flang \
CPPFLAGS="-I$NETCDF/include" LDFLAGS="-L$NETCDF/lib" \
PKG_CONFIG_PATH="$NETCDF/lib/pkgconfig" \
  ../netcdf-fortran-4.6.2/configure \
    --prefix="$NETCDFF" --disable-shared --enable-static
make -j8
make install
cd ..

Build PnetCDF without its gfortran-dependent Fortran interface. CheMPAS-A uses PnetCDF through PIO's C interface on this platform:

curl -L https://parallel-netcdf.github.io/Release/pnetcdf-1.14.1.tar.gz \
  -o pnetcdf-1.14.1.tar.gz
tar -xf pnetcdf-1.14.1.tar.gz
cd pnetcdf-1.14.1
CC=clang MPICC=mpicc ./configure --prefix="$PNETCDF" \
  --disable-cxx --disable-fortran --disable-shared
make -j8
make install
cd ..

export PATH="$PNETCDF/bin:$PATH"
export PKG_CONFIG_PATH="$NETCDFF/lib/pkgconfig:$NETCDF/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"
test "$(pnetcdf-config --prefix)" = "$PNETCDF"
test "$(pkg-config --modversion netcdf-fortran)" = 4.6.2

Build PIO through the Open MPI wrappers selected by OMPI_CC=clang and OMPI_FC=flang. PIO detects LLVM flang and uses mpif.h instead of the incompatible Homebrew mpi.mod:

git clone --depth 1 --branch pio2_6_9 \
  https://github.com/NCAR/ParallelIO.git ParallelIO-2.6.9
cmake -S ParallelIO-2.6.9 -B ParallelIO-2.6.9/build-chempas-llvm \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$PIO" \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_Fortran_COMPILER=mpifort \
  -DPIO_ENABLE_TIMING=OFF \
  -DPIO_ENABLE_TESTS=OFF
cmake --build ParallelIO-2.6.9/build-chempas-llvm --parallel 8
cmake --install ParallelIO-2.6.9/build-chempas-llvm

Put the flang-built NetCDF-Fortran metadata first and build MUSICA with direct compilers. MUSICA is rank-local in the macOS coupling, so its own MPI option is disabled; the CheMPAS-A executable remains MPI-enabled.

cmake -S MUSICA-CheMPAS-A -B MUSICA-CheMPAS-A/build-macos-llvm \
  -DFETCHCONTENT_TRY_FIND_PACKAGE_MODE=NEVER \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$MUSICA_PREFIX" \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_Fortran_COMPILER=flang \
  -DBUILD_SHARED_LIBS=OFF \
  -DBUILD_TESTING=OFF \
  -DMUSICA_BUILD_FORTRAN_INTERFACE=ON \
  -DMUSICA_ENABLE_MPI=OFF \
  -DMUSICA_ENABLE_MICM=ON \
  -DMUSICA_ENABLE_MIEM=ON \
  -DMUSICA_ENABLE_TUVX=ON \
  -DMUSICA_ENABLE_CARMA=OFF \
  -DMUSICA_ENABLE_MIAM=OFF \
  -DMUSICA_USE_FMT=OFF
cmake --build MUSICA-CheMPAS-A/build-macos-llvm --parallel 8
cmake --install MUSICA-CheMPAS-A/build-macos-llvm

Derecho with GCC

Derecho users can link against a shared, prebuilt PIO and MUSICA installation (option A) or build both themselves (option B). Both options use ncarenv/25.10 with GCC 14.3.0, Cray MPICH, and the NCAR MPI compiler wrappers (mpicc, mpicxx, mpif90). A CheMPAS-A build, and every PBS run script, must load the same modules as the libraries it links.

Use gcc/14.3.0, not gcc/12.5.0. GCC 12's C++ library has no <format>, which MechanismConfiguration requires with MUSICA_USE_FMT=OFF.

The Cray compilers (CCE 19.0.0) do not currently work for this build. TUV-x does not compile without a source patch (ftn-629 in its get_updater functions). CCE's C++ compiler is pinned to GCC 12's C++ library, so <format> is missing. And the Cray ftn driver aborts during the final CheMPAS-A link (add_cce_postlink: Assertion 'clangrt_path != NULL' failed).

Option A: use the shared PIO and MUSICA installation

PIO 2.6.9 and MUSICA 0.16.5 (revision 1403e3d22717bc87f3bf9d0aa591caf039c92bbc) are installed under /glade/work/fillmore/chempas-a/ncarenv-25.10-gcc. They were built with ncarenv/25.10, GCC 14.3.0, Cray MPICH 8.1.32, NetCDF 4.9.3 (NetCDF-Fortran 4.6.2), and PnetCDF 1.14.1, and the MUSICA build passes all 30 MUSICA unit tests. The env.sh in that directory does three things:

  • purges your loaded modules and loads exactly that module set;
  • exports MUSICA_REV, NETCDF, NETCDFF, PNETCDF, CHEMPAS_DEPS, PIO, MUSICA_PREFIX, CHEMPAS_MAKE_TARGET, and PKG_CONFIG_PATH; and
  • stops with an error if the active gfortran is not the one that built MUSICA.
source /glade/work/fillmore/chempas-a/ncarenv-25.10-gcc/env.sh

Then continue with Verify the MUSICA installation and Build and link the CheMPAS-A cores. Source the same env.sh in PBS run scripts.

BUILD-INFO.md in the same directory records how the libraries were built and tested. The TUV-x data from the pinned MUSICA revision is in $MUSICA_PREFIX/share/tuvx/data. In run directories, link it as data in place of a MUSICA checkout's configs/tuvx/data.

Option B: build PIO and MUSICA yourself

Build on a compute node rather than a shared login node. Start an interactive development job, pin ncarenv/25.10, select GCC 14.3.0, and load compiler-matched I/O libraries. Replace YOUR_PROJECT_CODE below:

export PROJECT_CODE=YOUR_PROJECT_CODE
qinteractive -A "$PROJECT_CODE" -l walltime=02:00:00

module --force purge
module load ncarenv/25.10
module reset
module swap intel gcc/14.3.0
module load cmake netcdf parallel-netcdf
module -t list

command -v mpicc mpicxx mpif90 mpifort cmake pkg-config nc-config nf-config pnetcdf-config

export NETCDF="$(nc-config --prefix)"
export NETCDFF="$(nf-config --prefix)"
export PNETCDF="$(pnetcdf-config --prefix)"
export CHEMPAS_DEPS="/glade/work/$USER/chempas-a/ncarenv-25.10-gcc"
export PIO="$CHEMPAS_DEPS/pio-2.6.9"
export MUSICA_PREFIX="$CHEMPAS_DEPS/musica-0.16.5"
export CHEMPAS_MAKE_TARGET=gfortran

Build PIO 2.6.9 with the same MPI wrappers and I/O modules:

git clone --depth 1 --branch pio2_6_9 \
  https://github.com/NCAR/ParallelIO.git ParallelIO-2.6.9
cmake -S ParallelIO-2.6.9 -B ParallelIO-2.6.9/build-derecho-gcc \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$PIO" \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_Fortran_COMPILER=mpifort \
  -DPIO_ENABLE_TIMING=OFF \
  -DPIO_ENABLE_TESTS=OFF
cmake --build ParallelIO-2.6.9/build-derecho-gcc --parallel 8
cmake --install ParallelIO-2.6.9/build-derecho-gcc

Do not load Derecho's musica/0.10.1 module; it is too old and lacks the required MIEM revision/API. Build the pinned MUSICA package with the MPI wrappers. Set MUSICA_ENABLE_TESTS=OFF: BUILD_TESTING=OFF does not disable MUSICA's unit tests, and on Derecho they fail to link. The NCAR compiler wrappers add -I/glade/u/apps/derecho/25.10/opt/view/include, which puts a newer system googletest ahead of MUSICA's pinned copy.

cmake -S MUSICA-CheMPAS-A -B MUSICA-CheMPAS-A/build-derecho-gcc \
  -DFETCHCONTENT_TRY_FIND_PACKAGE_MODE=NEVER \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$MUSICA_PREFIX" \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpifort \
  -DBUILD_SHARED_LIBS=OFF \
  -DBUILD_TESTING=OFF \
  -DMUSICA_ENABLE_TESTS=OFF \
  -DMUSICA_BUILD_FORTRAN_INTERFACE=ON \
  -DMUSICA_ENABLE_MPI=ON \
  -DMUSICA_ENABLE_MICM=ON \
  -DMUSICA_ENABLE_MIEM=ON \
  -DMUSICA_ENABLE_TUVX=ON \
  -DMUSICA_ENABLE_CARMA=OFF \
  -DMUSICA_ENABLE_MIAM=OFF \
  -DMUSICA_USE_FMT=OFF
cmake --build MUSICA-CheMPAS-A/build-derecho-gcc --parallel 8
cmake --install MUSICA-CheMPAS-A/build-derecho-gcc

Use the same ncarenv, GCC, MPI, and I/O module list in PBS run scripts. NSF NCAR documents the wrappers and hierarchical modules in Compiling Code on Derecho.

Optional: run MUSICA's unit tests

To check a MUSICA build, configure a separate tree with the tests enabled and without the system googletest include:

(
  unset NCAR_INC_0_COMMON
  cmake -S MUSICA-CheMPAS-A -B MUSICA-CheMPAS-A/build-derecho-gcc-tests \
    -DFETCHCONTENT_TRY_FIND_PACKAGE_MODE=NEVER \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_C_COMPILER=mpicc \
    -DCMAKE_CXX_COMPILER=mpicxx \
    -DCMAKE_Fortran_COMPILER=mpifort \
    -DBUILD_SHARED_LIBS=OFF \
    -DMUSICA_ENABLE_TESTS=ON \
    -DMUSICA_BUILD_FORTRAN_INTERFACE=ON \
    -DMUSICA_ENABLE_MPI=ON \
    -DMUSICA_ENABLE_MICM=ON \
    -DMUSICA_ENABLE_MIEM=ON \
    -DMUSICA_ENABLE_TUVX=ON \
    -DMUSICA_ENABLE_CARMA=OFF \
    -DMUSICA_ENABLE_MIAM=OFF \
    -DMUSICA_USE_FMT=OFF
  cmake --build MUSICA-CheMPAS-A/build-derecho-gcc-tests --parallel 8
)

MUSICA registers each test as mpirun -v -np 2 <test>, but Derecho has no mpirun executable. Its PALS mpiexec also needs a host list outside a multi-node job. A small wrapper runs the tests on two ranks on the current node:

mkdir -p mpirun-shim
cat > mpirun-shim/mpirun <<'EOF'
#!/bin/bash
# MUSICA tests call "mpirun -v -np N <test>"; run them with PALS on this node
np=1
while [[ $# -gt 0 ]]; do
  case "$1" in
    -v) shift ;;
    -np|-n) np="$2"; shift 2 ;;
    *) break ;;
  esac
done
exec mpiexec --hosts localhost -n "$np" "$@"
EOF
chmod +x mpirun-shim/mpirun

PATH="$PWD/mpirun-shim:$PATH" \
  ctest --test-dir MUSICA-CheMPAS-A/build-derecho-gcc-tests --output-on-failure

All 30 tests pass with the ncarenv/25.10 GCC 14.3.0 stack.

Verify the MUSICA installation

After completing one platform section, put the private installation first in PKG_CONFIG_PATH and verify its recorded build metadata. CMake installs the libraries in lib64 on Derecho and in lib on Ubuntu and macOS, so the path lists both:

export PKG_CONFIG_PATH="${MUSICA_PREFIX}/lib64/pkgconfig:${MUSICA_PREFIX}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"

test "$(pkg-config --modversion musica-fortran)" = 0.16.5
test "$(pkg-config --variable=source_revision musica-fortran)" = "$MUSICA_REV"
test "$(pkg-config --variable=micm_revision musica-fortran)" = \
  bb57684a2047f0e58f30b199366294af879e8597
test "$(pkg-config --variable=miem_revision musica-fortran)" = \
  9fdf14a189262eecb677862d877ab72b06c95e21
test "$(pkg-config --variable=mechanism_configuration_revision musica-fortran)" = \
  82c159ae6d74934318ffd6c405a45c2159065b12
test "$(pkg-config --variable=tuvx_revision musica-fortran)" = \
  bbf7dd9a144fa0f0294b3779f3f993818638e20c
test "$(pkg-config --variable=miem_enabled musica-fortran)" = ON
test "$(pkg-config --variable=tuvx_enabled musica-fortran)" = ON
pkg-config --variable=fortran_compiler_id musica-fortran
pkg-config --variable=fortran_compiler_version musica-fortran
pkg-config --libs musica-fortran

The compiler metadata must name the active platform compiler. The reported library list is the complete static link closure; do not append a hard-coded -lstdc++ or -lc++.

Build and link the CheMPAS-A cores

Clone the MVP release tag:

git clone --branch v2026.08.01 --depth 1 \
  https://github.com/NCAR/CheMPAS-A.git
cd CheMPAS-A

Build the initialization core first. Then clean the shared framework before building the MUSICA-enabled atmosphere core:

set -o pipefail
make clean CORE=init_atmosphere
make -j8 "$CHEMPAS_MAKE_TARGET" CORE=init_atmosphere OPENMP=false \
  PIO="$PIO" NETCDF="$NETCDF" NETCDFF="$NETCDFF" PNETCDF="$PNETCDF" \
  PRECISION=double 2>&1 | tee build-init_atmosphere.log

make clean CORE=atmosphere
make -j8 "$CHEMPAS_MAKE_TARGET" CORE=atmosphere OPENMP=false \
  PIO="$PIO" NETCDF="$NETCDF" NETCDFF="$NETCDFF" PNETCDF="$PNETCDF" \
  PRECISION=double MUSICA=true 2>&1 | tee build-atmosphere.log

The Makefile performs a constructor-level MICM+MIEM link probe using only the installed musica-fortran.pc flags. Confirm that probe, the final link message, and the executables:

grep -F "Built a simple test program with MUSICA-Fortran version 0.16.5" \
  build-atmosphere.log
grep -F "MPAS was linked with the MUSICA-Fortran library version 0.16.5" \
  build-atmosphere.log
test -x init_atmosphere_model
test -x build_tables
test -x atmosphere_model

The MUSICA libraries are static, so ldd and otool -L are not expected to list them. Without MUSICA=true, the chemistry hooks compile out and the chemistry namelist records are ignored.

Clean boundaries

MPAS cores share generated framework objects, while their build options differ. Use an explicit clean between cores and before changing compiler target, precision, or MUSICA state:

make clean CORE=atmosphere

Never delete or copy individual .mod files to repair a mixed toolchain; clean the tree and rebuild every dependency with the selected compiler family.

CheMPAS-A

Documentation · Source release

Clone this wiki locally