Neko on Fugaku (hybrid MPI+OpenMP)

[njansson]

Instructions

To compile Neko on Fugaku for the A64FX CPU using the Fujitsu compiler, with hybrid MPI+OpenMP parallelism.

The Fujitsu cross-compilers (frtpx, fccpx, mpifrtpx, mpifccpx) are available on the login nodes by default. BLAS/LAPACK are provided by Fujitsu SSL II, so no external BLAS is needed — it is pulled in via the -SSL2 flag below.

Configure Neko using the following options.:

./configure FC=frtpx CC=fccpx \
            MPIFC=mpifrtpx MPICC=mpifccpx \
            FCFLAGS="-Kfast,openmp -Kocl -SSL2 -Nlst=t -Nlst=p -Nsetvalue=stack -Nfjomplib -X08" \

Build using:

make -j12 && make install

The A64FX has 4 Core-Memory-Groups (CMGs) of 12 cores each (48 compute cores/node), so a natural hybrid layout is 4 MPI ranks per node × 12 OpenMP threads per rank, which keeps each rank inside one CMG/NUMA domain.

Example PJM jobscript:

#!/bin/sh -x
#PJM -L  "node=9216"                  # number of nodes
#PJM --mpi "max-proc-per-node=4"    # ranks per node
#PJM -L  "elapse=01:00:00"
#PJM -L  "rscgrp=large"             # replace with your resource group
#PJM -s
 
export OMP_NUM_THREADS=12
export FLIB_FASTOMP=TRUE            # use the Fujitsu fast OpenMP runtime

mpiexec ./neko mycase.case

The job output is as follows (note the output of OpenMP threads per rank):

    _  __  ____  __ __  ____
   / |/ / / __/ / //_/ / __ \
  /    / / _/  / ,<   / /_/ /
 /_/|_/ /___/ /_/|_|  \____/

 (version: 1.99.3)
 (build: 2026-02-24 on aarch64-pc-linux-gnu using fujitsu)


 -------Job Information--------
 Start time: 21:05 / 2026-02-27
 Running on: 36864 MPI ranks, using 12 thrds each
 CPU type  : Fujitsu A64FX
 Bcknd type: CPU
 Real type : double precision

Important Notes

• Due to Fujitsu Fortran causes a memory leak from type-bound procedures #2638, patches/fujitsu_memleak.patch must be applied to avoid OOM for long running jobs
• For best performance, compare both the standard (MPI) and MPI neighbourhood collective (NEIGHBOUR) gather-scatter communication backend, the later seems to in general perform better on the Tofu network.