hipErrorOutOfMemory when running PeleLMex Flamesheet case with GPU

[hong27]

Dear AMReX-Combustion community,

I was trying to run PeleLMeX with GPU on the HPC Setonix. I have started with the Flamesheet tutorial case, whose performance for GPU runs has been reported in the documentation of PeleLMeX.

This case is successfully compiled with

ml PrgEnv-gnu/8.3.3 rocm/5.2.3 craype-accel-amd-gfx90a && export MPICH_GPU_SUPPORT_ENABLED=1
cd $PELE_HOME/Exec/RegTests/FlameSheet
make -j 8 USE_HIP=TRUE USE_MPI=TRUE TPL
make -j 8 USE_HIP=TRUE USE_MPI=TRUE DIM=3 Chemistry_Model=dodecane_lu_qss

I then requested a node to run it with

#SBATCH --partition=gpu            # Using the gpu partition
#SBATCH --ntasks=8                 # Total number of tasks
#SBATCH --ntasks-per-node=8        # Set this for 1 mpi task per compute device

#SBATCH --gpus-per-task=1          # How many HIP compute devices to allocate to a  task
#SBATCH --gpu-bind=closest         # Bind each MPI task to the nearest GPU
#SBATCH --exclusive                # Use this to request all the resources on a node
#SBATCH --time=00:20:00

export OMP_NUM_THREADS=8   
export OMP_PLACES=cores     
export OMP_PROC_BIND=close  

srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS -c $OMP_NUM_THREADS ./PeleLMeX3d.hip.x86-milan.MPI.HIP.ex inputs.3d_DodecaneQSS

The simulation failed to initialise with a hipErrorOutOfMemory, indicated as:

Initializing AMReX (23.12-24-g75571e2dcbf2-dirty)...
MPI initialized with 8 MPI processes
MPI initialized with thread support level 0
Initializing HIP...
HIP initialized with 8 devices.
amrex::Abort::0::HIP error in file /software/projects/w47/jlin/PeleLMeX-dev/Submodules/PelePhysics/Submodules/amrex/Src/Base/AMReX_Arena.cpp line 171 hipErrorOutOfMemory !!!
MPICH ERROR [Rank 0] [job id 11075568.0] [Tue Apr 23 14:42:11 2024] [nid002244] - Abort(6) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 6) - process 0

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 6) - process 0
amrex::Abort::4::HIP error in file /software/projects/w47/jlin/PeleLMeX-dev/Submodules/PelePhysics/Submodules/amrex/Src/Base/AMReX_Arena.cpp line 171 hipErrorOutOfMemory !!!
MPICH ERROR [Rank 4] [job id 11075568.0] [Tue Apr 23 14:42:11 2024] [nid002244] - Abort(6) (rank 4 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 6) - process 4

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 6) - process 4
amrex::Abort::6::HIP error in file /software/projects/w47/jlin/PeleLMeX-dev/Submodules/PelePhysics/Submodules/amrex/Src/Base/AMReX_Arena.cpp line 171 hipErrorOutOfMemory !!!
MPICH ERROR [Rank 6] [job id 11075568.0] [Tue Apr 23 14:42:11 2024] [nid002244] - Abort(6) (rank 6 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 6) - process 6

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 6) - process 6
amrex::Abort::2::HIP error in file /software/projects/w47/jlin/PeleLMeX-dev/Submodules/PelePhysics/Submodules/amrex/Src/Base/AMReX_Arena.cpp line 171 hipErrorOutOfMemory !!!
MPICH ERROR [Rank 2] [job id 11075568.0] [Tue Apr 23 14:42:11 2024] [nid002244] - Abort(6) (rank 2 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 6) - process 2

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 6) - process 2
amrex::Abort::5::HIP error in file /software/projects/w47/jlin/PeleLMeX-dev/Submodules/PelePhysics/Submodules/amrex/Src/Base/AMReX_Arena.cpp line 171 hipErrorOutOfMemory !!!
MPICH ERROR [Rank 5] [job id 11075568.0] [Tue Apr 23 14:42:11 2024] [nid002244] - Abort(6) (rank 5 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 6) - process 5

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 6) - process 5
srun: error: nid002244: tasks 0,2,4-6: Exited with exit code 255
srun: launch/slurm: _step_signal: Terminating StepId=11075568.0
slurmstepd: error: *** STEP 11075568.0 ON nid002244 CANCELLED AT 2024-04-23T14:42:11 ***
srun: error: nid002244: tasks 1,3,7: Terminated
srun: Force Terminated StepId=11075568.0

I am wondering if there might have been any missed compilation flags or specific setup adjusted needed to avoid this issue. Has anyone encountered a similar problem? Any suggestions or shared experiences would be greatly appreciated!

Jianhong

[baperry2]

@SreejithNREL - you've run on Setonix right? Did you encounter the issue described here?

[SreejithNREL]

I have not faced this error on Setonix, although I did face another error with amrex::InitRandom failing on GPULaunch

[esclapez]

By default, AMReX will try to reserve 3/4 of the available GPU memory for its Arena. Maybe something is preventing it from doing this on this platform. Can you try reducing that amount, using for instance:
amrex.the_arena_init_size=8589934592

(8Gb in bites).

[esclapez]

I have just tried to run this case on LUMI, which is very similar to Frontier and Setonix. The case ran with a couple of changes:

• Deactivating managed memory here
• Reduce the max_grid_size to 32. This is necessary when running with the magma_direct linear solver in the chemistry, default GMRES might get away with 64 because the memory requirements are much smaller with an iterative solve.

Are you using GPU-aware MPI ? I remember some issues with AMReX arena init size when using GPU-aware MPI on Frontier, so you'll probably need to request a smaller amount (unless the issue has been resolved) as described in the comment above.

[hong27]

Sorry for getting back to this a bit late. Thanks @esclapez I can now run the case in GPU on Setonix with the two settings. Much appreciated!

amrex.the_arena_is_managed=0
amr.max_grid_size   = 32