Sporadic CI failures at tpv5-poro

[davschneller]

Describe the bug
The tpv5-poro test fails from time to time in the CI due to a segfault. If it happens, it always does so about 30 seconds into the run. Given that the successful runtime is about 2 min 30 s, and the run taking pretty exactly 2 mins, I'd suspect that it happens in one of the first iterations.

Expected behavior
The test should pass consistently.

To Reproduce
Steps to reproduce the behavior: submit a CI job several times (e.g. create any new branch, or commit to an existing one), eventually one of them will fail at tpv5-poro.

1. Which version do you use? Happens with all current versions sporadically, cf. e.g. https://gitlab.lrz.de/ci_seissol/SeisSol/-/jobs/6781361
2. Which build settings do you use? Cf.

   SeisSol/gitlab-cpu-pipeline.yml

   Lines 36 to 78 in 4f89e56

   	build_seissol:
   	stage: build
   	tags:
   	- sccs
   	- build
   	needs:
   	- job: fetch_submodules
   	parallel:
   	matrix:
   	- precision: [double, single]
   	build_type: [Debug, Release]
   	equation: [elastic, anisotropic, poroelastic, viscoelastic2]
   	script:
   	- cmake --version
   	- mechanisms=0
   	- dirname="build_${equation}_${precision}_${build_type}";
   	if [ "$equation" = viscoelastic2 ] && [ "$build_type" == Debug ]; then echo "Skip ${dirname}" && exit 0; fi;
   	if [ "$equation" = anisotropic ] && [ "$build_type" == Debug ]; then echo "Skip ${dirname}" && exit 0; fi;
   	if [ "$equation" = poroelastic ] && [ "$build_type" == Debug ]; then echo "Skip ${dirname}" && exit 0; fi;
   	if [ "$equation" = viscoelastic2 ]; then mechanisms=3; else mechanisms=0; fi;
   	echo "mkdir $dirname";
   	mkdir -p $dirname && cd $dirname ;
   	pwd;
   	CMAKE_PREFIX_PATH=~ ;
   	cmake ..
   	-DNETCDF=ON
   	-DMETIS=ON
   	-DCOMMTHREAD=OFF
   	-DASAGI=OFF
   	-DHDF5=ON
   	-DCMAKE_BUILD_TYPE=$build_type
   	-DTESTING=ON
   	-DLOG_LEVEL=warning
   	-DLOG_LEVEL_MASTER=info
   	-DHOST_ARCH=${HOST}
   	-DPRECISION=$precision
   	-DEQUATIONS=$equation
   	-DNUMBER_OF_MECHANISMS=$mechanisms
   	-DDR_QUAD_RULE=stroud
   	-DGEMM_TOOLS_LIST=LIBXSMM
   	-DCMAKE_EXPORT_COMPILE_COMMANDS=ON;
   	make -j $(nproc);
   	cd .. ;

   with poroelastic material and Release configuration, precision does not matter. See here,

   SeisSol/gitlab-cpu-pipeline.yml

   Lines 115 to 153 in 4f89e56

   	run_tpv:
   	stage: test
   	allow_failure: false
   	tags:
   	- sccs
   	- cpu-hsw
   	needs:
   	- job: build_seissol
   	parallel:
   	matrix:
   	- precision: [double, single]
   	tpv: [5, 5-nuc, 5-poro, 5-visco, 6, 13, 16, 101, 101-slip, 104, 105, ahsp, yoffe, gaussian]
   	script:
   	- echo "run TPV${tpv}"
   	- git clone https://github.com/SeisSol/precomputed-seissol.git
   	- ls
   	- cd precomputed-seissol
   	- git checkout 8b1201c
   	- cd ..
   	- export equations=elastic
   	- if [ "$tpv" = ahsp ]; then equations=anisotropic; fi;
   	- if [ "$tpv" = 5-poro ]; then equations=poroelastic; fi;
   	- if [ "$tpv" = 5-visco ]; then equations=viscoelastic2; fi;
   	- cd build_${equations}_${precision}_Release
   	- cp -r ../precomputed-seissol/tpv${tpv}/* .
   	- ls
   	- mkdir ./output
   	- export NUM_COMP_CORES=$(expr $(nproc) - 1)
   	- export OMP_NUM_THREADS=$NUM_COMP_CORES
   	- export OMP_PLACES="cores($NUM_COMP_CORES)"
   	- ulimit -Ss unlimited
   	- ./SeisSol_Release_*_${equations} ./parameters.par
   	- mv ./precomputed/${precision} ./output/precomputed
   	- mv ./output ../output-tpv{$tpv}-${precision}
   	artifacts:
   	paths:
   	- output-*
   	expire_in: 2 days
   	retry: 2

   , choosing 5-poro.
3. On which machine does your problem occur? SeisSol CPU CI (i.e. SCCS CI) node.
4. Provide parameter/material files. See https://github.com/SeisSol/precomputed-seissol/tree/master/tpv5-poro

Screenshots/Console output
Cf. e.g. https://gitlab.lrz.de/ci_seissol/SeisSol/-/jobs/6781361

Especially line 241 there (https://gitlab.lrz.de/ci_seissol/SeisSol/-/jobs/6781361#L241 )

Additional context
None.