Continuous tests stuck in arm64 container

[bdevans]

Describe the bug
When testing the new 2024.1 release in a docker image, it builds fine and most of the Continuous tests run fine, but they get stuck just after TestCardiacSimulation (needing to be manually stopped after ~1 hour on an M1 Max MBP).

To Reproduce
Steps to reproduce the problem.

git clone https://github.com/Chaste/chaste-docker.git
cd chaste-docker
make setup  # Necessary if you are not on arm64 hardware 
PLATFORM="linux/arm64/v8" make release

Expected behavior
The whole Continuous test suite should only take a few minutes.

System
Please provide relevant details of your system, such as:

• OS version: macOS 14.3.1
• CMake version: 3.22.1
• C++ compiler version: 11.4.0
• Python version: 3.10.12
• Relevant Chaste dependency versions e.g. PETSc, Boost, Sundials etc.
  PETSc: 3.15.5
  Boost: 1.74.0
  HDF5: 1.10.7
  Parmetis: 4.0.3
  XSD: 4.0.0
  SUNDIALS: 5.8.0
  VTK: 7.1
  Xerces: 3.2.3

Please paste the full output in a code block. It will be similar to the truncated example below.

[+] Building 1478.4s (25/26)                                                                                                                           docker:desktop-linux
 => [base 14/16] RUN ln -s "/home/chaste/build" lib &&     ln -s "/home/chaste/output" testoutput                                                                      0.3s
 => [base 15/16] RUN git config --global --add safe.directory "/home/chaste/src"                                                                                       0.3s
 => [base 16/16] RUN apt-cache show chaste-dependencies > chaste-dependencies.txt &&     ctest --verbose -R TestChasteBuildInfo$                                       0.4s
 => [build 1/1] RUN build_chaste.sh 2024.1                                                                                                                          1012.7s
 => [test 1/1] RUN test.sh Continuous                                                                                                                                455.4s
 => => # 393/401 Test #449: TestHeartConfig ...................................................   Passed    1.72 sec                                                       
 => => # 394/401 Test #451: TestHeartGeometryInformation ......................................   Passed    1.55 sec                                                       
 => => # 395/401 Test #445: TestCardiacSimulationArchiver .....................................   Passed    2.83 sec                                                       
 => => # 396/401 Test #398: TestCvodeCellsWithDataClamp .......................................   Passed   46.48 sec                                                       
 => => # 397/401 Test #400: TestDynamicallyLoadedCellModels ...................................   Passed   47.72 sec                                                       
 => => # 398/401 Test #444: TestCardiacSimulation .............................................   Passed   18.83 sec                                                       

[bdevans]

This does not appear to be a problem when testing on noble, although 13 tests are currently failing.

[mirams]

Hmm, if it is after Cardiac Simulation it probably is unrelated, can you run with just one core so you see which one has launched and got stuck?

[bdevans]

Hmm, they all passed when running with a single core! Perhaps there's a race condition somewhere?

[bdevans]

I've just built a new chaste/release image (based on the latest ubuntu:noble image with chaste 2024.1 compiled) and run the tests with a single core (on arm64) and this is where they hang:

274/401 Test #310: TestRunningPottsBasedSimulationsTutorial ..........................   Passed    2.01 sec
        Start 311: TestRunningTumourSpheroidSimulationsTutorial
275/401 Test #311: TestRunningTumourSpheroidSimulationsTutorial ......................   Passed    0.41 sec
        Start 312: TestRunningVertexBasedSimulationsTutorial
276/401 Test #312: TestRunningVertexBasedSimulationsTutorial .........................   Passed    0.27 sec
        Start 313: TestVisualizingWithParaviewTutorial