C++ app in spack environment on Google cloud HPC with slurm -> illegal instruction. How to correctly build for the target VM.

[michele-colle]

Hello, I hope this is the right place to ask, I'm trying to deploy an x ray simulation on a Google cloud HPC cluster with slurm and I got the 2989 illegal instruction (core dumped) error.

I used a slightly modified version of the example present in the computing cluster repos which sets up a login and a controller node plus various computing nodes and a debug node. Here is the blueprint: https://github.com/michele-colle/CBCTSim/blob/main/HPCScripts/hpc-slurm.yaml

Than on the login node I installed the spack environment (https://github.com/michele-colle/CBCTSim/blob/main/HPC_env_settings/spack.yaml) and build the app with cmake and the appropriate, already present compiler.

After some try and error I was able to successfully run a test on the debug node (https://github.com/michele-colle/CBCTSim/blob/main/HPCScripts/test_debug.slurm)

Than I proceeded to try out a more intense operation (around 10 minutes work) on a compute node (https://github.com/michele-colle/CBCTSim/blob/main/HPCScripts/job_C2D.slurm) but I got the above error.

I am completely new on HPC computing but I struggle to find resources on CPP applications, I suspect it has something to do with the app building process but I am basically lost.

Any help is appreciated, thanks for reading:)

[bo-qeye]

I speculate that the problem lies in the fact that the software is compiled on one type of CPU, but executed on another. Typically, a compiler will auto-detect the instruction set available and (potentially) fully utilize that. If the resulting binary is then executed on a different CPU (typically older) it won't work.

You can try the following:

1. Ensure that your compute nodes are of the exact same GCP instance type as your debug node, where the code successfully runs. (It's the machine_type parameter, and you need to be explicit.)
2. Force the compilation with cmake to use a certain instruction set.

In fact, if you want to be absolutely sure, stick to the same machine_type on both login, controller and and compute nodes.

Protip, you can actually omit the login node and just use the controller instead. There's only a need for both in very large deployments.

[michele-colle]

Hello and thanks for the answer.
Looking at the Google cloud machine families I see that one of the most suitable for scientific computing is the h3 machine, which is included in the slurm HPC example yaml on this repo, can I get the instruction set and architecture of the h3 machine through some command? Or should I use the h3 machine to build its spack environment and compile?

[bo-qeye]

The easiest way to test whether the problem lies in the different architectures is to align the login and compute nodes. Doesn't have to be a fancy and modern VM type, C2 is fine for experimentation.

But to answer your question, lscpu and /proc/cpuinfo are your friends.

[bo-qeye]

I'm curious; did you get things up and running?

[michele-colle]

I paused the exploration due to issues on other applications I am developing, as soon as I get back to it I'll update this discussion.
Thanks for the interest!

[siddhartha-quad]

Hi, Closing the ticket for now as it is pending for long, Kindly feel free to re-open if there are any issues.

[michele-colle]

Yes sorry for not having closed it myself, I'm following another project right now so maybe in the future I'll get back to this.
Have a nice day.