Running MPI on slurm

[ksteczk]

Do you have a receipt for running MMseqs2 using MPI and slurm? I'm trying various strategies but some return errors, some spawn much too many threads on nodes (probably to my lack of expertise here). Did you run the program on cluster?

I have, let's suppose, 2 nodes, 24 threads each (not counting masternode of course). I'd like to use both nodes for clustering. I run the following on masternode:

RUNNER="mpirun -np 1"
salloc -N 2  mpirun mmseqs cluster nr70_rep nr60 tmp --min-seq-id 0.6 --threads 24

What is the meaning of RUNNER variable here? Where should it be set? On master? On each node? I guess MMseqs2 itself manages further the sub-tasks so I have no intuition how to make it work with slurm.

Thank you for any advice here.

[milot-mirdita]

MMseqs2 workflows (such as cluster or search) internally run scripts that call the mmseqs modules. cluster is a workflow that will execute mmseqs prefilter (among others).
If you set the RUNNER environment variable, it will call $RUNNER mmseqs prefilter instead.

Only the prefilter (alignment, and some others) are actually MPI parallelized. It will not use RUNNER for non-MPI modules, these get executed only on the head node, while the other nodes remain idle (but pretty much all heavy tools are parallelized).

I am not sure how SLURM works, but in our LSF (with openmpi), the call would look like this:

RUNNER="mpirun --npernode 1"
mmseqs cluster nr70_rep nr60 tmp --min-seq-id 0.6 --threads 24

Also please double check, that the MMseqs2 version string contains a -MPI suffix. (I added that feature a few weeks ago.)

[martin-steinegger]

I will close this issue. Please reopen it if the issue is not resolved.