You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm just getting started with Merlin and noticed that there are a few steps that need to be coordinated properly and it might be nice if there were utilities in Merlin to support that coordination.
Specifically, when I submit my merlin job to a parallel compute resource (e.g. via a slurm script as described in https://merlin.readthedocs.io/en/latest/merlin_commands.html), I specify the number of compute nodes. Then, within my merlin workflow, I specify the number of "nodes" for my workers in the workers sub-block of the merlin block (e.g. number of workers). Then I also specify the number of nodes and processors for each sample directly using the cmd: keyword in a step defined in the study block.
A utility to generate an sbatch file from a merlin spec would help keep things sorted, it seems like there's enough information contained there to do such a thing.
Also, as a novice to celery, I found the nodes keyword in the workers block to be somewhat misleading - it seems to control the number of workers, but my first instinct was that it was some kind of 'node allocation' assigned to the worker. Particularly if you're working with multi-node tasks, I can see others making the same mistake.
The text was updated successfully, but these errors were encountered:
I'm just getting started with Merlin and noticed that there are a few steps that need to be coordinated properly and it might be nice if there were utilities in Merlin to support that coordination.
Specifically, when I submit my merlin job to a parallel compute resource (e.g. via a slurm script as described in https://merlin.readthedocs.io/en/latest/merlin_commands.html), I specify the number of compute nodes. Then, within my merlin workflow, I specify the number of "nodes" for my workers in the workers sub-block of the merlin block (e.g. number of workers). Then I also specify the number of nodes and processors for each sample directly using the cmd: keyword in a step defined in the study block.
A utility to generate an sbatch file from a merlin spec would help keep things sorted, it seems like there's enough information contained there to do such a thing.
Also, as a novice to celery, I found the nodes keyword in the workers block to be somewhat misleading - it seems to control the number of workers, but my first instinct was that it was some kind of 'node allocation' assigned to the worker. Particularly if you're working with multi-node tasks, I can see others making the same mistake.
The text was updated successfully, but these errors were encountered: