Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Synchronicity between merlin allocations and workflow computing resources #24

Closed
srcopela opened this issue Dec 4, 2019 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@srcopela
Copy link

srcopela commented Dec 4, 2019

I'm just getting started with Merlin and noticed that there are a few steps that need to be coordinated properly and it might be nice if there were utilities in Merlin to support that coordination.

Specifically, when I submit my merlin job to a parallel compute resource (e.g. via a slurm script as described in https://merlin.readthedocs.io/en/latest/merlin_commands.html), I specify the number of compute nodes. Then, within my merlin workflow, I specify the number of "nodes" for my workers in the workers sub-block of the merlin block (e.g. number of workers). Then I also specify the number of nodes and processors for each sample directly using the cmd: keyword in a step defined in the study block.

A utility to generate an sbatch file from a merlin spec would help keep things sorted, it seems like there's enough information contained there to do such a thing.

Also, as a novice to celery, I found the nodes keyword in the workers block to be somewhat misleading - it seems to control the number of workers, but my first instinct was that it was some kind of 'node allocation' assigned to the worker. Particularly if you're working with multi-node tasks, I can see others making the same mistake.

@lucpeterson lucpeterson added good first issue Good for newcomers enhancement New feature or request labels Dec 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants