Task batching #3909
Draft
Task batching #3909
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
pditommaso
force-pushed
the
master
branch
2 times, most recently
from
81f7cb7
to
8a43489
Compare
|
Link to GNU parallel page at NCI HPC centre (Canberra AUS) : https://opus.nci.org.au/display/Help/GNU%20parallel |
marcodelapierre
assigned marcodelapierre and bentsherman and unassigned marcodelapierre
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
|
Had a first quick look - Good! Slick design and implementation that build on top of the job array effort Will review in greater detail after finalising the Job Array work package. |
|
I had a quick thrill worrying that arrays are not supported in Bash 3.x, which is super old but still the default version on Macs. But my memory was only half valid: Bash 3.x does support indexed arrays (the ones used here in TaskArrayCollector), while it does not support associative arrays (maps). |
|
I dreamt tonight that this had been merged |
This was referenced Apr 29, 2024
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Good morning Europe! 😄
This PR implements task grouping as described in #2527. The implementation is inspired by #3905.
Each task processor has a "task group collector" that collects tasks and submits them as groups. The task group itself is just a task run, but with a special wrapper script that executes each child task sequentially. See the docs for details. Basically, a task group should behave exactly like a task array, except the child tasks run sequentially on the same node.
So far it's working on the local executor, haven't tested anything beyond that. For now this PR is just a proof of concept to help us think through task arrays vs task groups.