add --taskmap=hostfile:FILE support
#5844
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
grondo
force-pushed
the
taskmap-hostfile
branch
from
6fc96a6
to
dd324de
Compare
garlick
approved these changes
Mar 30, 2024
src/shell/taskmap/hostfile.c
Outdated
Comment on lines
81
to
82
| if (!(fp = fopen (path, "r"))) | ||
| shell_die_errno (1, "failed to open hostfile: %s", path); |
There was a problem hiding this comment.
Any particular reason why this function calls shell_die_errno() on error while hostlist_from_R() does not?
There was a problem hiding this comment.
No, that was left over from a prototype and should probably be fixed.
grondo
force-pushed
the
taskmap-hostfile
branch
from
dd324de
to
c536d2e
Compare
Problem: The active_shells idset is leaked in the shell output plugin. Add the missing idset_destroy() in shell_output_destroy().
Problem: Builtin shell taskmap plugins are currently stored at the top level in src/shell, but it would be better to keep these in a subdirectory to avoid naming confusion. Move the cyclic taskmap plugin into src/shell/taskmap.
Problem: There is currently no simple way to map tasks to specific hosts using a list of hostnames. Add a hostfile taskmap plugin to the job shell. This taskmap scheme assigns tasks to hosts in order as they appear in a file provided by the scheme value. The hostfile is read as a list of newline separated hostnames or RFC 29 Hostlist strings. If there are not as many hosts as tasks in the hostfile, then the list is reused, and if more hosts than tasks appear in the file, then the list is truncated. Like other taskmap schemes, it is an error if the same task count is not assigned to each host as in the default task map. Fixes flux-framework#5794
Problem: There are no tests of the job shell hostfile taskmap support. Since the hostfile support requires different hostnames for each "node" in an instance, create a new test specific to this taskmap type using the "system" test personality, which assigns fake hostnames to brokers.
Problem: The hostfile taskmap scheme is not documented. Mention this scheme in flux-shell(1) along with the other builtin taskmap schemes, and document it in the cli submission commands.
grondo
force-pushed
the
taskmap-hostfile
branch
from
c536d2e
to
f31c26b
Compare
|
Ok, addressed comments from @garlick and I'll set MWP. |
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #5844 +/- ##
========================================
Coverage 83.28% 83.29%
========================================
Files 510 511 +1
Lines 82594 82694 +100
========================================
+ Hits 68785 68876 +91
- Misses 13809 13818 +9
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new taskmap job shell plugin with the traditional "hostfile" mechanism for assigning task ids based on a list of host names. This was requested as a means to do very specific arbitrary task mapping in an easy to understand manner. It was easy to implement and will probably be nice to have around, even if not useful a majority of the time.