Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fast_job_submission: add new guide #195

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

chu11
Copy link
Member

@chu11 chu11 commented Feb 8, 2023

Add new guide to improve job submission speed in Flux. I was on and off writing this for a bit, finally decided to write it for real.

Note, for now I just stuck this stupidly under "jobs" set of docs. I'm pretty sure its not the right place.

Possibly a "guide" section under tutorials? Ehhh?

@chu11 chu11 changed the title [WIP] fast_job_submission: add new guide fast_job_submission: add new guide Feb 10, 2023
@chu11
Copy link
Member Author

chu11 commented Feb 10, 2023

re-pushed, I decided to put the tutorial under a new "how to" section of tutorials. The idea is that the "how to" topics will likely involve multiple commands or multiple prior topics, used together to describe something.

@vsoch
Copy link
Member

vsoch commented Feb 10, 2023

@chu11 I think (my preference) for the time being since we have an entire jobs section would be to keep the tutorial under there, and when it's warranted to re-organized we think of all the pages in this section in that context. We are also working on the Command tutorials section, and I think it's going to get too watered down to start with "Howto" and "Commands." Arguably, everything is a "How to" and it's a category that moreso reflects we don't know exactly where to put it (and I'm going to suggest the well-established Jobs section for now).

@chu11
Copy link
Member Author

chu11 commented Feb 10, 2023

yeah, i feel its a little bit in between the two. But lets go with the jobs section for now, moved it back.

@chu11
Copy link
Member Author

chu11 commented Feb 10, 2023

rebased and re-pushed, "sub-instance" -> "subinstance"

Copy link
Member

@vsoch vsoch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this tutorial is great and I don't see any issues, however I'm not experienced with this kind of submission.

@chu11
Copy link
Member Author

chu11 commented Feb 10, 2023

re-pushed, cut and pasted code-block:: sh in some places where it should be code-block:: console`.

@chu11
Copy link
Member Author

chu11 commented Feb 11, 2023

I think this tutorial is great and I don't see any issues, however I'm not experienced with this kind of submission.

Perhaps @grondo or @garlick could take a quick skim, make sure what I'm writing sounds good.

Copy link
Member

@SteVwonder SteVwonder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!


Jobs can be asynchronously submitted via several mechanisms. This will allow us to significantly reduce the slow iterative process of submitting jobs one by one.

The first mechanism is the ``--cc`` (carbon copy) roption in ``flux mini submit``. It will allow the user to replicate every id specified in an :ref:`IDSET<idset>`. Along with the ``{cc}`` substitution string, we can submit all 1000 scripts on the command line like so:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The first mechanism is the ``--cc`` (carbon copy) roption in ``flux mini submit``. It will allow the user to replicate every id specified in an :ref:`IDSET<idset>`. Along with the ``{cc}`` substitution string, we can submit all 1000 scripts on the command line like so:
The first mechanism is the ``--cc`` (carbon copy) option in ``flux mini submit``. It will allow the user to replicate every id specified in an :ref:`IDSET<idset>`. Along with the ``{cc}`` substitution string, we can submit all 1000 scripts on the command line like so:

@chu11
Copy link
Member Author

chu11 commented Mar 27, 2023

re-pushed, with comment fixes from above and removing "flux mini" references

Add new guide to improve job submission speed in Flux.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants