Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

desi_job_graph makes job dependency and status graph webpage #1896

Merged
merged 4 commits into from Nov 3, 2022
Merged

Conversation

sbailey
Copy link
Contributor

@sbailey sbailey commented Nov 3, 2022

This PR adds a script that makes a webpage with the job dependency graph and status for a given night, e.g.

image
image

Clicking on a box takes you to the log for that job. This graph is based upon the processing table and SLURM job return codes; it does not check for missing job outputs like the dashboard and if any steps were run outside of the pipeline scripts that update the processing table, this won't know about them. Overall the dashboard is better for getting the big picture across many nights and for checking actual file outputs, while I find this view to be easier to identify what steps went wrong when there is a problem in a night, and for distinguishing "crashed" from "in progress".

Currently this is a single script for simplicity. If we find that we want to call it from other code (e.g. a spin webapp), it could be factored out into a separate module of desispec.

usage: desi_job_graph [-h] -n NIGHT [-o OUTPUT] [-s SPECPROD]

optional arguments:
  -h, --help            show this help message and exit
  -n NIGHT, --night NIGHT
                        night to process
  -o OUTPUT, --output OUTPUT
                        output HTML file (default run/jobgraph/jobgraph-NIGHT.html in reduxdir)
  -s SPECPROD, --specprod SPECPROD
                        override $SPECPROD, or full path to override $DESI_SPECTRO_REDUX/$SPECPROD

@sbailey sbailey requested a review from akremin November 3, 2022 19:04
Copy link
Member

@akremin akremin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a handy script for visualizing a night of processing. I am already imagining additional features but I will hold back on that now. I have some comments in-line that would be nice to have addressed before merging. Then we can build up the feature set with future PR's.

bin/desi_job_graph Show resolved Hide resolved
bin/desi_job_graph Show resolved Hide resolved
bin/desi_job_graph Outdated Show resolved Hide resolved
@sbailey
Copy link
Contributor Author

sbailey commented Nov 3, 2022

Thanks for the good comments; I believe all are addressed, and a slipped in one more feature of adding a link to the specproddir instead of just the test of the specproddir.

Copy link
Member

@akremin akremin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All comments have been resolved. The script looks good and my test run succeeded. I'll approve and merge this now.

@akremin akremin merged commit 34a424d into main Nov 3, 2022
@akremin akremin deleted the jobgraph branch November 3, 2022 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants