Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

desi_queue_status #1986

Merged
merged 2 commits into from Jan 28, 2023
Merged

desi_queue_status #1986

merged 2 commits into from Jan 28, 2023

Conversation

sbailey
Copy link
Contributor

@sbailey sbailey commented Jan 27, 2023

This PR adds a new script desi_queue_status for summarizing jobs in the queue and recent history; this is useful for monitoring during a production run. Currently it tracks SUCCESS, FAILED, and TIMEOUT jobs, but not CANCELLED (e.g. cancelled due to a dependency failing). It could be better, but it is useful as-is and I'd like to get it into the next tag, and then continue improvements in future PRs (like adding CANCELLED once I figure out how to get sacct to report those too without already knowing the job IDs).

Example output, running as sjbailey but checking the desi queue for jobs from yesterday+today (default is today only, but we haven't run much today to be interesting):

[login08 desispec] desi_queue_status -u desi -S 2023-01-26
Fri Jan 27 15:50:36 2023
0 jobs running, 0 pending, and 0 waiting on dependencies
Recent job completion history:
     HOUR     COMPLETED FAILED TIMEOUT
------------- --------- ------ -------
2023-01-26T08       102     33       1
2023-01-26T09       659      1       0
2023-01-26T10       696      1       1
2023-01-26T11        28      1       0
2023-01-26T12        19      0       1
2023-01-26T13         5      0       0
2023-01-26T21         3      0       0
2023-01-27T15         1      0       0

@sbailey sbailey merged commit d1f7477 into main Jan 28, 2023
@sbailey sbailey deleted the queue_status branch January 28, 2023 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

1 participant