Fix job attribution for OSPool grand-total report - #22
Merged
Conversation
extract_job_name_from_metl() checked for "000 (cluster_id." and "DAG Node:" on the same line, but DAG Node always appears on the line after the submission event. This made the fallback always return None, silently dropping every job that relies on it (e.g. DAGs with no local .dagman.out file) as unmapped. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the just recipes that pull the misconfigured-DAG data and generate full_ospool-grand-total.csv, which relies on the DAG Node fallback fix in mldag/report/csv.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
extract_job_name_from_metl()checked for the000 (cluster_id.submission line andDAG Node:on the same line, butDAG Node:always appears on the following line. This made the fallback always returnNone.full_ospool-grand-total.csvhave no local.dagman.outfile, so they relied entirely on this broken fallback for job attribution. Every one of their execution events was silently dropped as unmapped, making the grand-total CSV identical tofull_ospool.csv._refresh-ospool-grand-totaland_csv-ospool-grand-totaljust recipes that producefull_ospool-grand-total.csvusing this fix.Test plan
full_ospool-grand-total.csvwithjust _csv-ospool-grand-total: attempts went from 1384 to 2362, correctly split between the two misconfigured DAG sources (505 / 473 rows)full_ospool.csvis still present unchanged in the regenerated grand-total CSV