Resolve the premix file list once instead of in every job - #17
Merged
Conversation
kandrosov
force-pushed
the
premix-filelist
branch
from
August 28, 2026 15:17
c149263 to
4c5e073
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #16 — until that merges this PR also shows its four commits. Merge #16 first, then this.
What this fixes
All 500 failures of the 5000-job production were one error:
--pileup_input dbs:<dataset>makes every job resolve the premix dataset — 37 993 files — withits own DAS query. At a few thousand concurrent jobs the queries start coming back empty, cmsDriver
writes a config with no secondary input, and cmsRun dies after the job has already produced its
GEN-SIM. The damage was concentrated where outbound access is poorest: T1_IT_CNAF 159,
T2_TR_METU 135, T2_EE_Estonia 60, T1_US_FNAL 34. At 115 100 jobs it gets worse, not better.
How
PremixFileList, a local per-era task, resolves the dataset once and stores the plain file list at<output>/premix/<era>.txtonfs_default.RunProdrequires it, localizes it like the gridpack,and passes
--pileup_input filelist:<path>.filesFromList(Configuration/Applications/python/ConfigBuilder.py:104) reads one LFN per line —exactly
dasgoclientoutput — so the job gets precisely what a successful DAS query would haveproduced. Nothing about the physics changes; 115 100 DAS queries become five.
Details worth knowing:
<output>_test: the list depends only on the era, so a testrun reuses it (same rule as gridpacks).
submitted_task_family()guard from Do not let a production job build its own gridpack #11) — resolvingit on a worker is the exact thing being eliminated. Submitting
PremixFileListitself stillworks, so lists can be pre-built on the grid.
dbs:unchanged, so no existing setup breaks.Cost
The list is 5.7 MB, downloaded per job, against the ~29 MB gridpack each job already pulls — under
20 % on top of existing per-job input. A deterministic per-job slice would cut it to near zero but
changes which premix events each job sees, so that is left as a deliberate choice rather than made
silently here.
Testing
Ten unit checks: the dataset is discovered from the conditions for all five eras; without a list the
command still says
dbs:; with one it saysfilelist:and contains nodbs:anywhere; a stepwithout pileup gets no
--pileup_input; the task branches once per era; a test run resolves to theproduction path; a worker running
RunProdrefuses whilePremixFileListitself proceeds.End to end on Run3_2023 (
--test 5, local), all green:The stored list is 37 993 lines / 5 667 589 B — matching DAS's
nfilesfor the dataset exactly —and DIGIPremixHLT read the premix events from it, which is the point of the change.
mkdocs build --strictpasses;run_tools/apply_format.shis a no-op on the diff.