Skip to content

Resolve the premix file list once instead of in every job - #17

Merged
kandrosov merged 6 commits into
cms-flaf:mainfrom
kandrosov:premix-filelist
Aug 28, 2026
Merged

Resolve the premix file list once instead of in every job#17
kandrosov merged 6 commits into
cms-flaf:mainfrom
kandrosov:premix-filelist

Conversation

@kandrosov

Copy link
Copy Markdown
Contributor

Stacked on #16 — until that merges this PR also shows its four commits. Merge #16 first, then this.

What this fixes

All 500 failures of the 5000-job production were one error:

NoSecondaryFiles: RootEmbeddedFileSequence no input files specified for secondary input source
  Constructing module: class=PreMixingModule label='mixData'

--pileup_input dbs:<dataset> makes every job resolve the premix dataset — 37 993 files — with
its own DAS query. At a few thousand concurrent jobs the queries start coming back empty, cmsDriver
writes a config with no secondary input, and cmsRun dies after the job has already produced its
GEN-SIM. The damage was concentrated where outbound access is poorest: T1_IT_CNAF 159,
T2_TR_METU 135, T2_EE_Estonia 60, T1_US_FNAL 34. At 115 100 jobs it gets worse, not better.

How

PremixFileList, a local per-era task, resolves the dataset once and stores the plain file list at
<output>/premix/<era>.txt on fs_default. RunProd requires it, localizes it like the gridpack,
and passes --pileup_input filelist:<path>.

filesFromList (Configuration/Applications/python/ConfigBuilder.py:104) reads one LFN per line —
exactly dasgoclient output — so the job gets precisely what a successful DAS query would have
produced. Nothing about the physics changes; 115 100 DAS queries become five.

Details worth knowing:

  • Stored in the production area, never <output>_test: the list depends only on the era, so a test
    run reuses it (same rule as gridpacks).
  • A batch node refuses to build one (the submitted_task_family() guard from Do not let a production job build its own gridpack #11) — resolving
    it on a worker is the exact thing being eliminated. Submitting PremixFileList itself still
    works, so lists can be pre-built on the grid.
  • With no list configured the command falls back to dbs: unchanged, so no existing setup breaks.

Cost

The list is 5.7 MB, downloaded per job, against the ~29 MB gridpack each job already pulls — under
20 % on top of existing per-job input. A deterministic per-job slice would cut it to near zero but
changes which premix events each job sees, so that is left as a deliberate choice rather than made
silently here.

Testing

Ten unit checks: the dataset is discovered from the conditions for all five eras; without a list the
command still says dbs:; with one it says filelist: and contains no dbs: anywhere; a step
without pileup gets no --pileup_input; the task branches once per era; a test run resolves to the
production path; a worker running RunProd refuses while PremixFileList itself proceeds.

End to end on Run3_2023 (--test 5, local), all green:

PremixFileList[Run3_2023]: 37993 files from /Neutrino_E-10_gun/Run3Summer21PrePremix-...
--pileup_input "filelist:/tmp/kandroso/luigi-tmp-034755548.txt"
Scheduled 9 tasks — 6 ran successfully, no failed tasks
staging/nanoAOD_v12/.../nano_v12_1.root  2 869 283
staging/nanoAOD_v15/.../nano_v15_1.root  3 211 983

The stored list is 37 993 lines / 5 667 589 B — matching DAS's nfiles for the dataset exactly —
and DIGIPremixHLT read the premix events from it, which is the point of the change.

mkdocs build --strict passes; run_tools/apply_format.sh is a no-op on the diff.

@kandrosov
kandrosov merged commit 7837542 into cms-flaf:main Aug 28, 2026
6 checks passed
@kandrosov
kandrosov deleted the premix-filelist branch August 28, 2026 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant