MSPileup: consider Neutrino PDs as premix #11543
Conversation
|
Jenkins results:
|
| @@ -96,7 +96,10 @@ def parseCampaigns(campDocs, logger): | |||
| # for each secondary dataset, create one pileup document | |||
| for puName, puRSEs in camp.get("Secondaries", {}).items(): | |||
| puDoc = getPUSchema(puName, pileupDocs, logger) | |||
| puType = "premix" if puName.split("/")[-1] == "PREMIX" else "classic" | |||
| if puName.startswith("/Neutrino") or puName.split("/")[-1] == "PREMIX": | |||
There was a problem hiding this comment.
since it starts diverging from original I rather suggest to take these patterns and provide options in script to allow specify them with default values as these two. But I don't want to block it if you think we need it quickly, I only care if more pattern will appear in a long run.
There was a problem hiding this comment.
It should be the other way around, which is also confirmed from a document that Dima collected for PREMIX data placement. AFAIK, for the last year or two, all the premix pileup samples have the PREMIX datatier.
This "Neutrino" bit is just to deal with "legacy" pileup.
|
There is a flaw in this script which allows duplicate pileup names to be defined in different dict/objects, e.g.: I have to fix it and create a new json dump. |
amaltaro
force-pushed
the
fix-11522-take2
branch
from
ea2021e
to
e174512
Compare
|
Now we have unique pileup objects, going down from 125 to 75 documents. I also think that these json dumps should be moved from the Please let me know your thoughts, Todor and Valentin. |
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
MSPileup: consider Neutrino PDs as premix Ensure pileup documents are unique
Move pileup JSON dump under test/data directory New dump of pileup configuration update list of pileups.json after many checks from P&R
amaltaro
force-pushed
the
fix-11522-take2
branch
from
42c447f
to
9af433f
Compare
|
Jenkins results:
|
|
After the pretty good clean up performed by the P&R, campaigns have been updated in WMCore and I remade the pileup json that we had previously constructed a week ago (coming down from 79 unique pileups to only 24). Merging it now. |
Fixes #11522 (take2)
Status
ready
Description
Minor updates to the script parsing WMCore campaigns and extracting pileup configuration, such as:
Pileup JSON dump have been moved under
test/data/WMCore/MicroService/DataStructs/Is it backward compatible (if not, which system it affects?)
YES
Related PRs
Complement to #11521
External dependencies / deployment changes
None