-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update HLT/HLT relval/Express/Prompt GTs #33206
Update HLT/HLT relval/Express/Prompt GTs #33206
Conversation
test parameters
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33206/21640
|
A new Pull Request was created by @christopheralanwest for master. It involves the following packages: Configuration/AlCa @malbouis, @andrius-k, @yuanchao, @kmaeshima, @fwyzard, @christopheralanwest, @ErnestaP, @ahmad3213, @Martin-Grunewald, @cmsbuild, @jfernan2, @tlampen, @pohsun, @rvenditti, @francescobrivio can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
please test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the update. Some observations / questions below:
# GlobalTag for Run3 HLT: it points to the online GT | ||
'run3_hlt' : '112X_dataRun3_HLT_v2', | ||
# GlobalTag with fixed snapshot time for Run1 HLT RelVals: customizations to run with fixed L1 Menu | ||
'run1_hlt_relval' : '112X_dataRun2_HLT_relval_v5', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since it points anyway to the same GT as the Run2 key, what about enforcing the identity, by using the compatibility dictionary below?
Same thing is true for the run1_data
and run2_data
keys.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The equality of each pair is now guaranteed by commit 988b4e0. These keys cannot be added with the compatibility keys as the run1_hlt_relval
key is needed by autoCondHLT
earlier in the file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wouldn't a re-ordering be enough here?
diff --git a/Configuration/AlCa/python/autoCond.py b/Configuration/AlCa/python/autoCond.py
index f4f08d09b5c..c94856c1d01 100644
--- a/Configuration/AlCa/python/autoCond.py
+++ b/Configuration/AlCa/python/autoCond.py
@@ -84,29 +84,11 @@ aliases = {
'BASEGT' : 'BASE1_V1|BASE2_V1'
}
-### Run 1 data GTs ###
+### OLD KEYS ### kept for backward compatibility
# GlobalTag with fixed snapshot time for Run1 HLT RelVals: customizations to run with fixed L1 Menu
autoCond['run1_hlt_relval'] = autoCond['run2_hlt_relval']
# GlobalTag for Run1 data reprocessing
autoCond['run1_data'] = autoCond['run2_data']
-
-# dedicated GlobalTags for HLT
-from Configuration.HLT.autoCondHLT import autoCondHLT
-autoCond = autoCondHLT(autoCond)
-
-# dedicated GlobalTags for phase-2 (specializing conditions for each geometry)
-from Configuration.AlCa.autoCondPhase2 import autoCondPhase2
-autoCond = autoCondPhase2(autoCond)
-
-# special cases modifier for autoCond GTs
-from Configuration.AlCa.autoCondModifiers import autoCond0T
-autoCond = autoCond0T(autoCond)
-
-# special GT for 2015 HLT HI run
-from Configuration.AlCa.autoCondModifiers import autoCondHLTHI
-autoCond = autoCondHLTHI(autoCond)
-
-### OLD KEYS ### kept for backward compatibility
# GlobalTag for MC production with perfectly aligned and calibrated detector
autoCond['mc'] = ( autoCond['run1_design'] )
# GlobalTag for MC production with realistic alignment and calibrations
@@ -125,3 +107,19 @@ autoCond['upgradePLS150ns'] = ( autoCond['run2_mc_50ns'] )
autoCond['upgrade2017'] = ( autoCond['phase1_2017_design'] )
autoCond['upgrade2021'] = ( autoCond['phase1_2021_design'] )
autoCond['upgradePLS3'] = ( autoCond['phase2_realistic'] )
+
+# dedicated GlobalTags for HLT
+from Configuration.HLT.autoCondHLT import autoCondHLT
+autoCond = autoCondHLT(autoCond)
+
+# dedicated GlobalTags for phase-2 (specializing conditions for each geometry)
+from Configuration.AlCa.autoCondPhase2 import autoCondPhase2
+autoCond = autoCondPhase2(autoCond)
+
+# special cases modifier for autoCond GTs
+from Configuration.AlCa.autoCondModifiers import autoCond0T
+autoCond = autoCond0T(autoCond)
+
+# special GT for 2015 HLT HI run
+from Configuration.AlCa.autoCondModifiers import autoCondHLTHI
+autoCond = autoCondHLTHI(autoCond)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer the current ordering as the keys run1_hlt_relval
and run1_data
are standard autoCond
keys, even though they each are fixed to have the same value as another key. I would rather keep the deprecated keys separate at the end of the file.
# GlobalTag for Run3 data relvals | ||
'run3_data_promptlike' : '112X_dataRun3_Prompt_v3', | ||
'run3_data_promptlike' : '112X_dataRun3_Prompt_v4', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is a real infinite time snapshot GlobalTag suitable for Prompt Reco (with all the connected risks associated to it) what's the point of calling it 'promptlike'? Would not it be better to call it run3_prompt
? Also is the plan during pp operations to keep updating it as frequently as Tier-0 does?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the key name to run3_data_prompt
in commit f9b641c. Once an offline Run 3 GT is available, workflows that use run3_data_prompt
would be converted to a run3_data
key. Of course, it would be possible to create an offline GT for Run 3. But since that GT would currently have no other uses, I think it's preferable to exercise a GT that is actually used.
As long as there are no tags contained in the prompt GTs with synchronizations not allowed by the prompt queue, I think that the synchronization policy would be sufficient to ensure reproducibility. Is there another potential way to violate reproducibility that I am not considering?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As long as there are no tags contained in the prompt GTs with synchronizations not allowed by the prompt queue, I think that the synchronization policy would be sufficient to ensure reproducibility. Is there another potential way to violate reproducibility that I am not considering?
I am mostly concerned with private and "semi-official" (such as AlCaRelVals for conditions validation) usage of the autoCond
keys. As soon as we have 24/7 operations someone might be tempted to use this key to process fresh data, possibly even earlier that PromptReco is released for that run. So in principle you might get different results with that recipe depending if you run before or after than the PCL and O2Os for that run have completed. For RelVals and unit tests using a fixed run (from the relatively distant past) the prompt synchronization will enforce reproducibility.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With a fixed snapshot time, analysis of runs taken after the snapshot time could receive incorrect, old conditions. If someone tries to process data before the PromptReco is released for that run, they will run into problems independent of whether the prompt GT is used in autoCond or not. I will let the other AlCa conveners decide what they prefer to do here, as they will be maintaining these GTs in the future, not me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe for the time being we could choose to keep this GT in autoCond as @christopheralanwest suggests and reassess it in the future, in case we think it is necessary.
|
||
for key,val in six.iteritems(autoCond): | ||
if key == 'run2_hlt_relval': # modification of HLT relval GT | ||
GlobalTagsHLTHI['run2_hlt_hi'] = (autoCond[key], FullPedestalsForHLTHI, MenuForHLTHI) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@icali @CesarBernardes please take note of this. This GlobalTag, suitable for Hybrid ZeroSuppression emulation on 2015 Heavy Ion ViriginRaw data has become symbolic. This means it's fine for local tests and RelVals but not e.g. for central samples production. Please comment if this might constitue a problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Of course, if it is desired to reconstruct 2015 HI data in 11_2_X or later, AlCa could trivially create a regular, non-symbolic GT. Our assumption is that this would be needed rarely enough that it is simpler to maintain the GT as a symbolic GT based on the run2_hlt_relval
GT.
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aebe2a/13578/summary.html Comparison Summary@slava77 comparisons for the following workflows were not done due to missing matrix map:
Summary:
|
+1 For the online GTs the snapshot time is as expected and the right PFCalibration tag is used, PFCalibration_HLT_2017_25ns_Spring17_v3_hlt. |
+1 |
+1 |
+Upgrade This PR is to organize GTs. |
+1 |
+1 |
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2) |
+1 |
PR description:
This PR makes several changes to the online GTs to bring several up-to-date and simplify maintenance.
run1_hlt
key is replaced by arun1_hlt_relval
key as the only usage of this key is in Run 1 addOnTests in which the L1T menu is intended to be fixed, as in relvalsrun2_hlt
key has been removed, as it is not used anywhere in CMSSWrun3_hlt
has been introduced and is intended to be kept up-to-date with the actual HLT GT. This key is used by theDQM/Integration
package and is also used in the legacyautoCond['hltonline']
key (intended to represent the online HLT). There are no other uses of this key in CMSSW.run2_hlt_relval
has been brought up-to-date with respect to the current online GT. It now is equivalent to the online HLT key, aside from a fixed snapshot time and L1T trigger menurun2_hlt_relval_hi
key was superfluous. It differs from therun2_hlt_relval
key only by the L1 trigger menu. Since only usage was inautoCondHLT
, which overrides the trigger menu, I have modified the relevant entry to use therun2_hlt_relval
key and removed therun2_hlt_relval_hi
key.run2_hlt_hi
is used only by workflow 140.55, representing 2015 HI data and requires a special tagSiStripFullPedestals_GR10_v1_hlt
and a special L1T menu. Since this is a very special scenario, this GT has been represented by a symbolic GT that implements these changes on top of therun2_hlt_relval
key.During the course of preparing this PR, I discovered that an the
PFCalibrationRcd
with label "HLT" was incorrect in the HLT, Express and Prompt GTs. The corrected tag is now used.The GT diffs are as follows:
Run 2 HLT RelVals
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/112X_dataRun2_HLT_relval_v4/112X_dataRun2_HLT_relval_v5
Run 3 data (express)
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/112X_dataRun3_Express_v3/112X_dataRun3_Express_v4
Run 3 data (prompt)
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/112X_dataRun3_Prompt_v3/112X_dataRun3_Prompt_v4
The large number of EGM calibration records that are present in old Run 2 HLT relvals are not consumed (or else a crash would result in various tests).
Attn: @mmusich @Martin-Grunewald
PR validation:
This PR is mainly technical and has been validated with:
if this PR is a backport please specify the original PR and why you need to backport that PR:
Since this PR affects GTs in the 112X release, a backport to 11_2_X is intended.