New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Save DQM output per lumisection for UL rereco #26232
Conversation
The code-checks are being triggered in jenkins. |
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26232/8861
|
A new Pull Request was created by @schneiml (Marcel Schneider) for master. It involves the following packages: DQMServices/Core @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
please test |
The tests are being triggered in jenkins. |
-1 Tested at: a4918f5 You can see the results of the tests here: I found follow errors while testing this PR Failed tests: UnitTests
I found errors in the following unit tests: ---> test TestDQMServicesFwkIOScripts had ERRORS |
Comparison job queued. |
Comparison is ready Comparison Summary:
|
One of the things that broke in this PR are the cmssw/DQMOffline/JetMET/src/METAnalyzer.cc Lines 1059 to 1074 in c1e72b4
this is not too big of a loss, given this type of operation in |
The more detailed bin-to-bin comparison shows 1000's of changes, waiting for another run to get the output uploaded to DQMGUI. I hope some of the PR tests actually cover more than one lumisection, to make this a meaningful test... |
Comparison is ready Comparison Summary:
|
The bin-by-bin comparison noticed that the reference histograms stored by CMSSW turned empty. Though, looking at the names, we don't produce these MEs; the reference data must be from run1, since we have not produced e.g. 'TEC/side1' MEs since 2013 (was renamed to 'TEC/MINUS'). Not sure if these references are still used anywhere (in offline), I'll try to get them back but it might not be worth the effort. |
the reference in the IB system is produced from a recent IB - so if thats the reference you are talking about, then I guess there is still some dqm code using the old convention. Or is there some other reference in your tests?
… On Mar 25, 2019, at 3:38 PM, Marcel Schneider ***@***.***> wrote:
The bin-by-bin comparison noticed that the reference histograms stored by CMSSW turned empty. Though, looking at the names, we don't produce these MEs; the reference data must be from run1, since we have not produced e.g. 'TEC/side1' MEs since 2013 (was renamed to 'TEC/MINUS').
Not sure if these references are still used anywhere (in offline), I'll try to get them back but it might not be worth the effort.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
@davidlange6 , no, I am talking about the reference that is handled inside the DQMStore and saved into the legacy root file going to the DQMGUI. I believe this mechanism was not used in Run2, except maybe for a few subsystems in online. Consequently, I have very little of an idea how it works, except for the fact that it exists... |
a4918f5
to
22e4128
Compare
Pull request #26232 was updated. @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @franzoni, @jfernan2, @fioriNTU, @fabiocos, @davidlange6 can you please check and sign again. |
please abort |
please test |
The tests are being triggered in jenkins. |
Comparison job queued. |
Comparison is ready Comparison Summary:
|
+1 |
@@ -1935,6 +1935,8 @@ def expandMapping(self,seqList,mapping,index=None): | |||
def prepare_DQM(self, sequence = 'DQMOffline'): | |||
# this one needs replacement | |||
|
|||
# any 'DQM' job should use DQMStore in non-legacy mode (but not HARVESTING) | |||
self.loadAndRemember("DQMServices/Core/DQMStoreNonLegacy_cff") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@schneiml this essentially sets always DQMStore.enableMultiThread = True
, and it should not change behaviour for multi-threaded workflows in my understanding, and not prevent HARVESTING to run anyway, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is completely independent of HARVESTING, which should be configured somewhere else.
And yes, all serious workflows run with enableMultiThread = True
anyways, so it makes sense to turn it on even on single-threaded workflows to get consistent behavior.
@@ -52,7 +52,7 @@ def mergeProcess(*inputFiles, **options): | |||
#// | |||
if newDQMIO: | |||
process.source = Source("DQMRootSource") | |||
process.add_(Service("DQMStore")) | |||
process.add_(Service("DQMStore", forceResetOnBeginLumi = CfgTypes.untracked.bool(True))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@schneiml if the feature needs to be off by default, should not this flag be turned to False?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, the changes in HARVESTING behavior are spread over many places and unconditional. This fixes a number of problems with the existing per-lumi plots (at the risk of causing some misbehavior in said plots, as explained in the PR message).
Conditional is only the change in the RECO step, which turns on per-lumi saving on all plots (that allow it).
@schneiml could you please address my two questions? I would like to move forward with this PR before pre4 if possible, I just need a clarification |
+operations |
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2) |
+1 |
PR description:
This PR enables per-lumisection saving of all DQM output for the UL rereco. This wish came up for machine learning studies targeting per-LS data certification or online monitoring.
In prompt reco, we have per-LS DQM output available due to how Tier0 processes only ~1 LS per job, and the per-job DQMIO output files are available. It would be very nice to have the same granularity of data preserved for the UL rereco, which in addition provides the full run2 dataset processed using the same software.
This is safe to do for most of DQM, since HARVESTING will re-generate per run/job statistics from the per-lumisection snapshots. It is not save for legacy modules that modify MEs at non-standard times or that do not support merging correctly; none of these should exist in RECO/DQM step since the threaded migration, but since there are still a few, I apply the change only to the non-legacy MEs.
PR validation:
Validated on a modified
136.85
that crosses a lumi boundary:runTheMatrix.py -l 136.85 --command '--customise_commands "if process.process == \"reHLT\": process.source.skipEvents=cms.untracked.uint32(3800)\n"' --ibeos -t 8
. Will need some special attention during relval since the DQMIO output file size will increase.Known problems: