Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save DQM output per lumisection for UL rereco #26232

Merged
merged 18 commits into from Apr 18, 2019

Conversation

schneiml
Copy link
Contributor

@schneiml schneiml commented Mar 21, 2019

PR description:

This PR enables per-lumisection saving of all DQM output for the UL rereco. This wish came up for machine learning studies targeting per-LS data certification or online monitoring.

In prompt reco, we have per-LS DQM output available due to how Tier0 processes only ~1 LS per job, and the per-job DQMIO output files are available. It would be very nice to have the same granularity of data preserved for the UL rereco, which in addition provides the full run2 dataset processed using the same software.

This is safe to do for most of DQM, since HARVESTING will re-generate per run/job statistics from the per-lumisection snapshots. It is not save for legacy modules that modify MEs at non-standard times or that do not support merging correctly; none of these should exist in RECO/DQM step since the threaded migration, but since there are still a few, I apply the change only to the non-legacy MEs.

PR validation:

Validated on a modified 136.85 that crosses a lumi boundary: runTheMatrix.py -l 136.85 --command '--customise_commands "if process.process == \"reHLT\": process.source.skipEvents=cms.untracked.uint32(3800)\n"' --ibeos -t 8. Will need some special attention during relval since the DQMIO output file size will increase.

Known problems:

  • Various "known unstable" plots change behaviour. These change behaviour on trivial changes such as adding debug output, and might be caused by race conditions.
  • The behaviour of per-lumi plots in harvesting changes. This can lead to "double counting" in the existing offline per-lumi harvesting code, and cause subtly incorrect results. Since there is very little such code, I manually adapted the places I could find; however, due to the freedom that harvesting code has, I can not guarantee that these are all cases.

@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26232/8861

  • This PR adds an extra 16KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @schneiml (Marcel Schneider) for master.

It involves the following packages:

DQMServices/Core

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

@andrius-k
Copy link

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 22, 2019

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/33717/console Started: 2019/03/22 12:31

@cmsbuild
Copy link
Contributor

-1

Tested at: a4918f5

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26232/33717/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

  • Unit Tests:

I found errors in the following unit tests:

---> test TestDQMServicesFwkIOScripts had ERRORS

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26232/33717/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 2 differences found in the comparisons
  • DQMHistoTests: Total files compared: 32
  • DQMHistoTests: Total histograms compared: 3114829
  • DQMHistoTests: Total failures: 221
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3114411
  • DQMHistoTests: Total skipped: 197
  • DQMHistoTests: Total Missing objects: 0

@schneiml
Copy link
Contributor Author

One of the things that broke in this PR are the METRate plots in multiple places. These seem to be the only 15 differences that RelMon observes. Judging by the code that makes them

totltime = double(totlsec*90); // one lumi sec ~ 90 (sec)
}
if (totltime==0.) totltime=1.;
std::string dirName = FolderName_+metCollectionLabel_.label()+"/";
//dbe_->setCurrentFolder(dirName);
//below is the original METAnalyzer formulation
for (std::vector<std::string>::const_iterator ic = folderNames_.begin(); ic != folderNames_.end(); ic++) {
std::string DirName;
DirName = dirName+*ic;
makeRatePlot(DirName,totltime);

this is not too big of a loss, given this type of operation in endRun was not supported since at least the threaded migration of 2015 (it would need to happen in harvesting), and since the code assumes that a lumisection is 90 seconds I would not trust the computed values anyways.

@schneiml
Copy link
Contributor Author

The more detailed bin-to-bin comparison shows 1000's of changes, waiting for another run to get the output uploaded to DQMGUI. I hope some of the PR tests actually cover more than one lumisection, to make this a meaningful test...

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26232/33717/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 2 differences found in the comparisons
  • DQMHistoTests: Total files compared: 32
  • DQMHistoTests: Total histograms compared: 3114829
  • DQMHistoTests: Total failures: 221
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3114411
  • DQMHistoTests: Total skipped: 197
  • DQMHistoTests: Total Missing objects: 0

@schneiml
Copy link
Contributor Author

The bin-by-bin comparison noticed that the reference histograms stored by CMSSW turned empty. Though, looking at the names, we don't produce these MEs; the reference data must be from run1, since we have not produced e.g. 'TEC/side1' MEs since 2013 (was renamed to 'TEC/MINUS').

Not sure if these references are still used anywhere (in offline), I'll try to get them back but it might not be worth the effort.

@davidlange6
Copy link
Contributor

davidlange6 commented Mar 25, 2019 via email

@schneiml
Copy link
Contributor Author

@davidlange6 , no, I am talking about the reference that is handled inside the DQMStore and saved into the legacy root file going to the DQMGUI. I believe this mechanism was not used in Run2, except maybe for a few subsystems in online. Consequently, I have very little of an idea how it works, except for the fact that it exists...

@cmsbuild
Copy link
Contributor

Pull request #26232 was updated. @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @franzoni, @jfernan2, @fioriNTU, @fabiocos, @davidlange6 can you please check and sign again.

@andrius-k
Copy link

please abort

@andrius-k
Copy link

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 15, 2019

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34201/console Started: 2019/04/15 11:45

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26232/34201/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 32
  • DQMHistoTests: Total histograms compared: 3140767
  • DQMHistoTests: Total failures: 1643
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3138927
  • DQMHistoTests: Total skipped: 197
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -601.622 KiB( 31 files compared)
  • DQMHistoSizes: changed ( 11624.0,... ): -42.973 KiB CTPPS/TrackingStrip
  • Checked 133 log files, 14 edm output root files, 32 DQM output files

@andrius-k
Copy link

+1

@@ -1935,6 +1935,8 @@ def expandMapping(self,seqList,mapping,index=None):
def prepare_DQM(self, sequence = 'DQMOffline'):
# this one needs replacement

# any 'DQM' job should use DQMStore in non-legacy mode (but not HARVESTING)
self.loadAndRemember("DQMServices/Core/DQMStoreNonLegacy_cff")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@schneiml this essentially sets always DQMStore.enableMultiThread = True, and it should not change behaviour for multi-threaded workflows in my understanding, and not prevent HARVESTING to run anyway, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is completely independent of HARVESTING, which should be configured somewhere else.

And yes, all serious workflows run with enableMultiThread = True anyways, so it makes sense to turn it on even on single-threaded workflows to get consistent behavior.

@@ -52,7 +52,7 @@ def mergeProcess(*inputFiles, **options):
#//
if newDQMIO:
process.source = Source("DQMRootSource")
process.add_(Service("DQMStore"))
process.add_(Service("DQMStore", forceResetOnBeginLumi = CfgTypes.untracked.bool(True)))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@schneiml if the feature needs to be off by default, should not this flag be turned to False?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the changes in HARVESTING behavior are spread over many places and unconditional. This fixes a number of problems with the existing per-lumi plots (at the risk of causing some misbehavior in said plots, as explained in the PR message).

Conditional is only the change in the RECO step, which turns on per-lumi saving on all plots (that allow it).

@fabiocos
Copy link
Contributor

@schneiml could you please address my two questions? I would like to move forward with this PR before pre4 if possible, I just need a clarification

@fabiocos
Copy link
Contributor

+operations

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

@fabiocos
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 660613c into cms-sw:master Apr 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants