Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate DQMEDAnalyzers from stream to one modules #22218

Merged
merged 2 commits into from Feb 20, 2018

Conversation

fwyzard
Copy link
Contributor

@fwyzard fwyzard commented Feb 14, 2018

Migrate all DQMEDAnalyzer modules from edm::stream::EDAnalyzer<...> to edm::one::EDAnalyzer<...>.

This should significantly reduce the memory usage of a multithreaded DQM job, at the price of preventing the framework from analysing multiple runs or lumisections at the same time.

Lumisection-based histograms are handled in the postModuleGlobalEndLumi signal of the DQMStore.

If DQMEDanalyzer modules are migrated from EDAnalyzer to EDProduce<Accumulator>, that signal should be removed and the handling moved to the modules' endLuminosityBlockProduce method.

@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@fwyzard
Copy link
Contributor Author

fwyzard commented Feb 14, 2018

First attempt at a simpler, partial migration for #22157 .

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

DQMServices/Core

@vazzolini, @kmaeshima, @dmitrijus, @cmsbuild, @jfernan2, @vanbesien can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

@fwyzard
Copy link
Contributor Author

fwyzard commented Feb 14, 2018

@cmsbuild please test

@cmsbuild
Copy link
Contributor

The tests are being triggered in jenkins.

@schneiml
Copy link
Contributor

relative to #22157, I see

  • pro: no mess with config changes.
  • pro: no issues related to the new edm::Accumulator feature.
  • con: these two were not the problems discussed in Migrate DQM to edm::Accumulator #22157.
  • con: no idea if there is a disadavantage in using EDAnalyzer, what does @Dr15Jones think?

@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@fwyzard
Copy link
Contributor Author

fwyzard commented Feb 14, 2018

@cmsbuild please test

@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Pull request #22218 was updated. @vazzolini, @kmaeshima, @dmitrijus, @cmsbuild, @jfernan2, @vanbesien can you please check and sign again.

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Comparison job queued.

@dmitrijus
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-22218/26190/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 32 differences found in the comparisons
  • DQMHistoTests: Total files compared: 28
  • DQMHistoTests: Total histograms compared: 2498032
  • DQMHistoTests: Total failures: 9
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2497847
  • DQMHistoTests: Total skipped: 176
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.829999999951 KiB( 23 files compared)
  • Checked 115 log files, 9 edm output root files, 28 DQM output files

@fabiocos
Copy link
Contributor

+1

@@ -856,7 +707,7 @@ DQMStore::book_(const std::string &dir, const std::string &name,
{
// Create and initialise core object.
assert(dirs_.count(dir));
MonitorElement proto(&*dirs_.find(dir), name, run_, streamId_, moduleId_);
MonitorElement proto(&*dirs_.find(dir), name, run_, moduleId_);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fwyzard ugh... What have you done there? Did you also remove the streamId from the MonitorElement constructor? It should be removed, but the change is not in this PR, so now streamIds and moduleIds are messed up!

@fwyzard
Copy link
Contributor Author

fwyzard commented Feb 23, 2018 via email

@fwyzard
Copy link
Contributor Author

fwyzard commented Feb 26, 2018

@schneiml from a cursory look, I think I have updated all the calls to the MonitorElement constructors.

If you can point me where you see an inconsistency between the streamId and moduleId I will try to fix it.

@schneiml
Copy link
Contributor

@fwyzard streamId is still in the constructor declaration, so you are passing moduleId instead of streamId. Which would be sort of fine if it was used consistently, but some places do nontrival queries on the moduleId (e.g., save()). Just remove streamId from the constructor and see where it fails to compile...

@fwyzard
Copy link
Contributor Author

fwyzard commented Feb 26, 2018

Ah, I see... and it was not spot before by the compiler because of all the default parameters.
OK, let's see if #22364 helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants