Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

give OnlineLuminosityRecord info to HLT's LumiMonitor plugin #39859

Merged
merged 2 commits into from Nov 4, 2022

Conversation

missirol
Copy link
Contributor

@missirol missirol commented Oct 26, 2022

resolves #39756

PR description:

This PR address the 2nd item in the description of #39756, plus a related comment in #39756 (comment).

The LumiMonitor DQM plugin, used inside HLT, is updated to read the values of online instantaneous luminosity and mean PU from the so-called 'online-metadata' digis, when the corresponding info from SCAL is not available (i.e. from Run 3 onwards).

To improve clarity, some of the plot labels, transient variables, and configuration parameters are renamed. I changed "scal lumi/PU" to a more generic "online lumi/PU", but of course feel free to suggest better names.

Minor technical improvements (e.g. const-ness of class data members) were done in the process.

375912f fixes a couple of pixel-related harvesting outputs which depend on the outputs of the LumiMonitor plugin; please see #39859 (comment) for details.

(In case this is backported to 12_4_X, it should not warrant a new ConfDB parsing, since the new configuration parameter of LumiMonitor, i.e. onlineMetaDataDigis, already defaults to the intended collection; at the same time, hltOnlineMetaDataDigis should in principle be included in the DQMHistograms EndPath in the HLT menus.)

PR validation:

A local test confirms that the plots in question (profile vs PU, vs lumi) now have entries (see attachment).

If this PR is a backport, please specify the original PR and why you need to backport that PR. If this PR will be backported, please specify to which release cycle the backport is meant for:

CMSSW_12_4_X
CMSSW_12_5_X

Copy link
Contributor Author

@missirol missirol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments inline.

online_pu = scalit->pileup();
} else if (onlineMetaDataDigisHandle.isValid()) {
online_lumi = onlineMetaDataDigisHandle->instLumi();
online_pu = onlineMetaDataDigisHandle->avgPileUp();
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic of this choice was inspired by

if (dcsStatus.isValid() && !dcsStatus->empty()) {

and then I noticed it is basically the same as

if (lumiScalers.isValid() && !lumiScalers->empty()) {

process.fastTimerServiceClient.doPlotsVsPixelLumi = False
process.fastTimerServiceClient.scalLumiME = cms.PSet(
process.fastTimerServiceClient.onlineLumiME = cms.PSet(
folder = cms.string('HLT/LumiMonitoring'),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only place I could find where the parameter names appeared, and needed to be changed.

Comment on lines +91 to +92
trkTopoToken_(doPixelLumi_ ? esConsumes<TrackerTopology, TrackerTopologyRcd>()
: edm::ESGetToken<TrackerTopology, TrackerTopologyRcd>()),
Copy link
Contributor Author

@missirol missirol Oct 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is meant to avoid consuming the record when the plugin does not make use of it.

The following would be slightly more compact

Suggested change
trkTopoToken_(doPixelLumi_ ? esConsumes<TrackerTopology, TrackerTopologyRcd>()
: edm::ESGetToken<TrackerTopology, TrackerTopologyRcd>()),
trkTopoToken_(doPixelLumi_ ? esConsumes<TrackerTopology, TrackerTopologyRcd>() : trkTopoToken_),

but I haven't seen this approach elsewhere, and I don't know if this is a good idea.

The even-more-compact version

Suggested change
trkTopoToken_(doPixelLumi_ ? esConsumes<TrackerTopology, TrackerTopologyRcd>()
: edm::ESGetToken<TrackerTopology, TrackerTopologyRcd>()),
trkTopoToken_(doPixelLumi_ ? esConsumes() : trkTopoToken_),

unfortunately didn't compile:

error: operands to '?:' have different types 'edm::EDConsumerBaseESAdaptor<edm::Transition::Event>' and 'const edm::ESGetToken<TrackerTopology, TrackerTopologyRcd>'
   91 |       trkTopoToken_(doPixelLumi_ ? esConsumes()
      |                     ~~~~~~~~~~~~~^~~~~~~~~~~~~~
   92 |                                  : trkTopoToken_),
      |                                  ~~~~~~~~~~~~~~~

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an aside: if there is feedback on this technical point, I'd be interested to know (doing x_{bool ? a : x_} in the initialiser list, and the fact that one of the options doesn't compile, which is probably expected, but I'm wondering if there is a way to make it work).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem like a good idea, because x_ would end up uninitialised ?

Simple example using integers and strings:

#include <iostream>
#include <string>

struct Data {
public:
  Data(int x, std::string s) : x_{x}, s_{std::move(s)}
  {}

  Data() : x_{x_}, s_{s_}
  {}

  int x() const {
    return x_;
  }

  std::string const& s() const {
    return s_;
  }

private:
  int x_;
  std::string s_;
};

int main(void) {

  Data value{42, "answer"};
  std::cout << value.s() << " is " << value.x() << std::endl;

  Data data;
  std::cout << data.s() << " is " << data.x() << std::endl;

  return 0;
}

The compiler warns that x_ is initialised:

$ g++ -std=c++17 -O3 -flto -g -Wall -fPIC -MMD -march=native -mtune=native  -c test.cc -o test.cc.o
test.cc: In member function ‘Data::Data()’:
test.cc:9:15: warning: ‘this_6(D)->x_’ is used uninitialized [-Wuninitialized]
    9 |   Data() : x_{x_}, s_{s_}
      |               ^~

Worse, there is no warning about s_ being uninitialised, but I get a crash:

$ ./test 
answer is 42
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed.. thanks the example, it's clearer now!

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39859/32758

  • This PR adds an extra 20KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @missirol (Marino Missiroli) for master.

It involves the following packages:

  • DQM/HLTEvF (dqm, hlt)
  • DQM/Integration (dqm)
  • HLTrigger/Timer (hlt)

@Martin-Grunewald, @emanueleusai, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@batinkov, @battibass, @silviodonato, @mtosi, @Martin-Grunewald, @fwyzard, @threus, @francescobrivio this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

histograms.lumiVsLS = me;

me = booker.bookProfile("puVsLS",
"scal PU vs LS",
"online PU vs LS",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you also change PU to pileup and LS to lumisection in the labels ?

@@ -267,6 +262,7 @@ void LumiMonitor::fillDescriptions(edm::ConfigurationDescriptions& descriptions)
edm::ParameterSetDescription desc;
desc.add<edm::InputTag>("pixelClusters", edm::InputTag("hltSiPixelClusters"));
desc.add<edm::InputTag>("scalers", edm::InputTag("hltScalersRawToDigi"));
desc.add<edm::InputTag>("onlineMetaDataDigis", edm::InputTag("hltOnlineMetaDataDigis"));
desc.add<std::string>("FolderName", "HLT/LumiMonitoring");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you change this to folderName, for consistency with all the other parameters that are lowercase ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. For the 12_4_X backport, we might want to avoid this imho, to spare FOG from template migrations just because of this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

How are the changes to the client configuration propagated to the online DQM ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good question. So far, I only found one place, #39859 (comment). I'll double-check, and hopefully DQM reviewers will know better than me.

if (doPlotsVsScalLumi_)
fillPlotsVsLumi(booker, getter, current_path, "VsScalLumi", scalLumiMEPSet_);
if (doPlotsVsOnlineLumi_)
fillPlotsVsLumi(booker, getter, current_path, "vs_lumi", onlineLumiMEPSet_);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fillPlotsVsLumi(booker, getter, current_path, "vs_lumi", onlineLumiMEPSet_);
fillPlotsVsLumi(booker, getter, current_path, "_vs_lumi", onlineLumiMEPSet_);

if (doPlotsVsPixelLumi_)
fillPlotsVsLumi(booker, getter, current_path, "VsPixelLumi", pixelLumiMEPSet_);
fillPlotsVsLumi(booker, getter, current_path, "vs_pixelLumi", pixelLumiMEPSet_);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fillPlotsVsLumi(booker, getter, current_path, "vs_pixelLumi", pixelLumiMEPSet_);
fillPlotsVsLumi(booker, getter, current_path, "_vs_pixelLumi", pixelLumiMEPSet_);

if (doPlotsVsPU_)
fillPlotsVsLumi(booker, getter, current_path, "VsPU", puMEPSet_);
fillPlotsVsLumi(booker, getter, current_path, "vs_pileup", puMEPSet_);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fillPlotsVsLumi(booker, getter, current_path, "vs_pileup", puMEPSet_);
fillPlotsVsLumi(booker, getter, current_path, "_vs_pileup", puMEPSet_);

if (doPlotsVsScalLumi_)
fillPlotsVsLumi(booker, getter, subsubdir, "VsScalLumi", scalLumiMEPSet_);
if (doPlotsVsOnlineLumi_)
fillPlotsVsLumi(booker, getter, subsubdir, "vs_lumi", onlineLumiMEPSet_);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fillPlotsVsLumi(booker, getter, subsubdir, "vs_lumi", onlineLumiMEPSet_);
fillPlotsVsLumi(booker, getter, subsubdir, "_vs_lumi", onlineLumiMEPSet_);

and so on for the others

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mh.. but one underscore is added here (I also added a screenshot to the description). Or do we really want the double underscore?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah... no, we definitely do not want the double underscore :-)

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39859/32763

  • This PR adds an extra 32KB to repository

@cmsbuild
Copy link
Contributor

Pull request #39859 was updated. @Martin-Grunewald, @emanueleusai, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

@missirol
Copy link
Contributor Author

Tried to address the comments received so far.

Regarding #39859 (comment), I probably need feedback from @cms-sw/dqm-l2. If it's useful, before we consider merging this and backporting it, I could make a 124X version of this PR to be tested in the online DQM (we would not see the empty HLT plots getting filled, because that requires rerunning the HLT, but we would verify that things still work okay).

@fwyzard
Copy link
Contributor

fwyzard commented Oct 26, 2022

👍🏻

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39859/32765

  • This PR adds an extra 32KB to repository

@missirol
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39859/32855

  • This PR adds an extra 12KB to repository

@cmsbuild
Copy link
Contributor

Pull request #39859 was updated. @Martin-Grunewald, @emanueleusai, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

@missirol
Copy link
Contributor Author

please test

In Run-3 data wfs, harvesting outputs vs_lumi and vs_pileup now have entries as expected: example.

The bin-by-bin DQM differences also show two TProfiles in PixelPhase1/Tracks/hitefficiency_per_scalLumi_*; both are built based on information from the TProfile lumiVsLS, which this PR touches. For the wfs checked in PR tests, these two pixel plots seem to be somewhat broken (example from wf 11634.0, example from wf 139.001). Maybe this is because they are created with xmin == xmax (yMax2 == 0). In MC (e.g. wf 11634.0), this happens because the lumiVsLS TProfile has GetMaximum == 0. In wf 139.001 (data), lumiVsLS.GetMaximum = 0.013, but hitefficiency_per_scalLumi_*.xmax == xmin == 0, because yMax2 is of type int (so 0.013 becomes a zero).

I tried to fix this in 375912f; in the MC case, the 2 profiles wrt lumi (which would always be empty) are not produced anymore.

I don't have other changes planned for this PR.

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-62cf54/28647/summary.html
COMMIT: 375912f
CMSSW: CMSSW_12_6_X_2022-10-30-0000/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39859/28647/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 10 differences found in the comparisons
  • DQMHistoTests: Total files compared: 48
  • DQMHistoTests: Total histograms compared: 3416356
  • DQMHistoTests: Total failures: 97
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3416237
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 55.515 KiB( 47 files compared)
  • DQMHistoSizes: changed ( 1000.0,... ): 0.039 KiB DQM/TimerService
  • DQMHistoSizes: changed ( 138.5,... ): 27.582 KiB PixelPhase1/Tracks
  • Checked 206 log files, 48 edm output root files, 48 DQM output files
  • TriggerResults: no differences found

@missirol
Copy link
Contributor Author

missirol commented Oct 30, 2022

+hlt

@missirol
Copy link
Contributor Author

missirol commented Nov 1, 2022

@cms-sw/dqm-l2 , any comments on this PR? It would be nice to have its backport merged in 12_4_X sooner rather than later.

@emanueleusai
Copy link
Member

testing the 12_4 backport at p5 now, apologies for the delay

@syuvivida
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 3, 2022

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

perrotta commented Nov 4, 2022

+1

@cmsbuild cmsbuild merged commit 56f444d into cms-sw:master Nov 4, 2022
@missirol missirol deleted the devel_onlineMetaDataToLumiMon branch November 5, 2022 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Issues related to FastTimerService and HLT
6 participants