Using new Run3ScoutingHitPattern class for hitPattern in Run-3 scouting #35685

mmasciov · 2021-10-15T02:51:03Z

PR description:

This PR is meant to address issue #32219.
It is introducing a new Run3ScoutingHitPattern class in DataFormats/Scouting, to be used for Run-3 scouting (in Run3ScoutingMuon). The class is a "frozen" copy of version 13 (present version) of reco::HitPattern.
A function is defined in order to fill Run3ScoutingHitPattern from reco::HitPattern as well as to define an IO rule for backwards compatibility (with version 3 of Run3ScoutingMuon).

PR validation:

Rerunning the HLT step ("TEST"), and reading hit pattern for "TEST" and for original "HLT" instance, same objects are retrieved:

Events->Scan("Run3ScoutingMuons_hltScoutingMuonPacker__TEST.obj.trk_run3ScoutingHitPattern_.hitCount:Run3ScoutingMuons_hltScoutingMuonPacker__HLT.obj.trk_run3ScoutingHitPattern_.hitCount")
***********************************************
*    Row   * Instance * Run3Scout * Run3Scout *
***********************************************
*        0 *        0 *           *           *
*        1 *        0 *        13 *        13 *
*        2 *        0 *        11 *        11 *
*        2 *        1 *        14 *        14 *
*        3 *        0 *        12 *        12 *
*        4 *        0 *        13 *        13 *
*        5 *        0 *           *           *
*        6 *        0 *        15 *        15 *
*        7 *        0 *           *           *
*        8 *        0 *           *           *
*        9 *        0 *        12 *        12 *
***********************************************

FYI @dsperka @jsalfeld

…ng data

cmsbuild · 2021-10-15T02:57:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35685/25976

This PR adds an extra 40KB to repository

cmsbuild · 2021-10-15T02:57:46Z

A new Pull Request was created by @mmasciov (Mario Masciovecchio) for master.

It involves the following packages:

DataFormats/Scouting (core)
DataFormats/TrackReco (reconstruction)
HLTrigger/Muon (hlt)

@smuzaffar, @Dr15Jones, @makortel, @cmsbuild, @missirol, @Martin-Grunewald, @slava77, @jpata can you please review it and eventually sign? Thanks.
@sscruz, @Fedespring, @missirol, @silviodonato, @trocino, @abbiendi, @JanFSchulte, @jhgoh, @VinInn, @Martin-Grunewald, @calderona, @HuguesBrun, @rovere, @gpetruc, @mmusich, @mtosi, @cericeci this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

makortel · 2021-10-15T15:17:31Z

Adding @davidlange6

makortel · 2021-10-15T15:21:17Z

DataFormats/Scouting/src/classes_def.xml

+	     nhp.fillRun3ScoutingHitPatternWithHitPattern(onfile.trk_hitPattern_);
+	     trk_run3ScoutingHitPattern_ = nhp;
+    ]]>
+  </ioread>


I think this would mean making the dependence on DataFormats/TrackReco permanent for DataFormats/Scouting. Given the RAW-like constraints/guarantees for DataFormats/Scouting I'd prefer to avoid that if possible.

(I understood from private discussion with @mmasciov that it is not necessary to be able to read EDM ROOT files with the version 3 of Run3ScoutingMuon; the streamer file format anyway does not support schema evolution)

After checking with @dsperka and the scouting group, it was concluded that indeed it is not necessary to be able to read ROOT files with version 3 of Run3ScoutingMuon.
Will therefore update the PR.

Martin-Grunewald · 2021-10-21T07:28:22Z

@mmasciov
Are you going to address the comment above?

mmasciov · 2021-10-21T08:54:39Z

@mmasciov Are you going to address the comment above?

@Martin-Grunewald Yes. Please, see also:
#35685 (comment)

slava77 · 2021-10-26T16:28:15Z

I'm trying to understand where this is going related to DataFormats/Scouting/interface/Run3ScoutingHitPattern.h and Run3ScoutingHitPattern.cc, which are essentially full copies of the DataFormats/TrackReco/interface/HitPattern.h

From the preceding discussion(s) in the scouting meeting I thought that the idea here was to POD-ify the data format of the current version of DataFormats/TrackReco/interface/HitPattern.h for storage. Beyond making a class that would allow to stream this version, I thought that there will just be a conversion module/method that can fill the standard DataFormats/TrackReco/interface/HitPattern.h so that all existing algorithms already using the HitPattern can be reused.

But perhaps I misinterpreted and the goal for the use cases is to have the full copy.

What will happen if we make some updates in DataFormats/TrackReco/interface/HitPattern.h interface (as we did already e.g. #35205 and #35288 ) without changing the persisted data layout? Wouldn't this need to be propagated to Run3ScoutingHitPattern?

BTW, is the current copy in Run3ScoutingHitPattern done after #35288 was merged (Sep 16)?

mmasciov · 2021-10-26T16:48:48Z

I'm trying to understand where this is going related to DataFormats/Scouting/interface/Run3ScoutingHitPattern.h and Run3ScoutingHitPattern.cc, which are essentially full copies of the DataFormats/TrackReco/interface/HitPattern.h

From the preceding discussion(s) in the scouting meeting I thought that the idea here was to POD-ify the data format of the current version of DataFormats/TrackReco/interface/HitPattern.h for storage. Beyond making a class that would allow to stream this version, I thought that there will just be a conversion module/method that can fill the standard DataFormats/TrackReco/interface/HitPattern.h so that all existing algorithms already using the HitPattern can be reused.

But perhaps I misinterpreted and the goal for the use cases is to have the full copy.

What will happen if we make some updates in DataFormats/TrackReco/interface/HitPattern.h interface (as we did already e.g. #35205 and #35288 ) without changing the persisted data layout? Wouldn't this need to be propagated to Run3ScoutingHitPattern?

Yes, this is what I'm currently working on to update this PR: given that we do not need to read the existing scouting samples that made use of HitPattern, I will just do as you describe above.

BTW, is the current copy in Run3ScoutingHitPattern done after #35288 was merged (Sep 16)?

Answer to this is yes, BTW.

slava77 · 2021-11-01T14:15:13Z

@cmsbuild please test

cmsbuild · 2021-11-01T15:39:13Z

-1

Failed Tests: Build HeaderConsistency ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-713d17/20155/summary.html
COMMIT: 35e09bd
CMSSW: CMSSW_12_2_X_2021-11-01-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35685/20155/install.sh to create a dev area with all the needed externals and cmssw changes.

Build

I found compilation error when building:

>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2021-11-01-1100/src/DataFormats/PatCandidates/src/Jet.cc
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2021-11-01-1100/src/DataFormats/PatCandidates/src/JetCorrFactors.cc
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2021-11-01-1100/src/DataFormats/PatCandidates/src/MHT.cc
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2021-11-01-1100/src/DataFormats/PatCandidates/src/MET.cc
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2021-11-01-1100/src/DataFormats/PatCandidates/src/Muon.cc
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2021-11-01-1100/src/DataFormats/PatCandidates/src/Conversion.cc:6:1: error: reference to 'Conversion' is ambiguous
    6 | Conversion::Conversion(int index) : vtxProb_(0.0), lxy_(0.0), nHitsMax_(0) { index_ = index; }
      | ^~~~~~~~~~
In file included from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2021-11-01-1100/src/DataFormats/PatCandidates/src/Conversion.cc:1:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_2_X_2021-11-01-1100/src/DataFormats/PatCandidates/interface/Conversion.h:21:9: note: candidates are: 'class pat::Conversion'
   21 |   class Conversion {

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

makortel · 2021-11-01T18:14:31Z

That is a strange compilation error for this PR.

slava77 · 2021-11-05T14:00:35Z

DataFormats/Scouting/interface/Run3ScoutingHitPattern.h

+
+class TrackerTopology;
+
+using namespace reco;


this is not allowed in header files global scope

slava77 · 2021-11-05T14:01:53Z

That is a strange compilation error for this PR.

I think that this is from using namespace reco

missirol · 2021-11-15T08:33:53Z

Kind ping. I'm wondering if this PR is still targeting 12_2_X (~Nov-30) or not.

slava77 · 2021-12-06T15:19:18Z

-1

just taking this off the list of currently active PRs for reco
some updates are expected anyways

makortel · 2022-01-31T17:16:08Z

What is the plan with this PR?

dsperka · 2022-01-31T17:29:01Z

@mmasciov should comment. We need to "freeze" the data format for the Run3 MC production soon.

makortel · 2022-02-22T16:42:44Z

Can this PR be expected to be updated? @mmasciov @dsperka

Or is some other resolution expected for #32219 ?

What is the timescale for freezing the Run 3 scouting data format? (12_3_0? 12_4_0?)

mmasciov · 2022-02-22T16:51:34Z

@dsperka, @makortel, I shall be able to be back on this in the coming days. Will probably close this PR and open a new one at this stage.

About the timescale, I let @dsperka answer as I am not sure.

dsperka · 2022-02-22T19:17:37Z

We need to freeze the scouting data format absolute latest by April 15th for the Run 3 MC production.

makortel · 2022-02-22T19:36:22Z

We need to freeze the scouting data format absolute latest by April 15th for the Run 3 MC production.

Thanks @dsperka. Just curious, could you clarify what is driving the schedule? I'm just wondering because it doesn't seem to correspond to the last open pre-release of either 12_3_X (March 8) or 12_4_X (May 10-15).

dsperka · 2022-02-22T19:44:01Z

The TSG set a deadline for the V2 HLT menu 1 month before the 12_4_X deadline

dsperka · 2022-03-02T14:38:42Z

@makortel @mmasciov It was mentioned in Today's TSG meeting that data taking will begin already with the V1 menu, and therefore the need to solve this is more urgent (in principle should have been done already). Mario, can you give a precise ETA?

mmasciov · 2022-03-05T19:31:34Z

Closing this PR in favor of #37149 (FYI: @dsperka, @makortel).

Using new Run3ScoutingHitPattern class for hitPattern in Run-3 scouti…

35e09bd

…ng data

cmsbuild added this to the CMSSW_12_1_X milestone Oct 15, 2021

cmsbuild added code-checks-pending core-pending hlt-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Oct 15, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 15, 2021

makortel reviewed Oct 15, 2021

View reviewed changes

smuzaffar modified the milestones: CMSSW_12_1_X, CMSSW_12_2_X Oct 29, 2021

cmsbuild added tests-started tests-rejected and removed tests-pending tests-started labels Nov 1, 2021

slava77 reviewed Nov 5, 2021

View reviewed changes

cmsbuild added reconstruction-rejected and removed reconstruction-pending labels Dec 6, 2021

smuzaffar modified the milestones: CMSSW_12_2_X, CMSSW_12_3_X Dec 6, 2021

cmsbuild mentioned this pull request Jan 31, 2022

Update Scouting Run3 format #36840

Merged

cmsbuild mentioned this pull request Feb 17, 2022

Add the detector ids to the scouting electron and photon collection for Run 3 #36990

Merged

makortel mentioned this pull request Feb 22, 2022

ROOT issue in RECO-only workflows #37013

Closed

mmasciov mentioned this pull request Mar 5, 2022

Revisit usage of HitPattern in DataFormats/Scouting #37149

Merged

mmasciov closed this Mar 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using new Run3ScoutingHitPattern class for hitPattern in Run-3 scouting #35685

Using new Run3ScoutingHitPattern class for hitPattern in Run-3 scouting #35685

mmasciov commented Oct 15, 2021 •

edited

cmsbuild commented Oct 15, 2021

cmsbuild commented Oct 15, 2021

makortel commented Oct 15, 2021

makortel Oct 15, 2021

mmasciov Oct 21, 2021

Martin-Grunewald commented Oct 21, 2021

mmasciov commented Oct 21, 2021

slava77 commented Oct 26, 2021

mmasciov commented Oct 26, 2021

slava77 commented Nov 1, 2021

cmsbuild commented Nov 1, 2021

makortel commented Nov 1, 2021

slava77 Nov 5, 2021

slava77 commented Nov 5, 2021

missirol commented Nov 15, 2021

slava77 commented Dec 6, 2021

makortel commented Jan 31, 2022

dsperka commented Jan 31, 2022

makortel commented Feb 22, 2022

mmasciov commented Feb 22, 2022

dsperka commented Feb 22, 2022

makortel commented Feb 22, 2022

dsperka commented Feb 22, 2022

dsperka commented Mar 2, 2022

mmasciov commented Mar 5, 2022

Using new Run3ScoutingHitPattern class for hitPattern in Run-3 scouting #35685

Using new Run3ScoutingHitPattern class for hitPattern in Run-3 scouting #35685

Conversation

mmasciov commented Oct 15, 2021 • edited

PR description:

PR validation:

cmsbuild commented Oct 15, 2021

cmsbuild commented Oct 15, 2021

makortel commented Oct 15, 2021

makortel Oct 15, 2021

Choose a reason for hiding this comment

mmasciov Oct 21, 2021

Choose a reason for hiding this comment

Martin-Grunewald commented Oct 21, 2021

mmasciov commented Oct 21, 2021

slava77 commented Oct 26, 2021

mmasciov commented Oct 26, 2021

slava77 commented Nov 1, 2021

cmsbuild commented Nov 1, 2021

Build

Clang Build

makortel commented Nov 1, 2021

slava77 Nov 5, 2021

Choose a reason for hiding this comment

slava77 commented Nov 5, 2021

missirol commented Nov 15, 2021

slava77 commented Dec 6, 2021

makortel commented Jan 31, 2022

dsperka commented Jan 31, 2022

makortel commented Feb 22, 2022

mmasciov commented Feb 22, 2022

dsperka commented Feb 22, 2022

makortel commented Feb 22, 2022

dsperka commented Feb 22, 2022

dsperka commented Mar 2, 2022

mmasciov commented Mar 5, 2022

mmasciov commented Oct 15, 2021 •

edited