Update of mkFit for 12_1_0_pre2 #34921

mmasciov · 2021-08-17T17:33:54Z

PR description:

This PR follows #34395 and updates mkFit for physics validation in CMSSW_12_1_0_pre2 (https://its.cern.ch/jira/browse/PDMVRELVALS-130).

In detail, this PR includes:

update usage of SiStrip quality DB information for mkFit;
enable usage of SiStrip quality DB information for mkFit;
enable by default only InitialStepPreSplitting, InitialStep, HighPtTripletStep and DetachedQuadStep in trackingMkFit procModifier;
other technical optimizations.

It requires cms-sw/cmsdist#7217 and cms-data/RecoTracker-MkFit#3

Developments have been presented at Tracking POG (https://indico.cern.ch/event/1065324/#3-mkfit-status-report) and agreed with PPD (https://indico.cern.ch/event/1063451/#8-mkfit-report).

PR validation:

MTV results for tracking-only validation in TTbar RelVal (with mkFit enabled for 4 iterations as specified above):

on /RelValTTbar_14TeV/CMSSW_12_0_0_pre6-PU_120X_mcRun3_2021_realistic_v4-v1/GEN-SIM-DIGI-RAW: https://mmasciov.web.cern.ch/mmasciov/MkFitCMSSW/MTV_mkFitPR_TTbarPU/
on /RelValTTbar_14TeV/CMSSW_12_0_0_pre6-PU_120X_mcRun3_2021_realistic_v4_JIRA_129-v1/GEN-SIM-DIGI-RAW: https://mmasciov.web.cern.ch/mmasciov/MkFitCMSSW/MTV_mkFitPR_TTbarPU_JIRA129/

FYI @mtosi @vmariani @mmusich
(@slava77 @makortel)

…, HighPtTripletStep and DetachedQuadStep

… module is inactive

cmsbuild · 2021-08-17T17:40:16Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34921/24731

This PR adds an extra 20KB to repository

cmsbuild · 2021-08-17T17:40:41Z

A new Pull Request was created by @mmasciov (Mario Masciovecchio) for master.

It involves the following packages:

Configuration/ProcessModifiers (operations)
RecoTracker/MkFit (reconstruction)

@perrotta, @silviodonato, @cmsbuild, @slava77, @jpata, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@fabiocos, @makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @ebrondol, @gpetruc, @mmusich, @mtosi, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

slava77 · 2021-08-17T17:48:52Z

test parameters:

workflow = 11634.7
pull_request = Update mkfit to V3.1.1 cmsdist#7217,Update per-iteration JSON files for mkFit configuration (121X) cms-data/RecoTracker-MkFit#3

slava77 · 2021-08-17T17:49:32Z

@cmsbuild please test

cmsbuild · 2021-08-17T21:47:58Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b18488/17834/summary.html
COMMIT: 30567c1
CMSSW: CMSSW_12_1_X_2021-08-17-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34921/17834/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 3000352
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3000318
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 38 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

jpata

The reco changes in the TICL tracksters do not look related to this PR (not sure about the reason at this moment). Minor comments in the code inline.

@mtosi @vmariani please let me know if you have any comments (with this targeting pre2, closing on Monday)

jpata · 2021-08-18T13:23:17Z

RecoTracker/MkFit/plugins/MkFitEventOfHitsProducer.cc

+          if (isBad)
+            LogTrace("SiStripBadComponents") << "bad apv " << apv << " on " << bs.detid;
+          if (isBad) {
+            if (lastApv == -1) {


why not like this? Is there something special about LogTrace?

Suggested change

if (isBad)

LogTrace("SiStripBadComponents") << "bad apv " << apv << " on " << bs.detid;

if (isBad) {

if (lastApv == -1) {

if (isBad) {

LogTrace("SiStripBadComponents") << "bad apv " << apv << " on " << bs.detid;

if (lastApv == -1) {

yes, the suggestion is OK, I was moving these lines a few too many times.
LogTrace does not have additional header/footer message formatting and is easier to parse than LogDebug.

Updated in commit 91d1caf

jpata · 2021-08-18T13:24:04Z

RecoTracker/MkFit/plugins/MkFitEventOfHitsProducer.cc

+          dv.push_back({phi1, phi2, q1, q2});
+        };
+
+        for (int apv = 0; apv < 6; ++apv) {


might the apv max number be a named constant?

@pieterdavid @mmusich
do we have max APVs and n strips per APV formalized in common constants somewhere?
I tried to git grep, but all that I found was hardcoded 6 and 128.

what about modules with less than 6 APVs?

do we have max APVs and n strips per APV formalized in common constants somewhere?

they are (at least) in this file: https://github.com/cms-sw/cmssw/blob/master/CalibTracker/SiStripCommon/data/SiStripDetInfo.dat.
Or you can do something like:

unsigned short nApvs = stripDet->specificTopology().nstrips() / 128;

128 is univeral for all modules, I am not sure there's a constant somewhere.

the constants are also here (but they aren't used much in the strips code): https://github.com/cms-sw/cmssw/blob/master/DataFormats/SiStripCommon/interface/ConstantsForHardwareSystems.h#L38-L43 (6 as max number of APVs indirectly, as 2 per optical link * max 3 of these per module - it may be worth adding)

what about modules with less than 6 APVs?

it shouldn't matter here, if the BadApvs is filled correctly.

Updated in commit 91d1caf

…on directly included packages

cmsbuild · 2021-08-18T18:59:17Z

Pull request #34921 was updated. @perrotta, @cmsbuild, @slava77, @jpata, @qliphy, @fabiocos, @davidlange6 can you please check and sign again.

slava77 · 2021-08-18T19:02:56Z

@cmsbuild please test

cmsbuild · 2021-08-18T23:06:11Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b18488/17871/summary.html
COMMIT: 91d1caf
CMSSW: CMSSW_12_1_X_2021-08-18-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34921/17871/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 3000352
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3000324
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 38 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

jpata · 2021-08-19T13:08:57Z

Thanks for the update. I have no more comments myself. I can sign after waiting until e.g. tomorrow if something comes up from the TRK groups.

mmasciov · 2021-08-22T21:59:42Z

Thanks for the update. I have no more comments myself. I can sign after waiting until e.g. tomorrow if something comes up from the TRK groups.

As I understand pre2 is closing tomorrow or soon after and there was no new comment since Thursday, is there anything preventing this PR (+related ones) from being merged? Thanks.

jpata · 2021-08-23T08:53:23Z

+reconstruction

no reco changes in standard workflows (mkFit workflow is not tracked)
requires Update mkfit to V3.1.1 cmsdist#7217 and Update per-iteration JSON files for mkFit configuration (121X) cms-data/RecoTracker-MkFit#3

perrotta · 2021-08-25T05:39:04Z

+operations

cmsbuild · 2021-08-25T05:39:27Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2021-08-25T05:40:30Z

@smuzaffar should we wait for you merging cms-sw/cmsdist#7229, or will you do it as soon as we merge this PR in cmsssw?

smuzaffar · 2021-08-25T06:25:53Z

@perrotta , you can also merge the cmsdist PRs ( cms-sw/cmsdist#7217 and cms-sw/cmsdist#7229 ) when you merge this one. just issue merge comment on those PRs :-)

perrotta · 2021-08-25T06:35:10Z

@perrotta , you can also merge the cmsdist PRs ( cms-sw/cmsdist#7217 and cms-sw/cmsdist#7229 ) when you merge this one. just issue merge comment on those PRs :-)

Thank you @smuzaffar, I will.
Since I'm kind of new in the game, I just wanted to be sure that I did not bypass your signature without your agreement

perrotta · 2021-08-25T06:35:27Z

+1

mmasciov added 5 commits August 16, 2021 07:34

Enable mkFit by default only for InitialStepPreSplitting, InitialStep…

c9dd8e6

…, HighPtTripletStep and DetachedQuadStep

Enable usage of SiStrip quality DB information by default for mkFit

8d21af8

Restrict SiStrip DB information to invalid APVs if only a part of the…

fbbdad2

… module is inactive

From code checks/format

2b3e7b3

Update README with description of mkFit-specific parameters

30567c1

mmasciov mentioned this pull request Aug 17, 2021

Update per-iteration JSON files for mkFit configuration (121X) cms-data/RecoTracker-MkFit#3

Merged

cmsbuild added this to the CMSSW_12_1_X milestone Aug 17, 2021

cmsbuild added code-checks-pending operations-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Aug 17, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 17, 2021

cmsbuild added the requires-external label Aug 17, 2021

cmsbuild added tests-started and removed tests-pending labels Aug 17, 2021

cmsbuild added tests-approved and removed tests-started labels Aug 17, 2021

jpata reviewed Aug 18, 2021

View reviewed changes

Use constants and derive nAPVs from the strip topology, and add deps …

91d1caf

…on directly included packages

cmsbuild added code-checks-pending and removed tests-approved code-checks-approved labels Aug 18, 2021

cmsbuild added the code-checks-approved label Aug 18, 2021

cmsbuild added tests-started and removed tests-pending labels Aug 18, 2021

cmsbuild added tests-approved and removed tests-started labels Aug 18, 2021

smuzaffar mentioned this pull request Aug 20, 2021

Update mkfit to V3.1.1 cms-sw/cmsdist#7217

Merged

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Aug 23, 2021

mrodozov mentioned this pull request Aug 23, 2021

Update tag for RecoTracker-MkFit to V00-03-00 cms-sw/cmsdist#7229

Merged

cmsbuild added fully-signed operations-approved and removed operations-pending pending-signatures labels Aug 25, 2021

cmsbuild added orp-approved and removed orp-pending labels Aug 25, 2021

cmsbuild merged commit 3c88baa into cms-sw:master Aug 25, 2021

cmsbuild mentioned this pull request Aug 27, 2021

DataFormats/FEDRawData UBSAN warning fixed #35039

Merged

mmasciov mentioned this pull request Sep 8, 2021

Update of mkFit for 12_1_0_pre3 #35199

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update of mkFit for 12_1_0_pre2 #34921

Update of mkFit for 12_1_0_pre2 #34921

mmasciov commented Aug 17, 2021

cmsbuild commented Aug 17, 2021

cmsbuild commented Aug 17, 2021

slava77 commented Aug 17, 2021

slava77 commented Aug 17, 2021

cmsbuild commented Aug 17, 2021

jpata left a comment

jpata Aug 18, 2021

slava77 Aug 18, 2021

mmasciov Aug 18, 2021

jpata Aug 18, 2021

slava77 Aug 18, 2021

mmusich Aug 18, 2021

mmusich Aug 18, 2021

pieterdavid Aug 18, 2021 •

edited

slava77 Aug 18, 2021

mmasciov Aug 18, 2021

cmsbuild commented Aug 18, 2021

slava77 commented Aug 18, 2021

cmsbuild commented Aug 18, 2021

jpata commented Aug 19, 2021

mmasciov commented Aug 22, 2021 •

edited

jpata commented Aug 23, 2021

perrotta commented Aug 25, 2021

cmsbuild commented Aug 25, 2021

perrotta commented Aug 25, 2021

smuzaffar commented Aug 25, 2021

perrotta commented Aug 25, 2021

perrotta commented Aug 25, 2021

Update of mkFit for 12_1_0_pre2 #34921

Update of mkFit for 12_1_0_pre2 #34921

Conversation

mmasciov commented Aug 17, 2021

PR description:

PR validation:

cmsbuild commented Aug 17, 2021

cmsbuild commented Aug 17, 2021

slava77 commented Aug 17, 2021

slava77 commented Aug 17, 2021

cmsbuild commented Aug 17, 2021

Comparison Summary

jpata left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pieterdavid Aug 18, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmsbuild commented Aug 18, 2021

slava77 commented Aug 18, 2021

cmsbuild commented Aug 18, 2021

Comparison Summary

jpata commented Aug 19, 2021

mmasciov commented Aug 22, 2021 • edited

jpata commented Aug 23, 2021

perrotta commented Aug 25, 2021

cmsbuild commented Aug 25, 2021

perrotta commented Aug 25, 2021

smuzaffar commented Aug 25, 2021

perrotta commented Aug 25, 2021

perrotta commented Aug 25, 2021

pieterdavid Aug 18, 2021 •

edited

mmasciov commented Aug 22, 2021 •

edited