SiStrip Conditions tools: add `SiStripBadChannelPatcher` and related configurations #39534

mmusich · 2022-09-29T10:20:27Z

PR description:

It is from time to time necessary to patch preexisting SiStripBadStrip conditions objects to add and / or remove some DetIds or even full FEDs from the list of bad components.
A recent occurrence of this lead to the creation of the object proposed in this cmsTalk thread.
This PR adds a new plugin SiStripBadChannelPatcher in order to perform such operations, the related configuration and adds it to the existing battery of unit tests.

PR validation:

scram b runtests_testCondToolsSiStripBuildersReaders was run successfully.
I have also run the following script:

#!/bin/bash
cmsRun SiStripBadChannelPatcher_cfg.py

conddb_import -f sqlite_file:SiStripBadComponents_merged_v0.db -c sqlite_file:toCompare.db -i SiStripBadComponents_merged_v0 -t toCompare
conddb_import -f sqlite_file:SiStripBadComponents_merged_noFED124withFED50.db -c sqlite_file:toCompare.db -i SiStripBadComponents_merged_noFED124withFED50 -t toCompare -b 359334
getPayloadData.py --plugin pluginSiStripBadStrip_PayloadInspector --plot plot_SiStripBadStripFractionComparisonTrackerMapSingleTag --tag toCompare --time_type Run --iovs '{"start_iov": "1", "end_iov": "359334"}' --db sqlite_file:toCompare.db --test ;
getPayloadData.py --plugin pluginSiStripBadStrip_PayloadInspector --plot plot_SiStripBadStripFractionTrackerMap --tag toCompare --time_type Run --iovs '{"start_iov": "1", "end_iov": "1"}' --db sqlite_file:toCompare.db --test ;
getPayloadData.py --plugin pluginSiStripBadStrip_PayloadInspector --plot plot_SiStripBadStripFractionTrackerMap --tag toCompare --time_type Run --iovs '{"start_iov": "359334", "end_iov": "359334"}' --db  sqlite_file:toCompare.db --test ;

and obtained the following plots:

Source	Target	Comparison

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

N/A

cmsbuild · 2022-09-29T10:28:09Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39534/32306

This PR adds an extra 16KB to repository

cmsbuild · 2022-09-29T10:28:37Z

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

CondTools/SiStrip (db)

@malbouis, @cmsbuild, @saumyaphor4252, @ggovi, @francescobrivio, @tvami can you please review it and eventually sign? Thanks.
@mmusich, @VinInn this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

mmusich · 2022-09-29T10:40:32Z

please test

tvami

I just had some cosmetics comments.

CondTools/SiStrip/plugins/SiStripBadChannelPatcher.cc

cmsbuild · 2022-09-29T12:41:40Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39534/32312

This PR adds an extra 16KB to repository

cmsbuild · 2022-09-29T12:42:04Z

Pull request #39534 was updated. @malbouis, @cmsbuild, @saumyaphor4252, @ggovi, @francescobrivio, @tvami can you please check and sign again.

mmusich · 2022-09-29T12:43:04Z

@cmsbuild, please test

cmsbuild · 2022-09-29T17:09:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd3c7a/27847/summary.html
COMMIT: 9fafa7d
CMSSW: CMSSW_12_6_X_2022-09-29-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39534/27847/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-fd3c7a/41834.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3433154
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3433126
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 49 edm output root files, 49 DQM output files
TriggerResults: no differences found

mmusich · 2022-10-03T08:05:38Z

please test

cmsbuild · 2022-10-03T12:29:11Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd3c7a/27921/summary.html
COMMIT: bed6d37
CMSSW: CMSSW_12_6_X_2022-10-02-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39534/27921/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

136.7611136.7611_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM/step2_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM.log
136.7721136.7721_RunJetHT2016H_reminiaod+RunJetHT2016H_reminiaod+REMINIAOD_data2016+HARVESTDR2_REMINIAOD_data2016/step2_RunJetHT2016H_reminiaod+RunJetHT2016H_reminiaod+REMINIAOD_data2016+HARVESTDR2_REMINIAOD_data2016.log

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-fd3c7a/41834.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 5 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3432650
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3432622
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 49 edm output root files, 49 DQM output files
TriggerResults: no differences found

mmusich · 2022-10-03T12:33:33Z

@cmsbuild, please test

failure is unrelated. @cms-sw/db-l2 please consider signing. @perrotta please consider merging

cmsbuild · 2022-10-03T17:02:31Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd3c7a/27929/summary.html
COMMIT: bed6d37
CMSSW: CMSSW_12_6_X_2022-10-02-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39534/27929/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-fd3c7a/41834.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3432650
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3432625
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 49 edm output root files, 49 DQM output files
TriggerResults: no differences found

tvami · 2022-10-03T17:40:49Z

+db

cmsbuild · 2022-10-03T17:41:13Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

mmusich · 2022-10-04T05:59:21Z

@cmsbuild, please test

cmsbuild · 2022-10-04T10:31:16Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd3c7a/27971/summary.html
COMMIT: bed6d37
CMSSW: CMSSW_12_6_X_2022-10-03-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39534/27971/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-fd3c7a/41834.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3432650
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3432619
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 49 edm output root files, 49 DQM output files
TriggerResults: no differences found

mmusich · 2022-10-04T10:33:20Z

well, it's the third time in a row the relvals input are failing ... and this PR cannot possibly contribute to that.
Is there already an issue open somewhere?

perrotta · 2022-10-04T10:34:39Z

+1

RelValInput running out of time independently on this PR

perrotta · 2022-10-04T10:35:21Z

merge

cmsbuild added this to the CMSSW_12_6_X milestone Sep 29, 2022

cmsbuild added code-checks-pending db-pending orp-pending pending-signatures tests-pending labels Sep 29, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 29, 2022

cmsbuild added tests-started and removed tests-pending labels Sep 29, 2022

tvami reviewed Sep 29, 2022

View reviewed changes

CondTools/SiStrip/plugins/SiStripBadChannelPatcher.cc Outdated Show resolved Hide resolved

CondTools/SiStrip/plugins/SiStripBadChannelPatcher.cc Outdated Show resolved Hide resolved

mmusich force-pushed the SiStripBadChannelPatcher_v2 branch from 26cc910 to 9fafa7d Compare September 29, 2022 12:32

cmsbuild added code-checks-pending tests-pending and removed tests-started code-checks-approved labels Sep 29, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 29, 2022

cmsbuild added tests-started and removed tests-pending labels Sep 29, 2022

cmsbuild added tests-approved and removed tests-started labels Sep 29, 2022

mmusich force-pushed the SiStripBadChannelPatcher_v2 branch from 9fafa7d to 666fe0d Compare September 30, 2022 08:24

cmsbuild added tests-started and removed tests-pending labels Oct 3, 2022

cmsbuild added tests-rejected and removed tests-started labels Oct 3, 2022

cmsbuild added tests-started and removed tests-rejected labels Oct 3, 2022

cmsbuild added tests-rejected and removed tests-started labels Oct 3, 2022

cmsbuild added db-approved fully-signed and removed db-pending pending-signatures labels Oct 3, 2022

cmsbuild added tests-started and removed tests-rejected labels Oct 4, 2022

cmsbuild added tests-rejected and removed tests-started labels Oct 4, 2022

cmsbuild added orp-approved and removed orp-pending labels Oct 4, 2022

cmsbuild merged commit 1ae5ada into cms-sw:master Oct 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SiStrip Conditions tools: add `SiStripBadChannelPatcher` and related configurations #39534

SiStrip Conditions tools: add `SiStripBadChannelPatcher` and related configurations #39534

mmusich commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

mmusich commented Sep 29, 2022

tvami left a comment

cmsbuild commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

mmusich commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

mmusich commented Oct 3, 2022

cmsbuild commented Oct 3, 2022

mmusich commented Oct 3, 2022

cmsbuild commented Oct 3, 2022

tvami commented Oct 3, 2022

cmsbuild commented Oct 3, 2022

mmusich commented Oct 4, 2022

cmsbuild commented Oct 4, 2022

mmusich commented Oct 4, 2022

perrotta commented Oct 4, 2022

perrotta commented Oct 4, 2022

SiStrip Conditions tools: add SiStripBadChannelPatcher and related configurations #39534

SiStrip Conditions tools: add SiStripBadChannelPatcher and related configurations #39534

Conversation

mmusich commented Sep 29, 2022

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

mmusich commented Sep 29, 2022

tvami left a comment

Choose a reason for hiding this comment

cmsbuild commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

mmusich commented Sep 29, 2022

cmsbuild commented Sep 29, 2022

Comparison Summary

mmusich commented Oct 3, 2022

cmsbuild commented Oct 3, 2022

RelVals-INPUT

Comparison Summary

mmusich commented Oct 3, 2022

cmsbuild commented Oct 3, 2022

RelVals-INPUT

Comparison Summary

tvami commented Oct 3, 2022

cmsbuild commented Oct 3, 2022

mmusich commented Oct 4, 2022

cmsbuild commented Oct 4, 2022

RelVals-INPUT

Comparison Summary

mmusich commented Oct 4, 2022

perrotta commented Oct 4, 2022

perrotta commented Oct 4, 2022

SiStrip Conditions tools: add `SiStripBadChannelPatcher` and related configurations #39534

SiStrip Conditions tools: add `SiStripBadChannelPatcher` and related configurations #39534