cmsCodeRules fixes #37990

iarspider · 2022-05-17T18:58:05Z

PR description:

cppCommentSkipper: do not fail if line doesn't end with newline character
cppFunctionSkipper, cmsCodeRulesChecker: don't fail if non-UTF8 characters are present in source files

PR validation:

Local tests

If a line doesn't end with newline, the script may try to read past last character in line and fail

cmsbuild · 2022-05-17T19:05:23Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37990/30061

This PR adds an extra 16KB to repository

cmsbuild · 2022-05-17T19:05:48Z

A new Pull Request was created by @iarspider for master.

It involves the following packages:

Utilities/ReleaseScripts (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

makortel · 2022-05-17T21:24:24Z

@cmsbuild, please test

cmsbuild · 2022-05-17T23:06:20Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2489f7/24804/summary.html
COMMIT: f2aee86
CMSSW: CMSSW_12_5_X_2022-05-17-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2489f7/24804/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2489f7/24804/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test runtestRecoTauTagRecoTau had ERRORS

RelVals

136.7611

----- Begin Fatal Exception 18-May-2022 00:08:48 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=DeepTauId label='deepTau2018v2p5'
Exception Message:
edm::FileInPath unable to find file RecoTauTag/TrainingFiles/data/DeepTauId/deepTau_2018v2p5_core.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/external/slc7_amd64_gcc10/data:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/poison:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/src:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/external/slc7_amd64_gcc10/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.7611_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM
----- End Fatal Exception -------------------------------------------------

136.8311

----- Begin Fatal Exception 18-May-2022 00:08:48 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=DeepTauId label='deepTau2018v2p5'
Exception Message:
edm::FileInPath unable to find file RecoTauTag/TrainingFiles/data/DeepTauId/deepTau_2018v2p5_core.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/external/slc7_amd64_gcc10/data:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/poison:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/src:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/external/slc7_amd64_gcc10/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017
----- End Fatal Exception -------------------------------------------------

136.88811

----- Begin Fatal Exception 18-May-2022 00:08:50 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=DeepTauId label='deepTau2018v2p5'
Exception Message:
edm::FileInPath unable to find file RecoTauTag/TrainingFiles/data/DeepTauId/deepTau_2018v2p5_core.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24804/CMSSW_12_5_X_2022-05-17-1100/external/slc7_amd64_gcc10/data:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/poison:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/src:/cvmfs/cms-ib.cern.ch/nweek-02733/slc7_amd64_gcc10/cms/cmssw-patch/CMSSW_12_5_X_2022-05-17-1100/external/slc7_amd64_gcc10/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/matrix-results/136.88811_RunJetHT2018D_reminiaodUL+RunJetHT2018D_reminiaodUL+REMINIAOD_data2018UL+HARVEST2018_REMINIAOD_data2018UL
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

4.64.6_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1/step2_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1.log
136.72411136.72411_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM.log
136.72412136.72412_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM.log

Expand to see more relval errors ...

smuzaffar · 2022-05-18T06:18:19Z

please test

mbluj · 2022-05-18T08:28:41Z

Are required deepTau model files already part of cms-data? Corresponding PR is already merged (cms-data/RecoTauTag-TrainingFiles#8), but could it be that it is not propagated to IB, yet?

smuzaffar · 2022-05-18T08:31:42Z

yes @mbluj , cms-sw/cmsdist#7876 (which integrate cms-data/RecoTauTag-TrainingFiles#8 changes in to cmssw) has been merged for last night IB (12.5.X 23h00)

cmsbuild · 2022-05-18T10:42:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2489f7/24810/summary.html
COMMIT: f2aee86
CMSSW: CMSSW_12_5_X_2022-05-17-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37990/24810/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2489f7/11634.301_TTbar_14TeV+2021_Run3FS+TTbar_14TeV_TuneCP5_GenSim+HARVESTNano

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3672375
DQMHistoTests: Total failures: 97
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3672255
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-05-18T10:47:36Z

+core

cmsbuild · 2022-05-18T10:47:55Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-05-18T12:18:57Z

+1

iarspider · 2022-10-11T07:02:13Z

please test

…

cmsbuild · 2022-10-11T10:47:53Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2489f7/28169/summary.html
COMMIT: f2aee86
CMSSW: CMSSW_12_6_X_2022-10-10-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37990/28169/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2489f7/41834.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 5 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3392309
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3392281
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 49 edm output root files, 49 DQM output files
TriggerResults: no differences found

iarspider added 3 commits May 17, 2022 20:50

cppCommentSkipper: fix if newline is missing

86cdea1

If a line doesn't end with newline, the script may try to read past last character in line and fail

cppFunctionSkipper: deal with non-UTF8 characters

d131439

cmsCodeRulesChecker.py: protect from non-UTF8 characters

f2aee86

cmsbuild added this to the CMSSW_12_5_X milestone May 17, 2022

cmsbuild added code-checks-pending core-pending orp-pending pending-signatures tests-pending labels May 17, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels May 17, 2022

cmsbuild added tests-started and removed tests-pending labels May 17, 2022

cmsbuild added tests-rejected and removed tests-started labels May 17, 2022

cmsbuild added tests-started and removed tests-rejected labels May 18, 2022

cmsbuild added tests-approved and removed tests-started labels May 18, 2022

cmsbuild removed core-pending pending-signatures labels May 18, 2022

cmsbuild added core-approved fully-signed labels May 18, 2022

cmsbuild added orp-approved and removed orp-pending labels May 18, 2022

cmsbuild merged commit 4b1475e into cms-sw:master May 18, 2022

This was referenced May 19, 2022

Adding DeepMET SONIC Producer to CMSSW #37963

Merged

Adding SONIC ParticleNet Producer to CMSSW #37964

Merged

iarspider deleted the patch-6 branch May 20, 2022 08:52

cmsbuild added tests-started and removed tests-approved labels Oct 11, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmsCodeRules fixes #37990

cmsCodeRules fixes #37990

iarspider commented May 17, 2022

cmsbuild commented May 17, 2022

cmsbuild commented May 17, 2022

makortel commented May 17, 2022

cmsbuild commented May 17, 2022

smuzaffar commented May 18, 2022

mbluj commented May 18, 2022

smuzaffar commented May 18, 2022 •

edited

cmsbuild commented May 18, 2022

smuzaffar commented May 18, 2022

cmsbuild commented May 18, 2022

perrotta commented May 18, 2022

iarspider commented Oct 11, 2022 via email

cmsbuild commented Oct 11, 2022

cmsCodeRules fixes #37990

cmsCodeRules fixes #37990

Conversation

iarspider commented May 17, 2022

PR description:

PR validation:

cmsbuild commented May 17, 2022

cmsbuild commented May 17, 2022

makortel commented May 17, 2022

cmsbuild commented May 17, 2022

Unit Tests

RelVals

RelVals-INPUT

smuzaffar commented May 18, 2022

mbluj commented May 18, 2022

smuzaffar commented May 18, 2022 • edited

cmsbuild commented May 18, 2022

Comparison Summary

smuzaffar commented May 18, 2022

cmsbuild commented May 18, 2022

perrotta commented May 18, 2022

iarspider commented Oct 11, 2022 via email

cmsbuild commented Oct 11, 2022

Comparison Summary

smuzaffar commented May 18, 2022 •

edited