Add tests for Skeleton templates #37125

makortel · 2022-03-03T01:10:02Z

PR description:

This PR addresses partially #29850 by adding tests that include code-checks, code-format, and compilation (for selected subsets of the skeletons), following the suggestion in #29850 (comment)

PR validation:

Unit tests run.

cmsbuild · 2022-03-03T01:16:13Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37125/28650

This PR adds an extra 28KB to repository

cmsbuild · 2022-03-03T01:16:36Z

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

FWCore/Skeletons (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

makortel · 2022-03-03T01:20:37Z

@cmsbuild, please test

makortel · 2022-03-03T01:20:46Z

@smuzaffar, could you review this one?

cmsbuild · 2022-03-03T07:29:30Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-906b1b/22792/summary.html
COMMIT: 5e648c8
CMSSW: CMSSW_12_3_X_2022-03-02-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37125/22792/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3990927
DQMHistoTests: Total failures: 13
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3990891
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

FWCore/Skeletons/test/runtests.sh

smuzaffar · 2022-03-03T07:53:05Z

please test for slc7_aarch64_gcc11

lets tests this for non-x86_64 archs

smuzaffar · 2022-03-03T07:53:32Z

please test for slc7_ppc64le_gcc11

cmsbuild · 2022-03-03T10:08:10Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-906b1b/22795/summary.html
COMMIT: 5e648c8
CMSSW: CMSSW_12_3_X_2022-03-01-2300/slc7_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37125/22795/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-906b1b/22795/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-906b1b/22795/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test DRNTest had ERRORS
---> test materialBudgetTrackerPlots had ERRORS

cmsbuild · 2022-03-03T10:32:53Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-906b1b/22796/summary.html
COMMIT: 5e648c8
CMSSW: CMSSW_12_3_X_2022-03-01-2300/slc7_aarch64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/37125/22796/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-906b1b/22796/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-906b1b/22796/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test DRNTest had ERRORS

RelVals

4.53

----- Begin Fatal Exception 03-Mar-2022 10:34:02 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

140.56

----- Begin Fatal Exception 03-Mar-2022 10:43:28 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 03-Mar-2022 10:57:41 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

cmsbuild · 2022-03-10T16:21:05Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37125/28788

This PR adds an extra 12KB to repository

cmsbuild · 2022-03-10T16:21:32Z

Pull request #37125 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please check and sign again.

makortel · 2022-03-10T16:29:08Z

@cmsbuild, please test

cmsbuild · 2022-03-10T20:08:59Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-906b1b/23023/summary.html
COMMIT: a9f475b
CMSSW: CMSSW_12_3_X_2022-03-10-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37125/23023/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

The workflows 1001.0, 1000.0, 136.88811, 136.874, 136.8311, 136.793, 136.7611, 136.731, 4.22 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695161
DQMHistoTests: Total failures: 13
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3695125
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

makortel · 2022-03-10T20:58:01Z

@smuzaffar Anything else?

smuzaffar · 2022-03-10T21:26:25Z

No @makortel all look good now

makortel · 2022-03-10T21:34:42Z

+1

cmsbuild · 2022-03-10T21:35:06Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-03-12T01:35:26Z

+1

makortel added 2 commits March 2, 2022 22:28

Make skeletons (closer) to conform style

377038e

Add option to include a line only if no example argument is given

0a09698

cmsbuild added this to the CMSSW_12_3_X milestone Mar 3, 2022

cmsbuild added code-checks-pending core-pending orp-pending pending-signatures tests-pending labels Mar 3, 2022

makortel mentioned this pull request Mar 3, 2022

Improve Skeleton templates and scripts #29850

Open

2 tasks

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 3, 2022

cmsbuild added tests-started and removed tests-pending labels Mar 3, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 3, 2022

smuzaffar reviewed Mar 3, 2022

View reviewed changes

FWCore/Skeletons/test/runtests.sh Show resolved Hide resolved

makortel force-pushed the mkSkeletonTests branch from 5e648c8 to 4e1c69b Compare March 3, 2022 14:58

cmsbuild added code-checks-pending tests-pending and removed tests-approved code-checks-approved labels Mar 3, 2022

cmsbuild added code-checks-pending tests-pending and removed tests-approved code-checks-approved labels Mar 10, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 10, 2022

cmsbuild added tests-started and removed tests-pending labels Mar 10, 2022

cmsbuild added tests-approved and removed tests-started labels Mar 10, 2022

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Mar 10, 2022

smuzaffar modified the milestones: CMSSW_12_3_X, CMSSW_12_4_X Mar 11, 2022

cmsbuild added orp-approved and removed orp-pending labels Mar 12, 2022

cmsbuild merged commit 8838c53 into cms-sw:master Mar 12, 2022

makortel deleted the mkSkeletonTests branch March 14, 2022 13:01

makortel mentioned this pull request Mar 14, 2022

Add tests for Skeleton templates cms-sw/framework-team#373

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tests for Skeleton templates #37125

Add tests for Skeleton templates #37125

makortel commented Mar 3, 2022

cmsbuild commented Mar 3, 2022

cmsbuild commented Mar 3, 2022

makortel commented Mar 3, 2022

makortel commented Mar 3, 2022

cmsbuild commented Mar 3, 2022

smuzaffar commented Mar 3, 2022

smuzaffar commented Mar 3, 2022

cmsbuild commented Mar 3, 2022

cmsbuild commented Mar 3, 2022

cmsbuild commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

makortel commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

makortel commented Mar 10, 2022

smuzaffar commented Mar 10, 2022

makortel commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

qliphy commented Mar 12, 2022

Add tests for Skeleton templates #37125

Add tests for Skeleton templates #37125

Conversation

makortel commented Mar 3, 2022

PR description:

PR validation:

cmsbuild commented Mar 3, 2022

cmsbuild commented Mar 3, 2022

makortel commented Mar 3, 2022

makortel commented Mar 3, 2022

cmsbuild commented Mar 3, 2022

Comparison Summary

smuzaffar commented Mar 3, 2022

smuzaffar commented Mar 3, 2022

cmsbuild commented Mar 3, 2022

Unit Tests

cmsbuild commented Mar 3, 2022

Unit Tests

RelVals

cmsbuild commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

makortel commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

Comparison Summary

makortel commented Mar 10, 2022

smuzaffar commented Mar 10, 2022

makortel commented Mar 10, 2022

cmsbuild commented Mar 10, 2022

qliphy commented Mar 12, 2022