Update JSON files in line with changes done in cms-sw/cmssw#39866

[osschar]

Update JSON files in line with changes introduced in cms-sw/cmssw#39866

In particular:

• introduce standard function names so they can be set-up through JSON or CMSSW config;
• consolidate seed-cleaning and duplicate-cleaning parameters names and move them into the top-level IterationConfig struct (moved out of IterationParams, a sub-structure that exists in two instances, for forward search and for backward fit / search).

[cmsbuild]

A new Pull Request was created by @osschar (Matevž Tadel) for branch main.

@smuzaffar, @aandvalenzuela, @iarspider, @clacaputo, @cmsbuild, @mandrenguyen can you please review it and eventually sign? Thanks.
@VourMa, @makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @missirol, @gpetruc, @mmusich, @mtosi, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

Pull request #10 was updated.

[mandrenguyen]

+reconstruction
tested in #39866

[cmsbuild]

-1

Failed Tests: RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e2f397/28953/summary.html
COMMIT: ce12a6f
CMSSW: CMSSW_12_6_X_2022-11-09-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoTracker-MkFit/10/28953/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @dan131riley build cpu_features so that it can be linked into plugins cms-sw/cmsdist#8176

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e2f397/28953/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e2f397/28953/git-merge-result

RelVals

• 138.4

----- Begin Fatal Exception 10-Nov-2022 16:27:02 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 346512 lumi: 250 event: 243042266 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module DQMMessageLogger/'DQMMessageLogger'
   [3] Prefetching for module LogErrorHarvester/'logErrorHarvester'
   [4] Prefetching for module HcalNoiseInfoProducer/'hcalnoise'
   [5] Prefetching for module FastjetJetProducer/'ak4PFJets'
   [6] Prefetching for module PFLinker/'particleFlow'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module GsfTrackProducer/'electronGsfTracks'
   [11] Prefetching for module CkfTrackCandidateMaker/'electronCkfTrackCandidates'
   [12] Prefetching for module MeasurementTrackerEventProducer/'MeasurementTrackerEvent'
   [13] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [14] Prefetching for module PrimaryVertexProducer/'firstStepPrimaryVerticesPreSplitting'
   [15] Prefetching for module TrackProducer/'initialStepTracksPreSplitting'
   [16] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidatesPreSplitting'
   [17] Prefetching for module MkFitProducer/'initialStepTrackCandidatesMkFitPreSplitting'
   [18] Calling method for EventSetup module MkFitIterationConfigESProducer/'initialStepTrackCandidatesMkFitConfigPreSplitting'
Exception Message:
A std::exception was thrown.
[json.exception.out_of_range.403] key 'm_requires_quality_filter' not found
----- End Fatal Exception -------------------------------------------------

• 138.5

----- Begin Fatal Exception 10-Nov-2022 16:27:19 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 346512 lumi: 250 event: 243042266 stream: 0
   [1] Running path 'dqmoffline_13_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module MuonIdProducer/'muons1stStep'
   [5] Prefetching for module DuplicateListMerger/'generalTracks'
   [6] Prefetching for module TrackProducer/'mergedDuplicateTracks'
   [7] Prefetching for module DuplicateTrackMerger/'duplicateTrackCandidates'
   [8] Prefetching for module TrackCollectionMerger/'preDuplicateMergingGeneralTracks'
   [9] Prefetching for module TrackCollectionMerger/'earlyGeneralTracks'
   [10] Prefetching for module TrackProducer/'initialStepTracks'
   [11] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidates'
   [12] Prefetching for module MkFitEventOfHitsProducer/'mkFitEventOfHits'
   [13] Prefetching for module MkFitSiPixelHitConverter/'mkFitSiPixelHits'
   [14] Prefetching for module SiPixelRecHitConverter/'siPixelRecHits'
   [15] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [16] Prefetching for module PrimaryVertexProducer/'firstStepPrimaryVerticesPreSplitting'
   [17] Prefetching for module TrackProducer/'initialStepTracksPreSplitting'
   [18] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidatesPreSplitting'
   [19] Prefetching for module MkFitProducer/'initialStepTrackCandidatesMkFitPreSplitting'
   [20] Calling method for EventSetup module MkFitIterationConfigESProducer/'initialStepTrackCandidatesMkFitConfigPreSplitting'
Exception Message:
A std::exception was thrown.
[json.exception.out_of_range.403] key 'm_requires_quality_filter' not found
----- End Fatal Exception -------------------------------------------------

• 136.793

----- Begin Fatal Exception 10-Nov-2022 16:29:48 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 301998 lumi: 9 event: 8792128 stream: 0
   [1] Running path 'dqmoffline_9_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module MuonIdProducer/'muons1stStep'
   [5] Prefetching for module HBHEIsolatedNoiseReflagger/'hbhereco@cpu'
   [6] Prefetching for module TrackExtrapolator/'trackExtrapolator'
   [7] Prefetching for module DuplicateListMerger/'generalTracks'
   [8] Prefetching for module TrackProducer/'mergedDuplicateTracks'
   [9] Prefetching for module DuplicateTrackMerger/'duplicateTrackCandidates'
   [10] Prefetching for module TrackCollectionMerger/'preDuplicateMergingGeneralTracks'
   [11] Prefetching for module TrackCollectionMerger/'earlyGeneralTracks'
   [12] Prefetching for module TrackProducer/'initialStepTracks'
   [13] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidates'
   [14] Prefetching for module MkFitEventOfHitsProducer/'mkFitEventOfHits'
   [15] Prefetching for module MkFitSiPixelHitConverter/'mkFitSiPixelHits'
   [16] Prefetching for module SiPixelRecHitConverter/'siPixelRecHits'
   [17] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [18] Prefetching for module PrimaryVertexProducer/'firstStepPrimaryVerticesPreSplitting'
   [19] Prefetching for module TrackProducer/'initialStepTracksPreSplitting'
   [20] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidatesPreSplitting'
   [21] Prefetching for module MkFitProducer/'initialStepTrackCandidatesMkFitPreSplitting'
   [22] Calling method for EventSetup module MkFitIterationConfigESProducer/'initialStepTrackCandidatesMkFitConfigPreSplitting'
Exception Message:
A std::exception was thrown.
[json.exception.out_of_range.403] key 'm_requires_quality_filter' not found
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

• 136.874
• 140.56
• 139.001
• 312.0
• 10042.0
• 11634.7
• 10024.0
• 10824.0
• 11634.0
• 11634.914
• 10224.0
• 12434.0
• 11634.911
• 11834.0
• 250202.181

AddOn Tests

• hlt_mc_GRun

----- Begin Fatal Exception 10-Nov-2022 16:32:17 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 2
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Prefetching for module ReducedRecHitCollectionProducer/'reducedEcalRecHitsEB'
   [4] Prefetching for module InterestingDetIdCollectionProducer/'interestingEcalDetIdPFEB'
   [5] Prefetching for module PFECALSuperClusterProducer/'particleFlowSuperClusterECAL'
   [6] Prefetching for module RecoChargedRefCandidatePrimaryVertexSorter/'offlinePrimaryVertices'
   [7] Prefetching for module ChargedRefCandidateProducer/'trackRefsForJetsBeforeSorting'
   [8] Prefetching for module TrackWithVertexRefSelector/'trackWithVertexRefSelectorBeforeSorting'
   [9] Prefetching for module DuplicateListMerger/'generalTracks'
   [10] Prefetching for module TrackProducer/'mergedDuplicateTracks'
   [11] Prefetching for module DuplicateTrackMerger/'duplicateTrackCandidates'
   [12] Prefetching for module TrackCollectionMerger/'preDuplicateMergingGeneralTracks'
   [13] Prefetching for module TrackCollectionMerger/'earlyGeneralTracks'
   [14] Prefetching for module TrackProducer/'initialStepTracks'
   [15] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidates'
   [16] Prefetching for module MkFitEventOfHitsProducer/'mkFitEventOfHits'
   [17] Prefetching for module MkFitSiPixelHitConverter/'mkFitSiPixelHits'
   [18] Prefetching for module SiPixelRecHitConverter/'siPixelRecHits'
   [19] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [20] Prefetching for module PrimaryVertexProducer/'firstStepPrimaryVerticesPreSplitting'
   [21] Prefetching for module TrackProducer/'initialStepTracksPreSplitting'
   [22] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidatesPreSplitting'
   [23] Prefetching for module MkFitProducer/'initialStepTrackCandidatesMkFitPreSplitting'
   [24] Calling method for EventSetup module MkFitIterationConfigESProducer/'initialStepTrackCandidatesMkFitConfigPreSplitting'
Exception Message:
A std::exception was thrown.
[json.exception.out_of_range.403] key 'm_requires_quality_filter' not found
----- End Fatal Exception -------------------------------------------------

• hlt_mc_HIon

----- Begin Fatal Exception 10-Nov-2022 16:30:33 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 2 stream: 3
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Prefetching for module ReducedRecHitCollectionProducer/'reducedEcalRecHitsEB'
   [4] Prefetching for module InterestingDetIdCollectionProducer/'interestingEcalDetIdPFEB'
   [5] Prefetching for module PFECALSuperClusterProducer/'particleFlowSuperClusterECAL'
   [6] Prefetching for module RecoChargedRefCandidatePrimaryVertexSorter/'offlinePrimaryVertices'
   [7] Prefetching for module ChargedRefCandidateProducer/'trackRefsForJetsBeforeSorting'
   [8] Prefetching for module TrackWithVertexRefSelector/'trackWithVertexRefSelectorBeforeSorting'
   [9] Prefetching for module DuplicateListMerger/'generalTracks'
   [10] Prefetching for module TrackProducer/'mergedDuplicateTracks'
   [11] Prefetching for module DuplicateTrackMerger/'duplicateTrackCandidates'
   [12] Prefetching for module TrackCollectionMerger/'preDuplicateMergingGeneralTracks'
   [13] Prefetching for module TrackCollectionMerger/'earlyGeneralTracks'
   [14] Prefetching for module TrackProducer/'initialStepTracks'
   [15] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidates'
   [16] Prefetching for module MkFitEventOfHitsProducer/'mkFitEventOfHits'
   [17] Prefetching for module MkFitSiPixelHitConverter/'mkFitSiPixelHits'
   [18] Prefetching for module SiPixelRecHitConverter/'siPixelRecHits'
   [19] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [20] Prefetching for module PrimaryVertexProducer/'firstStepPrimaryVerticesPreSplitting'
   [21] Prefetching for module TrackProducer/'initialStepTracksPreSplitting'
   [22] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidatesPreSplitting'
   [23] Prefetching for module MkFitProducer/'initialStepTrackCandidatesMkFitPreSplitting'
   [24] Calling method for EventSetup module MkFitIterationConfigESProducer/'initialStepTrackCandidatesMkFitConfigPreSplitting'
Exception Message:
A std::exception was thrown.
[json.exception.out_of_range.403] key 'm_requires_quality_filter' not found
----- End Fatal Exception -------------------------------------------------

• hlt_mc_PIon

----- Begin Fatal Exception 10-Nov-2022 16:29:35 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 2 stream: 1
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Prefetching for module ReducedRecHitCollectionProducer/'reducedEcalRecHitsEB'
   [4] Prefetching for module InterestingDetIdCollectionProducer/'interestingEcalDetIdPFEB'
   [5] Prefetching for module PFECALSuperClusterProducer/'particleFlowSuperClusterECAL'
   [6] Prefetching for module RecoChargedRefCandidatePrimaryVertexSorter/'offlinePrimaryVertices'
   [7] Prefetching for module ChargedRefCandidateProducer/'trackRefsForJetsBeforeSorting'
   [8] Prefetching for module TrackWithVertexRefSelector/'trackWithVertexRefSelectorBeforeSorting'
   [9] Prefetching for module DuplicateListMerger/'generalTracks'
   [10] Prefetching for module TrackProducer/'mergedDuplicateTracks'
   [11] Prefetching for module DuplicateTrackMerger/'duplicateTrackCandidates'
   [12] Prefetching for module TrackCollectionMerger/'preDuplicateMergingGeneralTracks'
   [13] Prefetching for module TrackCollectionMerger/'earlyGeneralTracks'
   [14] Prefetching for module TrackProducer/'initialStepTracks'
   [15] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidates'
   [16] Prefetching for module MkFitEventOfHitsProducer/'mkFitEventOfHits'
   [17] Prefetching for module MkFitSiPixelHitConverter/'mkFitSiPixelHits'
   [18] Prefetching for module SiPixelRecHitConverter/'siPixelRecHits'
   [19] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [20] Prefetching for module PrimaryVertexProducer/'firstStepPrimaryVerticesPreSplitting'
   [21] Prefetching for module TrackProducer/'initialStepTracksPreSplitting'
   [22] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidatesPreSplitting'
   [23] Prefetching for module MkFitProducer/'initialStepTrackCandidatesMkFitPreSplitting'
   [24] Calling method for EventSetup module MkFitIterationConfigESProducer/'initialStepTrackCandidatesMkFitConfigPreSplitting'
Exception Message:
A std::exception was thrown.
[json.exception.out_of_range.403] key 'm_requires_quality_filter' not found
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

• hlt_mc_PRef
• hlt_data_GRun
• hlt_data_HIon
• hlt_data_PIon
• hlt_data_PRef

[slava77]

-1

Failed Tests:

@smuzaffar
is there a way for auto-submitted tests to figure out that they were already done in connection with another PR?

It seems like we are regularly getting failures on tests of externals that require a cms-sw/cmssw PR. The coupled tests done in cms-sw/cmssw PR plus the external pass OK, but the post-signoff auto-tests make an impression of broken PR (and also burn some unnecessary CPU).

[smuzaffar]

-1
Failed Tests:

@smuzaffar is there a way for auto-submitted tests to figure out that they were already done in connection with another PR?

It seems like we are regularly getting failures on tests of externals that require a cms-sw/cmssw PR. The coupled tests done in cms-sw/cmssw PR plus the external pass OK, but the post-signoff auto-tests make an impression of broken PR (and also burn some unnecessary CPU).

@slava77 , no currently there is no way to do that. We can disable auto-start of tests for data PRs on signature ( means for data PRs requires an explicit please test )

[slava77]

@slava77 , no currently there is no way to do that. We can disable auto-start of tests for data PRs on signature ( means for data PRs requires an explicit please test )

that could be an improvement
will it be a part of test parameters or a hardcoded default?

[smuzaffar]

@slava77 , no currently there is no way to do that. We can disable auto-start of tests for data PRs on signature ( means for data PRs requires an explicit please test )

that could be an improvement will it be a part of test parameters or a hardcoded default?

hard-coded default applied only to cms-data repositories

[mmusich]

I guess this PR needs to be merged now after cms-sw/cmssw#39866 is merged.
There are massive relval failures in the last IB

[perrotta]

+1

• It must go (and it was tested together) [MkFit] Phase2 geometry & String configurable standard functions cms-sw/cmssw#39866

[perrotta]

merge

[smuzaffar]

@slava77 , no currently there is no way to do that. We can disable auto-start of tests for data PRs on signature ( means for data PRs requires an explicit please test )

that could be an improvement will it be a part of test parameters or a hardcoded default?

@slava77 , bot now does not start tests on signatures for non-cmssw repositories. One need to explicitly request the start of tests by please test comment. Bot behavior is not changed for cmssw repo and it will trigger the tests on first signature

[slava77]

@slava77 , bot now does not start tests on signatures for non-cmssw repositories. One need to explicitly request the start of tests by please test comment. Bot behavior is not changed for cmssw repo and it will trigger the tests on first signature

Thank you.
Noted.

Taking this PR as an example, it may also help to figure out how to synchronously merge.
E.g. (less automation) if a PR is declared as a dependency the last "+1" from the ORP wouldn't yet merge and there would be a message reminding of the dependency.