Pr101x TTStubCluster Association in FEVTDEBUG eventcontent #22415

rekovic · 2018-03-02T09:41:15Z

Adding TTStub && TTCluster Association products to SimTrackerFEVTDEBUG EventContent:

'keep *_TTClusterAssociatorFromPixelDigis_*_*',
'keep *_TTStubAssociatorFromPixelDigis_*_*'

…FEVTDEBUG EventContent.

cmsbuild · 2018-03-02T09:41:35Z

The code-checks are being triggered in jenkins.

rekovic · 2018-03-02T09:42:08Z

please test

cmsbuild · 2018-03-02T09:43:00Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-22415/3659

cmsbuild · 2018-03-02T09:43:18Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/26430/console Started: 2018/03/02 10:43

cmsbuild · 2018-03-02T09:43:20Z

A new Pull Request was created by @rekovic for master.

It involves the following packages:

SimTracker/Configuration

@cmsbuild, @civanch, @mdhildreth can you please review it and eventually sign? Thanks.
@jlagram, @makortel, @sdevissc, @GiacomoSguazzoni, @gbenelli, @rovere, @VinInn, @cnuttens, @prolay, @ebrondol, @mmusich, @threus, @dgulhan, @LBeck this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2018-03-02T10:42:38Z

-1

Tested at: 930ceba

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-22415/26430/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following worklfows:
21234.0 step3

runTheMatrix-results/21234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D21_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D21+RecoFullGlobal_2023D21+HARVESTFullGlobal_2023D21/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D21_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D21+RecoFullGlobal_2023D21+HARVESTFullGlobal_2023D21.log

20434.0 step3

runTheMatrix-results/20434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D19_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D19+RecoFullGlobal_2023D19+HARVESTFullGlobal_2023D19/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D19_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D19+RecoFullGlobal_2023D19+HARVESTFullGlobal_2023D19.log

20034.0 step3

runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17.log

cmsbuild · 2018-03-02T10:42:40Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

rekovic · 2018-03-02T11:48:06Z

hmmm.... the three Phase-II workflows fail (200340, 20434 and 21234). Example log

Not immediately obvious how adding to EventContent does this. Investigating.

rekovic · 2018-03-06T12:42:13Z

Coming back to this problem.

The crash happens in a non-related code, in RecoTauTag/RecoTau in this while loop,
more precisely in the call to a set(reco::Candidate&) in an instance of AddFourMomenta class here .

Useful excerpt from valgrind report

==23928== Invalid read of size 8
==23928==    at 0x39E2AD60: AddFourMomenta::set(reco::Candidate&) const (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libPhysicsToolsCandUtils.so)
==23928==    by 0x4FADC726: reco::tau::RecoTauPiZeroStripPlugin2::operator()(reco::PFJet const&) const (in /afs/cern.ch/work/r/rekovic/private/relaeses/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/pluginRecoTauTagRecoTauPlugins.so)
==23928==    by 0x4FAA11D3: RecoTauPiZeroProducer::produce(edm::Event&, edm::EventSetup const&) (in /afs/cern.ch/work/r/rekovic/private/relaeses/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/pluginRecoTauTagRecoTauPlugins.so)
==23928==    by 0x4C88862: edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so)
==23928==    by 0x4C4A1A1: edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so)
==23928==    by 0x4B64E49: decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so)
==23928==    by 0x4B64FEA: bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so)
==23928==    by 0x4B6655A: std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so)
==23928==    by 0x4B67824: edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so)
==23928==    by 0x5FE342B: tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) (custom_scheduler.h:509)
==23928==    by 0x4C18D77: edm::EventProcessor::processLumis(std::shared_ptr<void> const&) (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so)
==23928==    by 0x4C1E01E: edm::EventProcessor::runToCompletion() (in /cvmfs/cms-ib.cern.ch/nweek-02513/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-02-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so)
==23928==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==23928==

attn @davidlange6 @fabiocos

rekovic · 2018-03-06T13:25:43Z

Recipe to quickly reproduce the problem:

cmsrel CMSSW_10_1_X_2018-03-05-1100
cd CMSSW_10_1_X_2018-03-05-1100/src
cmsenv
git cms-init
git cms-merge-topic -u 22415
scramv1 b -j 4
runTheMatrix.py -l 20034.0

Also atn @perrotta @slava77

rekovic · 2018-03-12T14:32:43Z

please test with #22584

cmsbuild · 2018-03-12T14:33:00Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/26792/console Started: 2018/03/12 15:33

cmsbuild · 2018-03-12T16:50:03Z

+1
Tested at: 930ceba
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-22415/26792/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
281b6bb
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-22415/26792/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-22415/26792/git-merge-result

cmsbuild · 2018-03-12T16:50:08Z

Comparison job queued.

rekovic · 2018-03-12T18:03:58Z

ok. the #22584 of @perrotta fixes the problems this PR had.

cmsbuild · 2018-03-12T18:33:47Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-22415/26792/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 29
DQMHistoTests: Total histograms compared: 2477919
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2477742
DQMHistoTests: Total skipped: 176
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1.1700000001 KiB( 23 files compared)
Checked 118 log files, 9 edm output root files, 29 DQM output files

civanch · 2018-03-13T14:59:35Z

+1

cmsbuild · 2018-03-13T14:59:51Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

rekovic · 2018-03-14T14:50:13Z

@fabiocos
Can you please merge this simple PR in 10_1_X, now that #22584 has been merged and running.

We need this PR in 10_1_X and have it run for a day or two, before we make 9_3_6 with its back-port
(i.e. #22584)

fabiocos · 2018-03-14T17:32:32Z

+1

fabiocos · 2018-03-14T17:33:36Z

@kpedro88 this is upgrade specific, and needed for the next 9_3_X build

Add TTCluster & TTStubAssociatorFromPixelDigis products to SimTracker…

930ceba

…FEVTDEBUG EventContent.

cmsbuild added this to the CMSSW_10_1_X milestone Mar 2, 2018

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures simulation-pending tests-pending labels Mar 2, 2018

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 2, 2018

cmsbuild added tests-started and removed tests-pending labels Mar 2, 2018

rekovic mentioned this pull request Mar 2, 2018

Enbale TrackTriggerAssociatorClustersStubs in L1TrackTrigger sequence. #22190

Merged

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Mar 2, 2018

This was referenced Mar 6, 2018

Segmentation violation in RecoTauTag/RecoTau in phase-2 workflows #22483

Closed

Next 9_3_X release for L1T Phase2 #22554

Closed

cmsbuild added comparison-pending tests-pending and removed comparison-notrun tests-rejected labels Mar 12, 2018

cmsbuild added tests-started and removed tests-pending labels Mar 12, 2018

cmsbuild added tests-approved and removed tests-started labels Mar 12, 2018

cmsbuild added comparison-available and removed comparison-pending labels Mar 12, 2018

rekovic mentioned this pull request Mar 13, 2018

Pr93x TTStubCluster Association in FEVTDEBUG eventcontent #22591

Merged

cmsbuild added fully-signed simulation-approved and removed pending-signatures simulation-pending labels Mar 13, 2018

cmsbuild added orp-approved and removed orp-pending labels Mar 14, 2018

cmsbuild merged commit 127b596 into cms-sw:master Mar 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pr101x TTStubCluster Association in FEVTDEBUG eventcontent #22415

Pr101x TTStubCluster Association in FEVTDEBUG eventcontent #22415

rekovic commented Mar 2, 2018 •

edited

Loading

cmsbuild commented Mar 2, 2018

rekovic commented Mar 2, 2018

cmsbuild commented Mar 2, 2018

cmsbuild commented Mar 2, 2018 •

edited

Loading

cmsbuild commented Mar 2, 2018

cmsbuild commented Mar 2, 2018

cmsbuild commented Mar 2, 2018

rekovic commented Mar 2, 2018

rekovic commented Mar 6, 2018

rekovic commented Mar 6, 2018

rekovic commented Mar 12, 2018

cmsbuild commented Mar 12, 2018 •

edited

Loading

cmsbuild commented Mar 12, 2018

cmsbuild commented Mar 12, 2018

rekovic commented Mar 12, 2018

cmsbuild commented Mar 12, 2018

civanch commented Mar 13, 2018

cmsbuild commented Mar 13, 2018

rekovic commented Mar 14, 2018

fabiocos commented Mar 14, 2018

fabiocos commented Mar 14, 2018

Pr101x TTStubCluster Association in FEVTDEBUG eventcontent #22415

Pr101x TTStubCluster Association in FEVTDEBUG eventcontent #22415

Conversation

rekovic commented Mar 2, 2018 • edited Loading

cmsbuild commented Mar 2, 2018

rekovic commented Mar 2, 2018

cmsbuild commented Mar 2, 2018

cmsbuild commented Mar 2, 2018 • edited Loading

cmsbuild commented Mar 2, 2018

cmsbuild commented Mar 2, 2018

cmsbuild commented Mar 2, 2018

rekovic commented Mar 2, 2018

rekovic commented Mar 6, 2018

rekovic commented Mar 6, 2018

rekovic commented Mar 12, 2018

cmsbuild commented Mar 12, 2018 • edited Loading

cmsbuild commented Mar 12, 2018

cmsbuild commented Mar 12, 2018

rekovic commented Mar 12, 2018

cmsbuild commented Mar 12, 2018

civanch commented Mar 13, 2018

cmsbuild commented Mar 13, 2018

rekovic commented Mar 14, 2018

fabiocos commented Mar 14, 2018

fabiocos commented Mar 14, 2018

rekovic commented Mar 2, 2018 •

edited

Loading

cmsbuild commented Mar 2, 2018 •

edited

Loading

cmsbuild commented Mar 12, 2018 •

edited

Loading