Trajectory cleanup #3010

VinInn · 2014-03-25T10:55:51Z

Pure technical changes to cleanup Trajectory and TrackCandidate memory churn.
Still a lot can be done (for instance in Conversion code).
More will be done once fitters will be migrated to use standard TRH instead of TTRH.

In any case it is a sort of lost battle as classes whose objects are stored in vector in the event cannot be made non-copiable due to Reflex. Will try again with Root6.

No regression expected.
No regression observed.

…metric Clusters)

…getByToken

… InOut conversions still problem

…1f-simdev Sim -- Simulation of BCM1F

cmsbuild · 2014-03-25T15:22:41Z

-1
When I ran the RelVals I found an error in the following worklfows:
4.22 step2

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD.log

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log

4.53 step3

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT/step3_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT.log

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3010/643/summary.html

VinInn · 2014-03-25T15:31:36Z

ok, strange but possible given the trace back (looks like a double delete)
I run myself limited matrix and succeeded

4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED - time date Tue Mar 25 10:05:34 2014-date Tue Mar 25 10:03:11 2014; exit: 0 0 0 0 0
4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED - time date Tue Mar 25 10:14:22 2014-date Tue Mar 25 10:03:15 2014; exit: 0 0 0 0
5.1_TTbar+TTbarFS+HARVESTFS Step0-PASSED Step1-PASSED - time date Tue Mar 25 10:05:09 2014-date Tue Mar 25 10:03:17 2014; exit: 0 0
8.0_BeamHalo+BeamHaloINPUT+DIGICOS+RECOCOS+ALCABH+HARVESTCOS Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED - time date Tue Mar 25 10:06:55 2014-date Tue Mar 25 10:03:20 2014; exit: 0 0 0 0 0
25.0_TTbar+TTbarINPUT+DIGI+RECO+HARVEST+ALCATT Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED - time date Tue Mar 25 10:15:22 2014-date Tue Mar 25 10:05:14 2014; exit: 0 0 0 0 0
1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED - time date Tue Mar 25 10:10:43 2014-date Tue Mar 25 10:05:35 2014; exit: 0 0 0 0 0
1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED - time date Tue Mar 25 10:10:12 2014-date Tue Mar 25 10:06:59 2014; exit: 0 0 0 0
7 7 6 6 4 tests passed, 0 0 0 0 0 failed

will check again,
checkdeps -a again
and see what's wrong

VinInn · 2014-03-25T15:36:46Z

not sure what I could ever done to crash in PF though
#17 0x00007fa35bd55f9d in KDTreeLinkerTrackEcal::buildTree() () from /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_1_X-slc6_amd64_gcc481/CMSSW_7_1_X_2014-03-25-0200/lib/slc6_amd64_gcc481/libRecoParticleFlowPFProducer.so
#18 0x00007fa35bd537da in KDTreeLinkerBase::process() () from /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_1_X-slc6_amd64_gcc481/CMSSW_7_1_X_2014-03-25-0200/lib/slc6_amd64_gcc481/libRecoParticleFlowPFProducer.so
#19 0x00007fa35bd7b60c in PFBlockAlgo::findBlocks() () from /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_1_X-slc6_amd64_gcc481/CMSSW_7_1_X_2014-03-25-0200/lib/slc6_amd64_gcc481/libRecoParticleFlowPFProducer.so
#20 0x00007fa35be53666 in PFBlockProducer::produce(edm::Event&, edm::EventSetup const&) () from /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_1_X-slc6_amd64_gcc481/CMSSW_7_1_X_2014-03-25-0200/lib/slc6_amd64_gcc481/pluginRecoParticleFlowPFProducerPlugins.so

VinInn · 2014-03-25T17:42:18Z

Sorry, I cannot reproduce the segfaults in my development area based on
/afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc490/cms/cmssw/CMSSW_7_1_X_2014-03-21-0200/

VinInn · 2014-03-25T19:48:56Z

did
cmsrel CMSSW_7_1_X_2014-03-25-1400
cd CMSSW_7_1_X_2014-03-25-1400
554 18:42 cmsenv
556 18:43 cd src
559 18:44 git cms-merge-topic VinInn:TrajectoryCleanup
560 19:01 scram b -j 8
runTheMatrix.py --useInput=all --list=limited > & lim.log &
7 7 6 6 4 tests passed, 0 0 0 0 0 failed

cmsbuild · 2014-03-25T22:51:06Z

-1
When I ran the RelVals I found an error in the following worklfows:
4.22 step2

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD.log

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log

4.53 step3

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT/step3_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT.log

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3010/650/summary.html

VinInn · 2014-03-26T09:47:47Z

I can reproduce the crash with gcc4.8.1
this points to a difference in behavior, between 4.8.1 and 4.9.0, of either move semantics or atomics: most probably a combination of both.
Valgrind is running.

If I cannot find a fix (and a reasonable justification) in a day or so, I will cherry pick few urgent mods that are in here and make a different pull request with those.

VinInn · 2014-03-26T13:10:32Z

so Valgrind on gcc490 gives no error of any sort
valgrind on gcc481 produces

==2917== Invalid read of size 8
==2917==    at 0x32DB5562: Trajectory::lastMeasurement() const (intrusive_ptr.hpp:150)
==2917==    by 0x32DB3516: CosmicMuonTrajectoryBuilder::buildSecondHalf(Trajectory&) (CosmicMuonTrajectoryBuilder.cc:573)
==2917==    by 0x32DB48CB: CosmicMuonTrajectoryBuilder::trajectories(TrajectorySeed const&) (CosmicMuonTrajectoryBuilder.cc:367)
==2917==    by 0x15CC35C8: MuonTrackFinder::reconstruct(edm::Handle<edm::View<TrajectorySeed> > const&, edm::Event&) (MuonTrackFinder.cc:95)

as predicted an intrusive ref-counting that is not doing its job...

we do not define constructors and assignment for
ConstReferenceCountingPointer
we rely on the behavior of boost::intrusive_ptr
(have to get rid of those!)
so most probably move (in gcc4.8) decrement the counter

cmsbuild · 2014-03-26T14:59:16Z

Pull request #3010 was updated. @civanch, @Dr15Jones, @danduggan, @ianna, @mdhildreth, @cmsbuild, @anton-a, @thspeer, @rovere, @deguio, @slava77, @Degano, @ojeda, @ktf, @nclopezo can you please check and sign again.

ianna · 2014-03-27T06:20:36Z

Geometry/ForwardCommonData/python/GeometryExtendedTest2014_cff.py

@@ -0,0 +1,9 @@
+import FWCore.ParameterSet.Config as cms


@VinInn - could you, please, delete this file from PR. It looks like this file is misplaced. There is another one like that in Configuration/Geometry already.

merge/rebase mess.
will try to rebase.
thanks for spotting this...

bapavlov and others added 29 commits March 11, 2014 20:02

Asymmetric Cluster Size

d4e3eeb

Asymmetric Cluster Size

2ead60a

removing Trajectory churn

6ceef64

somemore cleanup

37ad202

getByLabel replaced by getByToken and a new digi model is added (Asym…

d923ae7

…metric Clusters)

more cleanup

a122749

prepare to move splitting out of Trajectory

5e2dff2

add class to copy hits from Trajectory to Track with splitting

96d0064

getByLabel replaced by getByToken and a new digi model is added (Asymâ¦

863aa85

cosmic migrated

64a4526

TrackProducer migrated

60da62f

new digitizer added; legacy format supported; getByLabel replaced by …

ea1edf0

…getByToken

Merge http://github.com/bapavlov/cmssw into DifferentDigitizersTokens

8be8fdb

new digitizer added; legacy format supported; getByLabel replaced by …

1d8f85a

…getByToken

fixed reversing of trajectory. migrated all, removed recHitsV method,…

1767e4b

… InOut conversions still problem

Conversions Migrated, no regression

b3ae022

clean more trajectory churn

0058ec7

optimize memory in TransientInitialStateEstimator

17e8c84

add hit() to TrackingRecHit

92c10b1

Additions and modifications for BCM1F SIM

55d14d6

Additions and modifications for BCM1F SIM

7c1843f

Addition of BCM1F sensors to geometry

a1c08aa

Configuration of Test2014 geometry to be used with cmsDriver.py

25e5a3f

cleanup cout

7ba33cd

Updated version

411d1ab

more cleanup

f84cde5

temporary fix to avoid concurrency problems when accessing the data_

010e7c9

small improvements in producers

b8b1654

fix (make it slower) because of BeamHalo runs

2c9205f

VinInn added the core-pending label Mar 25, 2014

Merge pull request cms-sw#3001 from robervalwalsh/my-710pre4-bril-bcm…

c7231be

…1f-simdev Sim -- Simulation of BCM1F

VinInn added core-pending labels Mar 25, 2014

Merged TrajectoryCleanup from repository VinInn

a2a8757

nclopezo modified the milestones: CMSSW_7_1_0_pre6, CMSSW_7_1_0_pre5 Mar 26, 2014

use copy instead of move as gcc481 does not like it

be70d6d

VinInn added core-pending labels Mar 26, 2014

ianna reviewed Mar 27, 2014
View reviewed changes

cmsbuild mentioned this pull request Mar 27, 2014

Trajectory cleanup -- rebase #3046

Merged

VinInn closed this Mar 27, 2014

VinInn deleted the TrajectoryCleanup branch July 13, 2016 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trajectory cleanup #3010

Trajectory cleanup #3010

VinInn commented Mar 25, 2014

cmsbuild commented Mar 25, 2014

VinInn commented Mar 25, 2014

VinInn commented Mar 25, 2014

VinInn commented Mar 25, 2014

VinInn commented Mar 25, 2014

cmsbuild commented Mar 25, 2014

VinInn commented Mar 26, 2014

VinInn commented Mar 26, 2014

cmsbuild commented Mar 26, 2014

ianna Mar 27, 2014

VinInn Mar 27, 2014

Trajectory cleanup #3010

Trajectory cleanup #3010

Conversation

VinInn commented Mar 25, 2014

cmsbuild commented Mar 25, 2014

VinInn commented Mar 25, 2014

VinInn commented Mar 25, 2014

VinInn commented Mar 25, 2014

VinInn commented Mar 25, 2014

cmsbuild commented Mar 25, 2014

VinInn commented Mar 26, 2014

VinInn commented Mar 26, 2014

cmsbuild commented Mar 26, 2014

ianna Mar 27, 2014

Choose a reason for hiding this comment

VinInn Mar 27, 2014

Choose a reason for hiding this comment