Make CkfPattern MT ready #4754

VinInn · 2014-07-23T09:42:33Z

Technical change to allow to run CkfPattern in parallel.
For the time being is compiler flag protected.
Not obvious to use a unique central configuration switch to control it

cmsbuild · 2014-07-23T09:42:47Z

A new Pull Request was created by @VinInn (Vincenzo Innocente) for CMSSW_7_2_X.

Make CkfPattern MT ready

It involves the following packages:

RecoTracker/CkfPattern

@nclopezo, @cmsbuild, @Degano, @StoyanStoynev, @slava77 can you please review it and eventually sign? Thanks.
@ghellwig, @GiacomoSguazzoni, @rovere, @gpetruc, @cerati, @venturia this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.

VinInn · 2014-07-23T09:45:52Z

@Dr15Jones
when merged

git cms-addpkg RecoTracker/CkfPattern
scram b -j 8 USER_CXXFLAGS="-DVI_TBB"

cmsbuild · 2014-07-23T12:20:19Z

-1
Tested at: d090a8a
When I ran the RelVals I found an error in the following worklfows:
4.53 step2

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT/step2_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT.log
----- Begin Fatal Exception 23-Jul-2014 14:05:31 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

4.22 step2

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log
----- Begin Fatal Exception 23-Jul-2014 14:05:38 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

401.0 step1

runTheMatrix-results/401.0_TTbarNewMix+TTbarFSPU2+HARVESTFS/step1_TTbarNewMix+TTbarFSPU2+HARVESTFS.log
----- Begin Fatal Exception 23-Jul-2014 14:06:17 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoEgammaEgammaElectronProducersPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log
----- Begin Fatal Exception 23-Jul-2014 14:07:10 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD.log
----- Begin Fatal Exception 23-Jul-2014 14:08:13 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log
----- Begin Fatal Exception 23-Jul-2014 14:08:18 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

50101.0 step2

runTheMatrix-results/50101.0_SingleMuPt10+SingleMuPt10FSIdINPUT+SingleMuPt10FS_ID/step2_SingleMuPt10+SingleMuPt10FSIdINPUT+SingleMuPt10FS_ID.log
----- Begin Fatal Exception 23-Jul-2014 14:09:02 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoEgammaEgammaElectronProducersPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

25202.0 step2

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVEST+MINIAODMC/step2_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVEST+MINIAODMC.log
----- Begin Fatal Exception 23-Jul-2014 14:17:11 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4754/430/summary.html

VinInn · 2014-07-23T13:40:13Z

@Dr15Jones
we need a centralized solution to the TLS

Dr15Jones · 2014-07-23T13:48:05Z

We were hoping to avoid it given we have no way of controlling the use of TLS in externals. The plan was to get the 'dynamic' increase in TLS space working in the system libraries. However, that only works for shared libraries and not static libraries.

VinInn · 2014-07-23T13:51:32Z

is here the problem that "work_" is file-local and hidden?
should I make it global?

Dr15Jones · 2014-07-23T14:07:26Z

I doubt the problem originates from file local vs global.
@ktf Does David A's change print the message from the failure? It seems odd that the message say 'static'. With David's change, do you know what is the new limiting factors?

Dr15Jones · 2014-07-23T14:27:07Z

Doing a google search on the phrase from the error message gave this response which looks useful

http://stackoverflow.com/questions/19268293/matlab-error-cannot-open-with-static-tls

Dr15Jones · 2014-07-23T14:32:15Z

Following more of the google search I think this one looks useful
http://stackoverflow.com/questions/14892101/cannot-load-any-more-object-with-static-tls

basically it seems to imply the problem is if one compiles some code without -fPIC.

VinInn · 2014-07-23T14:41:33Z

On 23 Jul, 2014, at 4:32 PM, Chris Jones notifications@github.com wrote:

Following more of the google search I think this one looks useful
http://stackoverflow.com/questions/14892101/cannot-load-any-more-object-with-static-tls

basically it seems to imply the problem is if one compiles some code without -fPIC.
we compile with -fPIC

v.

cmsbuild · 2014-07-23T15:08:29Z

Pull request #4754 was updated. @nclopezo, @cmsbuild, @Degano, @StoyanStoynev, @slava77 can you please check and sign again.

Dr15Jones · 2014-07-23T15:11:30Z

@VinInn you can get rid of the headers all together by moving the declaration of the class into the appropriate .cc and move the edm plugin macro call to the same .cc file.

ktf · 2014-07-23T16:35:23Z

For the record. We do have the TLS fix from david in 72X already, since pre2. This is something else. Also we do have -fPIC everywhere. This needs further understanding.

cmsbuild · 2014-07-23T17:36:02Z

-1
Tested at: 8fbbfcf
When I ran the RelVals I found an error in the following worklfows:
4.53 step2

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT/step2_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT.log
----- Begin Fatal Exception 23-Jul-2014 19:19:32 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

4.22 step2

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log
----- Begin Fatal Exception 23-Jul-2014 19:19:42 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

401.0 step1

runTheMatrix-results/401.0_TTbarNewMix+TTbarFSPU2+HARVESTFS/step1_TTbarNewMix+TTbarFSPU2+HARVESTFS.log
----- Begin Fatal Exception 23-Jul-2014 19:20:28 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoEgammaEgammaElectronProducersPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log
----- Begin Fatal Exception 23-Jul-2014 19:21:34 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD.log
----- Begin Fatal Exception 23-Jul-2014 19:22:23 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log
----- Begin Fatal Exception 23-Jul-2014 19:22:28 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

50101.0 step2

runTheMatrix-results/50101.0_SingleMuPt10+SingleMuPt10FSIdINPUT+SingleMuPt10FS_ID/step2_SingleMuPt10+SingleMuPt10FSIdINPUT+SingleMuPt10FS_ID.log
----- Begin Fatal Exception 23-Jul-2014 19:23:34 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoEgammaEgammaElectronProducersPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

25202.0 step2

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVEST+MINIAODMC/step2_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVEST+MINIAODMC.log
----- Begin Fatal Exception 23-Jul-2014 19:32:00 CEST-----------------------
An exception of category 'PluginLibraryLoadError' occurred while
   [0] Constructing the EventProcessor
Exception Message:
unable to load /build/cmsbuild/jenkins-workarea/workspace/ib-integration-CMSSW_7_2_X-slc6_amd64_gcc481/CMSSW_7_2_X_2014-07-22-1400/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so because dlopen: cannot load any more object with static TLS
----- End Fatal Exception -------------------------------------------------

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4754/436/summary.html

VinInn · 2014-07-24T16:25:49Z

xrootd
/home/vin/smallMatrix/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC /home/vin/TrackerMT/CMSSW_7_2_X_2014-07-24-0200 
[innocent@vinavx2 4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC]$ cmsRun step2_RAW2DIGI_L1Reco_RECO_DQM_ALCA.py
24-Jul-2014 18:24:52 CEST  Initiating request to open file file:049F6443-8E53-E011-A943-003048F117EA.root
24-Jul-2014 18:24:52 CEST  Successfully opened file file:049F6443-8E53-E011-A943-003048F117EA.root
HLTConfigData::dump: ProcessName = HLT
HLTConfigData::dump: GlobalTag = GR_H_V15::All
HLTConfigData::dump: TableName = /cdaq/special/Interfill/v2.1/HLT/V2
Begin processing the 1st record. Run 160960, Event 10001082, LumiSection 277 at 24-Jul-2014 18:25:08.310 CEST
%MSG-w EcalDQM:  EcalDQMonitorTask:ecalMonitorTask 24-Jul-2014 18:25:13 CEST  Run: 160960 Event: 10001082
Ecal Monitor Source::runOnCollection: EBReducedRecHit does not exist
%MSG

works...

VinInn · 2014-07-24T17:22:06Z

from local file works, after a file is open from eos ldopen os unhappy

VinInn · 2014-07-24T17:46:51Z

with setenv LD_PRELOAD /home/vin/TrackerMT/CMSSW_7_2_X_2014-07-24-0200/lib/slc6_amd64_gcc481/pluginRecoTrackerCkfPatternPlugins.so

the matrix succeeds..

Dr15Jones · 2014-07-24T18:13:53Z

Reading from EOS is done via xrootd and xrootd starts and stops threads which causes the TLS problem.

VinInn · 2014-07-25T09:12:13Z

it is a bad idea to add -fopenmp only to a plugin
should be added to the main...

cmsbuild · 2014-07-25T09:12:51Z

Pull request #4754 was updated. @nclopezo, @cmsbuild, @Degano, @StoyanStoynev, @slava77 can you please check and sign again.

cmsbuild · 2014-07-25T10:06:08Z

+1
Tested at: b44cb33
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4754/482/summary.html

cmsbuild · 2014-07-25T10:50:01Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4754/482/summary.html

slava77 · 2014-07-28T12:58:02Z

+1

for #4754 b44cb33
checked in regular mode (no MT) in CMSSW_7_2_X_2014-07-25-0200 (are sign397)
no differences as expected

cmsbuild · 2014-07-28T13:02:49Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_2_X IBs unless changes (tests are also fine).

RecoTracker/CkfPattern -- Make CkfPattern MT ready

VinInn added 3 commits July 19, 2014 12:45

works!

d8fef6d

tbb

927ba02

prepare for MT

d090a8a

VinInn added comparison-pending labels Jul 23, 2014

move to plugin

8fbbfcf

VinInn added comparison-pending labels Jul 23, 2014

remove openmp for good, commenitng is not enough

b44cb33

VinInn added comparison-pending labels Jul 25, 2014

VinInn added comparison-available labels Jul 25, 2014

VinInn added comparison-available labels Jul 28, 2014

nclopezo added a commit that referenced this pull request Jul 28, 2014

Merge pull request #4754 from VinInn/TrackerOMP

1b6bfb5

RecoTracker/CkfPattern -- Make CkfPattern MT ready

nclopezo merged commit 1b6bfb5 into cms-sw:CMSSW_7_2_X Jul 28, 2014

VinInn deleted the TrackerOMP branch July 13, 2016 13:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make CkfPattern MT ready #4754

Make CkfPattern MT ready #4754

VinInn commented Jul 23, 2014

cmsbuild commented Jul 23, 2014

VinInn commented Jul 23, 2014

cmsbuild commented Jul 23, 2014

VinInn commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

VinInn commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

VinInn commented Jul 23, 2014

cmsbuild commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

ktf commented Jul 23, 2014

cmsbuild commented Jul 23, 2014

VinInn commented Jul 24, 2014

VinInn commented Jul 24, 2014

VinInn commented Jul 24, 2014

Dr15Jones commented Jul 24, 2014

VinInn commented Jul 25, 2014

cmsbuild commented Jul 25, 2014

cmsbuild commented Jul 25, 2014

cmsbuild commented Jul 25, 2014

slava77 commented Jul 28, 2014

cmsbuild commented Jul 28, 2014

Make CkfPattern MT ready #4754

Make CkfPattern MT ready #4754

Conversation

VinInn commented Jul 23, 2014

cmsbuild commented Jul 23, 2014

VinInn commented Jul 23, 2014

cmsbuild commented Jul 23, 2014

VinInn commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

VinInn commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

VinInn commented Jul 23, 2014

cmsbuild commented Jul 23, 2014

Dr15Jones commented Jul 23, 2014

ktf commented Jul 23, 2014

cmsbuild commented Jul 23, 2014

VinInn commented Jul 24, 2014

VinInn commented Jul 24, 2014

VinInn commented Jul 24, 2014

Dr15Jones commented Jul 24, 2014

VinInn commented Jul 25, 2014

cmsbuild commented Jul 25, 2014

cmsbuild commented Jul 25, 2014

cmsbuild commented Jul 25, 2014

slava77 commented Jul 28, 2014

cmsbuild commented Jul 28, 2014