DataFormats -- changes for large object passed by value found by clang s... #612
Conversation
|
A new Pull Request was created by @gartung (Patrick Gartung) for CMSSW_7_0_X. DataFormats -- changes for large object passed by value found by clang s... It involves the following packages: DataFormats/CSCRecHit @vciulli, @Dr15Jones, @thspeer, @slava77, @vadler, @ktf can you please review it and eventually sign? Thanks. |
|
Pull request #612 was updated. Signatures reset, please check and sign again. |
|
+1 |
|
working @slava77 |
|
I tried in CMSSW_7_0_X_2013-08-26-1400 This commit parent is e448454 What's the reference to use for testing? |
|
Let's ask the developer to rebase to the present head |
|
The following categories have been signed by @slava77: Reconstruction @cms-git-reconstruction |
|
The following categories have been signed by @vadler: Analysis @cms-git-analysis |
|
+1 |
|
Is L1 going to sign this? @mulhearn |
|
Sorry, just getting this now... as you can see I wasn't in the original notification. |
|
There's a problem: CMSSW_7_0_X_2013-08-27-1400 is no longer available... |
|
On 9/11/13 1:38 PM, mulhearn wrote:
I think the pull can be merged into a later IB as well.
Vyacheslav (Slava) Krutelyov |
|
Oh, yes, you are correct. When I saw the large number of packages that "git cms-merge-topic 612" under 09-11-0200 checked out I mistakenly thought something went wrong. In fact, there are just a lot of dependencies from these changes... recompiling will take a little while... |
|
Compiles under CMSSW_7_0_X_2013-09-11-0200... |
|
Under vanilla CMSSW_7_0_Z_2013-09-11-0200 I am able to e.g. runTheMatrix.py -l 30. But after git cms-merge-topic 612, which checks out a whole bunch of dependent packages, then rebuilding, I get complaints about missing HLTTrigger/HLTfilters/triggerResultsFilter_cfi (full listing below). mulhearn@lxplus0411>cmsDriver.py ZMM_8TeV_cfi --conditions auto:startup -s GEN,SIM --datatier GEN-SIM -n 10 --relval 18000,300 --eventcontent RAWSIM |
…g static analyzer cms.ArgSizeChecker
|
Seems to work after rebase Validation/RecoMET (header) |
|
+1 |
****************** PYINIT: initialization of PYTHIA routines *****************
==== PYTHIA WILL USE LHAPDF ====
*************************************
* LHAPDF Version 5.8.5 *
* Configured for the following: *
* All PDFs *
* LOW MEMORY option *
* Maximum 1 concurrent set(s) *
*************************************
>>>>>> PDF description: <<<<<<
CTEQ6L1 - LO with LO alpha_s
Reference:
J. Pumplin, D.R. Stump, J. Huston, H.L. Lai, P. Nadolsky,
W.K. Tung
hep-ph/0201195
>>>>>> <<<<<<
Parametrization: CTEQ6
==============================================
PDFset name /afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc5_amd64_gcc481/external/lhapdf/5.8
with 1 members
==== initialized. ===========================
Strong coupling at Mz for PDF is: 0.12978
==============================================================================
I I
I PYTHIA will be initialized for a p on p collider I
I at 8000.000 GeV center-of-mass energy I
I I
==============================================================================
******** PYMAXI: summary of differential cross-section maximum search ********
==========================================================
I I I
I ISUB Subprocess name I Maximum value I
I I I
==========================================================
I I I
I 1 f + fbar -> gamma*/Z0 I 2.1618D-06 I
I 96 Semihard QCD 2 -> 2 I 8.0723D+03 I
I I I
==========================================================
****** PYMULT: initialization of multiple interactions for MSTP(82) = 4 ******
pT0 = 2.70 GeV gives sigma(parton-parton) = 4.56D+02 mb: accepted
****** PYMIGN: initialization of multiple interactions for MSTP(82) = 4 ******
pT0 = 2.70 GeV gives sigma(parton-parton) = 1.80D+02 mb: accepted
********************** PYINIT: initialization completed **********************
Begin processing the 1st record. Run 1, Event 1, LumiSection 1 at 11-Sep-2013 19:42:45.841 CEST
Begin processing the 2nd record. Run 1, Event 2, LumiSection 1 at 11-Sep-2013 19:42:45.873 CEST
Begin processing the 3rd record. Run 1, Event 3, LumiSection 1 at 11-Sep-2013 19:42:45.877 CEST
Begin processing the 4th record. Run 1, Event 4, LumiSection 1 at 11-Sep-2013 19:42:45.931 CEST
Begin processing the 5th record. Run 1, Event 5, LumiSection 1 at 11-Sep-2013 19:42:59.548 CEST
Begin processing the 6th record. Run 1, Event 6, LumiSection 1 at 11-Sep-2013 19:43:31.565 CEST
Begin processing the 7th record. Run 1, Event 7, LumiSection 1 at 11-Sep-2013 19:44:28.384 CEST
Begin processing the 8th record. Run 1, Event 8, LumiSection 1 at 11-Sep-2013 19:45:34.736 CEST
Begin processing the 9th record. Run 1, Event 9, LumiSection 1 at 11-Sep-2013 19:46:11.248 CEST
Begin processing the 10th record. Run 1, Event 10, LumiSection 1 at 11-Sep-2013 19:46:46.052 CEST
1********* PYSTAT: Statistics on Number of Events and Cross-sections *********
==============================================================================
I I I I
I Subprocess I Number of points I Sigma I
I I I I
I----------------------------------I----------------------------I (mb) I
I I I I
I N:o Type I Generated Tried I I
I I I I
==============================================================================
I I I I
I 0 All included subprocesses I 10 16 I 1.144D-06 I
I 1 f + fbar -> gamma*/Z0 I 10 16 I 1.144D-06 I
I I I I
==============================================================================
********* Total number of errors, excluding junctions = 0 *************
********* Total number of errors, including junctions = 0 *************
********* Total number of warnings = 0 *************
********* Fraction of events that fail fragmentation cuts = 0.00000 *********
=============================================
MessageLogger Summary
Severity # Occurrences Total Occurrences
-------- ------------- -----------------
336.219u 1.691s 5:41.51 98.9% 0+0k 0+0io 886pf+0w
|
|
Working |
|
Sorry, @gartung, after rebuilding, the problem with runTheMatrix.py persists for me. I noticed in your above listing that you ran runTheMatrix.py without rebuilding first. I can run fine that way too. It looks to me like something is going wrong in the dependent packages during the build that breaks runTheMatrix.py / cmsDriver.py, but I haven't sorted it out... mulhearn@lxplus0404>cmsrel CMSSW_7_0_X_2013-09-11-0200
Preparing to run 30.0 ZMM+ZMM+DIGI+RECO+HARVEST in: /afs/cern.ch/work/m/mulhearn/relval/branch_7_0_X/CMSSW_7_0_X_2013-09-11-0200/src going to execute cd 30.0_ZMM+ZMM+DIGI+RECO+HARVESTcmsDriver.py ZMM_8TeV_cfi --conditions auto:startup -s GEN,SIM --datatier GEN-SIM -n 10 --relval 18000,300 --eventcontent RAWSIM --fileout file:step1.root > step1_ZMM+ZMM+DIGI+RECO+HARVEST.log 2>&1 ERROR executing cd 30.0_ZMM+ZMM+DIGI+RECO+HARVEST; cmsDriver.py ZMM_8TeV_cfi --conditions auto:startup -s GEN,SIM --datatier GEN-SIM -n 10 --relval 18000,300 --eventcontent RAWSIM --fileout file:step1.root > step1_ZMM+ZMM+DIGI+RECO+HARVEST.log 2>&1; ret= 256 |
|
Sorry forgot the pre block: mulhearn@lxplus0404>cmsrel CMSSW_7_0_X_2013-09-11-0200 mulhearn@lxplus0404>cd CMSSW_7_0_X_2013-09-11-0200/ mulhearn@lxplus0404>cd src/ mulhearn@lxplus0404>cmsenv mulhearn@lxplus0404>git cms-merge-topic 612 No release tags specified, using default CMSSW_7_0_X_2013-09-11-0200. You are on branch CMSSW_7_0_X Checking out FWCore/Version in tag CMSSW_7_0_X_2013-09-11-0200. Cloning into '/afs/cern.ch/work/m/mulhearn/relval/branch_7_0_X/CMSSW_7_0_X_2013-09-11-0200/src'... remote: Counting objects: 638, done. remote: Compressing objects: 100% (242/242), done. remote: Total 322 (delta 263), reused 136 (delta 77) Receiving objects: 100% (322/322), 53.12 KiB | 0 bytes/s, done. Resolving deltas: 100% (263/263), completed with 234 local objects. Branch from-CMSSW_7_0_X_2013-09-11-0200 set up to track remote branch CMSSW_7_0_X from official-cmssw. Switched to a new branch 'from-CMSSW_7_0_X_2013-09-11-0200' ... [SNIP] ... Validation/RecoMET (header) Validation/RecoMuon (header) Validation/RecoParticleFlow (header) Validation/RecoTau (header) Validation/RecoVertex (header) mulhearn@lxplus0404>scram b -j 8 ***WARNING: Multiple usage of "SimDataFormats/PileupSummaryInfo". Please cleanup "use" in "non-export" section of "src/Validation/RecoVertex/BuildFile". >> Local Products Rules ..... started >> Local Products Rules ..... done ****WARNING: Invalid tool HLTrigger/HLTanalyzers. Please fix src/DQM/HcalMonitorTasks/BuildFile.xml file. ****WARNING: Invalid tool CondCore/EgammaPlugins. Please fix src/RecoEgamma/EgammaTools/BuildFile.xml file. ****WARNING: Invalid tool DQMOffline/JetMET. Please fix src/DQMOffline/JetMET/plugins/BuildFile.xml file. >> Building CMSSW version CMSSW_7_0_X_2013-09-11-0200 ---- ------- copying files from src/Alignment/CommonAlignmentProducer/scripts ------- >> copied AlCaHLTBitMon_ParallelJobs.py >> copied AlCaHLTBitMon_QueryRunRegistry.py ------- copying files from src/Alignment/KalmanAlignmentAlgorithm/scripts ------- >> copied kaps.sh ... [SNIP] ... @@@@ ----> OK EDM Class Version mulhearn@lxplus0404>runTheMatrix.py -l 30 processing relval_standard request for INPUT for None processing relval_highstats request for INPUT for None processing relval_pileup request for INPUT for None processing relval_generator request for INPUT for None processing relval_production request for INPUT for None processing relval_ged request for INPUT for None ignoring relval_upgrade from default matrix processing relval_identity request for INPUT for None Running in 4 thread(s) Preparing to run 30.0 ZMM+ZMM+DIGI+RECO+HARVEST # in: /afs/cern.ch/work/m/mulhearn/relval/branch_7_0_X/CMSSW_7_0_X_2013-09-11-0200/src going to execute cd 30.0_ZMM+ZMM+DIGI+RECO+HARVEST cmsDriver.py ZMM_8TeV_cfi --conditions auto:startup -s GEN,SIM --datatier GEN-SIM -n 10 --relval 18000,300 --eventcontent RAWSIM --fileout file:step1.root > step1_ZMM+ZMM+DIGI+RECO+HARVEST.log 2>&1 ERROR executing cd 30.0_ZMM+ZMM+DIGI+RECO+HARVEST; cmsDriver.py ZMM_8TeV_cfi --conditions auto:startup -s GEN,SIM --datatier GEN-SIM -n 10 --relval 18000,300 --eventcontent RAWSIM --fileout file:step1.root > step1_ZMM+ZMM+DIGI+RECO+HARVEST.log 2>&1; ret= 256 30.0_ZMM+ZMM+DIGI+RECO+HARVEST Step0-FAILED Step1-NOTRUN Step2-NOTRUN Step3-NOTRUN - time date Thu Sep 12 10:23:40 2013-date Thu Sep 12 10:23:34 2013; exit: 256 0 0 0 0 0 0 0 tests passed, 1 0 0 0 failed |
|
It seems to me that the rebase was made on top of CMSSW_7_0_X_2013-09-11-1400 ... |
|
Thanks Slava. I am cutting and pasting your sequence at: /afs/cern.ch/work/m/mulhearn/relval/CMSSW_7_0_X_2013-09-11-1400-test612 Running exactly the same sequence with only caveat that I use tcsh (so "setenv SCRAM_ARCH slc5_amd64_gcc481") Where is your working directory? I am at |
|
The last steps after it built successfully were runTheMatrix.py -s --useInput all >& matrix.log & So, something else is going on in your case |
|
After cut and paste of exactly as posted, still fails: /afs/cern.ch/work/m/mulhearn/relval/CMSSW_7_0_X_2013-09-11-1400-test612/src/matrix.log |
|
weird. Something is getting wrong with your build. Looking at details: in your case in lxplus445:/tmp/mulhearn/CMSSW_7_0_X_2013-09-11-1400 it doesn't even find the directory Note that in my case I have the following directories in cfipython: you have only So, you should be looking for some error messages in the build log file. |
|
(I can't read files in your afs/work) |
|
I built and installed CMSSW_7_0_X_2013-09-11-1400 on a Fermilab build machine. I created a project area for CMSSW_7_0_X_2013-09-11-1400 and followed the steps Slava did. I also see many directories under cfipython. I am running runTheMatrix.py right now and it did not fail immediately as in your case. |
|
I think you refer to @slava77, rather than me. |
|
Had to step out for a minute... @slava77: changed AFS permissions, but also you can find it here lxplus445:/tmp/mulhearn/abuild.log |
|
+1 Above problem is clearly on my end, not replicated by @slava77, so trusting the above reports and signing off. |
|
+1 |
|
Testing on CMSSW_7_0_X_2013-09-09-1400 showed some problems in all generator workflows using LHE files 501.0_MinBias_TuneZ2star_8TeV_pythia6+MinBias_TuneZ2star_8TeV_pythia6+HARVGEN Step0-PASSED Step1-PASSED - time date Fri Sep 13 13:10:19 2013-date Fri Sep 13 13:09:45 2013; exit: 0 0 We need some further investigation |
|
-1 |
|
Ok, I'll move this to pre5. |
|
I suggest trying to run valgrind on this. Perhaps there is an uninitialized value and based on the memory configuration you get a different value which can cause it to fail. |
|
@nclopezo, can you add a bit in the jenkins pull request tests which runs Ciao, GiulioOffice: 6-2-25 CERN |
|
this comes out of |
| @@ -131,7 +131,7 @@ class FlavorHistoryEvent { | |||
| const_reverse_iterator rend() const { return histories_.rend(); } | |||
| // here is the proper mutable interface... this is done so that the cache is | |||
| // set by us, not the user | |||
| void push_back( value_type v ) { cached_ = false; histories_.push_back(v); } | |||
| void push_back( const value_type& v ) { cached_ = false; histories_.push_back(v); } | |||
There was a problem hiding this comment.
I have a hunch that this is the change that caused the error.
There was a problem hiding this comment.
Turns out not to be the case.
|
Appears to be broken even in CMSSW_7_0_0_pre3. Note the failure here is that the step2 cannot find the root file from step1. I ran the command for step1 and it dies without producing any root file. runTheMatrix.py -l 517 processing relval_standard request for INPUT for None processing relval_highstats request for INPUT for None processing relval_pileup request for INPUT for None processing relval_generator request for INPUT for None processing relval_production request for INPUT for None processing relval_ged request for INPUT for None ignoring relval_upgrade from default matrix processing relval_identity request for INPUT for None Running in 4 thread(s) Preparing to run 517.0 QCD_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+QCD_Ht-10 0To250_TuneZ2star_8TeV_madgraph-tauola+HARVGEN # in: /storage/local/data1/gartung/CMSSW_7_0_0_pre3 going to execute cd 517.0_QC D_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+QCD_Ht-100To250_TuneZ2star_8TeV_ma dgraph-tauola+HARVGEN cmsDriver.py Hadronizer_MgmMatchTuneZ2star_8TeV_madgraph_tauola_cff --conditio ns auto:startup --filein lhe:5475 -s GEN,VALIDATION:genvalid_all --datatier GEN --relval 1000000,20000 -n 10 --eventcontent RAWSIM --fileout file:step1.root > step1_QCD_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+QCD_Ht-100To250_TuneZ2star _8TeV_madgraph-tauola+HARVGEN.log 2>&1 ERROR executing cd 517.0_QCD_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+QCD_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+HARVGEN; cmsDriver.py Hadronizer_MgmMatchTuneZ2star_8TeV_madgraph_tauola_cff --conditions auto:startup --filein lhe:5475 -s GEN,VALIDATION:genvalid_all --datatier GEN --relval 1000000,20000 -n 10 --eventcontent RAWSIM --fileout file:step1.root > step1_QCD_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+QCD_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+HARVGEN.log 2>&1; ret= 18688 517.0_QCD_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+QCD_Ht-100To250_TuneZ2star_8TeV_madgraph-tauola+HARVGEN Step0-FAILED Step1-NOTRUN - time date Fri Sep 13 12:59:04 2013-date Fri Sep 13 12:58:16 2013; exit: 18688 0 0 0 tests passed, 1 0 failed Addendum: This happens at Fermilab because cmsDriver silently fails. It does not find "--filein lhe:5475" and leaves LHESource blank. python -i Hadronizer_MgmMatchTuneZ2star_8TeV_madgraph_tauola_cff_GEN_VALIDATION.py
|
|
Everything works fine for me on CMSSW_7_0_X_2013-09-13-1400 |
|
Hi all, I added to the Jenkins tests an option that runs valgrind for a given workflow, you can see the outputs of all the tests here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/525 And if you want to see the results for the valgrind run for 571.0, they are here in the file Valgrind.xml: I can run valgrind on Jenkins in case you need it again. |
|
+1 |
|
+1 |
|
This pull request is fully signed and it will be integrated in one of the next IBs unless changes or unless it breaks tests. @ktf can you please take care of it? |
Misc cleanups -- Cleanup large object passed by value
Use std::clamp(...) in device code now that CUDA supports c++17. Name reused constants in the vertex fitting and splitting.
...tatic analyzer cms.ArgSizeChecker