New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for multithreading to runTheMatrix #7754
Conversation
Adds --nThreads option for both IB and WMAgent, percolating it down to cmsDriver level.
A new Pull Request was created by @ktf (Giulio Eulisse) for CMSSW_7_4_X. Add support for multithreading to runTheMatrix It involves the following packages: Configuration/PyReleaseValidation @cmsbuild, @srimanob, @nclopezo, @boudoul, @franzoni can you please review it and eventually sign? Thanks. |
-1 runTheMatrix-results/5.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log ----- Begin Fatal Exception 17-Feb-2015 10:54:45 CET----------------------- An exception of category 'FatalRootError' occurred while [0] Processing run: 1 [1] Running path 'validation_step' [2] Calling endRun for module BasicHepMCValidation/'basicHepMCValidation' Additional Info: [a] Fatal Root Error: @SUB=TH1F::Add Attempt to add histograms with different axis limits ----- End Fatal Exception ------------------------------------------------- 135.4 step1 runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log ----- Begin Fatal Exception 17-Feb-2015 11:00:42 CET----------------------- An exception of category 'FatalRootError' occurred while [0] Processing run: 1 [1] Running path 'validation_step' [2] Calling endRun for module BasicHepMCValidation/'basicHepMCValidation' Additional Info: [a] Fatal Root Error: @SUB=TH1F::Add Attempt to add histograms with different axis limits ----- End Fatal Exception ------------------------------------------------- 1330.0 step3 runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15+MINIAODMCUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15+MINIAODMCUP15.log ----- Begin Fatal Exception 17-Feb-2015 12:02:53 CET----------------------- An exception of category 'FatalRootError' occurred while [0] Processing run: 1 [1] Running path 'validation_step' [2] Calling endRun for module BasicHepMCValidation/'basicHepMCValidation' Additional Info: [a] Fatal Root Error: @SUB=TH1F::Add Attempt to add histograms with different axis limits ----- End Fatal Exception ------------------------------------------------- 9.0 step3 runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log ----- Begin Fatal Exception 17-Feb-2015 11:41:04 CET----------------------- An exception of category 'FatalRootError' occurred while [0] Processing run: 1 [1] Running path 'validation_step' [2] Calling endRun for module BasicHepMCValidation/'basicHepMCValidation' Additional Info: [a] Fatal Root Error: @SUB=TH1F::Add Attempt to add histograms with different axis limits ----- End Fatal Exception ------------------------------------------------- 25.0 step3 runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECO+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECO+HARVEST+ALCATT.log ----- Begin Fatal Exception 17-Feb-2015 11:41:04 CET----------------------- An exception of category 'FatalRootError' occurred while [0] Processing run: 1 [1] Running path 'validation_step' [2] Calling endRun for module BasicHepMCValidation/'basicHepMCValidation' Additional Info: [a] Fatal Root Error: @SUB=TH1F::Add Attempt to add histograms with different axis limits ----- End Fatal Exception ------------------------------------------------- 1306.0 step3 runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15+MINIAODMCUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15+MINIAODMCUP15.log ----- Begin Fatal Exception 17-Feb-2015 11:54:02 CET----------------------- An exception of category 'FatalRootError' occurred while [0] Processing run: 1 [1] Running path 'validation_step' [2] Calling endRun for module BasicHepMCValidation/'basicHepMCValidation' Additional Info: [a] Fatal Root Error: @SUB=TH1F::Add Attempt to add histograms with different axis limits ----- End Fatal Exception ------------------------------------------------- 25202.0 step3 runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15+MINIAODMCUP15/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15+MINIAODMCUP15.log you can see the results of the tests here: |
I'm betting the problem histogram is It is possible that one or more streams never get any entries in the histogram since the fill is conditional If so, then this isn't strictly a threading problem since merging files from two different jobs could have the same problem. |
As discussed, it looks like this was running mult-threaded by accident. I changed it to single threaded by default. |
@cmsbuild can you please test |
+1 hello @ktf @sextonkennedy, the proposed modifications look good to me. if no --nThreads is set, the standard single-threaded behaviour is ok. If I specify --nThreads 2, the .py configuration reflects that, still I could only see one thread running (using top). Is this expected ? I could perform no test of injection, those cannot be done with an IB (a prerelease is necessary). I'd like that showWorkFlows displayed "--nThreads" in the cmsDriver commands when using -ne. Could you please look into that ? |
This pull request is fully signed and it will be integrated in one of the next CMSSW_7_4_X IBs unless changes or unless it breaks tests. This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @Degano, @ktf, @smuzaffar |
@cmsbuild please test @franzoni, it is not so smart yet... ;) Ciao, On 24 Feb 2015, at 20:11, Giovanni Franzoni wrote:
|
The tests are being triggered in jenkins. |
This pull request is fully signed and it will be integrated in one of the next CMSSW_7_4_X IBs unless changes or unless it breaks tests. This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @Degano, @ktf, @smuzaffar |
Add support for multithreading to runTheMatrix
Adds --nThreads option for both IB and WMAgent, percolating it
down to cmsDriver level.