Python3: Changes needed to run runTheMatrix with python2/3 #28021

smuzaffar · 2019-09-18T13:48:51Z

PR description:

Changes needed to run runTheMatrix with python3

PR validation:

Locally runing of runTheMatix (to dump relval commands) was successful. Order of command-line arguments to cmsDriver.py is not identical (because of change of dict behaviour in python3)

python2 runTheMatrix.py -n -e -l 137.8 -i all

found a total of  794  workflows:
      of which the following 1 were selected:

137.8  RunEGamma2018C+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+RunEGamma2018D+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Prompt_L1TEgDQM+HARVEST2018_L1TEgDQM_MULTIRUN [1]: input from: /EGamma/Run2018C-v1/RAW with run [] 
                                           [2]: cmsDriver.py step2  --datatier FEVTDEBUGHLT --conditions auto:run2_hlt_relval -s L1REPACK:Full,HLT:@relval2018 --process reHLT --data  --era Run2_2018 --eventcontent FEVTDEBUGHLT -n 100 
                                           [3]: cmsDriver.py step3  --conditions auto:run2_data -s RAW2DIGI,L1Reco,RECO,SKIM:ZElectron+SinglePhotonJetPlusHOFilter+EXOMONOPOLE,EI,PAT,ALCA:SiStripCalZeroBias+SiStripCalMinBias+SiStripCalSmallBiasScan+TkAlMinBias+EcalESAlign,DQM:@standardDQMFakeHLT+@miniAODDQM+@L1TEgamma --runUnscheduled  --process reRECO --data  --era Run2_2018 --eventcontent RECO,MINIAOD,DQM --hltProcess reHLT --scenario pp --datatier RECO,MINIAOD,DQMIO --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run2_2018 -n 100 
                                           [4]: input from: /EGamma/Run2018D-v1/RAW with run []
                                           [5]: cmsDriver.py step5  --datatier FEVTDEBUGHLT --conditions auto:run2_hlt_relval -s L1REPACK:Full,HLT:@relval2018 --process reHLT --data  --era Run2_2018 --eventcontent FEVTDEBUGHLT -n 100 
                                           [6]: cmsDriver.py step6  --conditions auto:run2_data_promptlike -s RAW2DIGI,L1Reco,RECO,SKIM:ZElectron+SinglePhotonJetPlusHOFilter+EXOMONOPOLE,EI,PAT,ALCA:SiStripCalZeroBias+SiStripCalMinBias+SiStripCalSmallBiasScan+TkAlMinBias+EcalESAlign,DQM:@standardDQMFakeHLT+@miniAODDQM+@L1TEgamma --runUnscheduled  --process reRECO --data  --era Run2_2018 --eventcontent RECO,MINIAOD,DQM --hltProcess reHLT --scenario pp --datatier RECO,MINIAOD,DQMIO --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run2_2018 -n 100 
                                           [7]: cmsDriver.py step7  --conditions auto:run2_data -s HARVESTING:@standardDQMFakeHLT+@miniAODDQM+@L1TEgamma --data  --era Run2_2018 --filein file:step6_inDQM.root,file:step3_inDQM.root --scenario pp --filetype DQM --customise Configuration/StandardSequences/DQMSaverAtJobEnd_cff -n 100 

1 workflows with 7 steps

python3 runTheMatrix.py -n -e -l 137.8 -i all

found a total of  794  workflows:
      of which the following 1 were selected:

137.8  RunEGamma2018C+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+RunEGamma2018D+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Prompt_L1TEgDQM+HARVEST2018_L1TEgDQM_MULTIRUN [1]: input from: /EGamma/Run2018C-v1/RAW with run [] 
                                           [2]: cmsDriver.py step2  --process reHLT -s L1REPACK:Full,HLT:@relval2018 --conditions auto:run2_hlt_relval --data  --eventcontent FEVTDEBUGHLT --datatier FEVTDEBUGHLT --era Run2_2018 -n 100 
                                           [3]: cmsDriver.py step3  --runUnscheduled  --conditions auto:run2_data -s RAW2DIGI,L1Reco,RECO,SKIM:ZElectron+SinglePhotonJetPlusHOFilter+EXOMONOPOLE,EI,PAT,ALCA:SiStripCalZeroBias+SiStripCalMinBias+SiStripCalSmallBiasScan+TkAlMinBias+EcalESAlign,DQM:@standardDQMFakeHLT+@miniAODDQM+@L1TEgamma --datatier RECO,MINIAOD,DQMIO --eventcontent RECO,MINIAOD,DQM --data  --process reRECO --scenario pp --era Run2_2018 --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run2_2018 --hltProcess reHLT -n 100 
                                           [4]: input from: /EGamma/Run2018D-v1/RAW with run []
                                           [5]: cmsDriver.py step5  --process reHLT -s L1REPACK:Full,HLT:@relval2018 --conditions auto:run2_hlt_relval --data  --eventcontent FEVTDEBUGHLT --datatier FEVTDEBUGHLT --era Run2_2018 -n 100 
                                           [6]: cmsDriver.py step6  --runUnscheduled  --conditions auto:run2_data_promptlike -s RAW2DIGI,L1Reco,RECO,SKIM:ZElectron+SinglePhotonJetPlusHOFilter+EXOMONOPOLE,EI,PAT,ALCA:SiStripCalZeroBias+SiStripCalMinBias+SiStripCalSmallBiasScan+TkAlMinBias+EcalESAlign,DQM:@standardDQMFakeHLT+@miniAODDQM+@L1TEgamma --datatier RECO,MINIAOD,DQMIO --eventcontent RECO,MINIAOD,DQM --data  --process reRECO --scenario pp --era Run2_2018 --customise Configuration/DataProcessing/RecoTLR.customisePostEra_Run2_2018 --hltProcess reHLT -n 100 
                                           [7]: cmsDriver.py step7  -s HARVESTING:@standardDQMFakeHLT+@miniAODDQM+@L1TEgamma --conditions auto:run2_data --data  --filetype DQM --scenario pp --era Run2_2018 --customise Configuration/StandardSequences/DQMSaverAtJobEnd_cff --filein file:step6_inDQM.root,file:step3_inDQM.root -n 100 

1 workflows with 7 steps

smuzaffar · 2019-09-18T13:49:06Z

FYI @davidlange6

cmsbuild · 2019-09-18T13:49:19Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-09-18T13:55:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28021/11932

This PR adds an extra 64KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, improve upgrade PU assignment #28005
- File Configuration/PyReleaseValidation/python/relval_upgrade.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983

cmsbuild · 2019-09-18T13:55:49Z

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for master.

It involves the following packages:

Configuration/Applications
Configuration/PyReleaseValidation
FWCore/PythonUtilities

@smuzaffar, @Dr15Jones, @chayanit, @zhenhu, @kpedro88, @pgunnell, @franzoni, @cmsbuild, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @wddgit this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

smuzaffar · 2019-09-18T14:14:42Z

please test

cmsbuild · 2019-09-18T14:15:13Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2562/console Started: 2019/09/18 16:15

cmsbuild · 2019-09-18T14:43:13Z

The code-checks are being triggered in jenkins.

davidlange6 · 2019-09-18T14:43:33Z

Configuration/PyReleaseValidation/scripts/das-selected-lumis.py

@@ -16,8 +16,8 @@
 lumi_data = lumi_data['data']

 def match_in(sub_list,lumi_list):
-  sub_list = map(int,sub_list)
-  lumi_list = map(int,lumi_list)
+  sub_list = list(map(int,sub_list))


this starts to be a complicated way of writing
for i in range(int(sub_list[0]),int(sub_list[1])+1):
if i >= int(lumi_list[0]) and i <= int(lumi_list[1]): return True
return False

thanks for suggestion. I have updated it

smuzaffar · 2019-09-25T18:40:50Z

please test

cmsbuild · 2019-09-25T18:41:14Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2681/console Started: 2019/09/25 20:42
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2682/console Started: 2019/09/25 20:48

smuzaffar · 2019-09-25T18:48:23Z

I have moved back to old commit so that we can integrate this PR. I will create a separate PR to fix some more python3 related errors.

cmsbuild · 2019-09-25T20:51:18Z

+1
Tested at: 92d000b
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-77c173/2682/summary.html

cmsbuild · 2019-09-25T20:51:25Z

Comparison job queued.

smuzaffar · 2019-09-25T20:55:31Z

unhold

cmsbuild · 2019-09-25T20:55:58Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2019-09-25T22:45:51Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-77c173/2682/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2958092
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2957749
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 145 log files, 15 edm output root files, 34 DQM output files

fabiocos · 2019-09-26T08:35:10Z

+1

it looks ready to be integrated, and with this a py3-based IB may be prepared for further tests

smuzaffar added 2 commits September 18, 2019 14:06

Python3: Changes needed to run runThePatrix with python2/3

79d3070

fix das-selected-lumis.py for python3

5b8a35b

cmsbuild added this to the CMSSW_11_0_X milestone Sep 18, 2019

cmsbuild added code-checks-pending comparison-pending core-pending operations-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Sep 18, 2019

smuzaffar changed the title ~~Python3: Changes needed to run runThePatrix with python2/3~~ Python3: Changes needed to run runTheMatrix with python2/3 Sep 18, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 18, 2019

cmsbuild added tests-started and removed tests-pending labels Sep 18, 2019

use obj.lstrip() instead of string.lstrip(obj)

4a068da

cmsbuild added code-checks-pending tests-pending and removed code-checks-approved tests-started labels Sep 18, 2019

davidlange6 reviewed Sep 18, 2019

View reviewed changes

cmsbuild added the code-checks-approved label Sep 25, 2019

cmsbuild added tests-started core-approved operations-approved pdmv-approved upgrade-approved and removed tests-pending core-pending operations-pending pdmv-pending upgrade-pending labels Sep 25, 2019

cmsbuild added tests-approved and removed tests-started labels Sep 25, 2019

cmsbuild added fully-signed and removed hold pending-signatures labels Sep 25, 2019

cmsbuild added comparison-available and removed comparison-pending labels Sep 25, 2019

cmsbuild added orp-approved and removed orp-pending labels Sep 26, 2019

cmsbuild merged commit c1f0db5 into cms-sw:master Sep 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python3: Changes needed to run runTheMatrix with python2/3 #28021

Python3: Changes needed to run runTheMatrix with python2/3 #28021

smuzaffar commented Sep 18, 2019

smuzaffar commented Sep 18, 2019

cmsbuild commented Sep 18, 2019

cmsbuild commented Sep 18, 2019

cmsbuild commented Sep 18, 2019

smuzaffar commented Sep 18, 2019

cmsbuild commented Sep 18, 2019 •

edited

Loading

cmsbuild commented Sep 18, 2019

davidlange6 Sep 18, 2019

smuzaffar Sep 18, 2019

smuzaffar commented Sep 25, 2019

cmsbuild commented Sep 25, 2019 •

edited

Loading

smuzaffar commented Sep 25, 2019

cmsbuild commented Sep 25, 2019

cmsbuild commented Sep 25, 2019

smuzaffar commented Sep 25, 2019

cmsbuild commented Sep 25, 2019

cmsbuild commented Sep 25, 2019

fabiocos commented Sep 26, 2019

Python3: Changes needed to run runTheMatrix with python2/3 #28021

Python3: Changes needed to run runTheMatrix with python2/3 #28021

Conversation

smuzaffar commented Sep 18, 2019

PR description:

PR validation:

smuzaffar commented Sep 18, 2019

cmsbuild commented Sep 18, 2019

cmsbuild commented Sep 18, 2019

cmsbuild commented Sep 18, 2019

smuzaffar commented Sep 18, 2019

cmsbuild commented Sep 18, 2019 • edited Loading

cmsbuild commented Sep 18, 2019

davidlange6 Sep 18, 2019

Choose a reason for hiding this comment

smuzaffar Sep 18, 2019

Choose a reason for hiding this comment

smuzaffar commented Sep 25, 2019

cmsbuild commented Sep 25, 2019 • edited Loading

smuzaffar commented Sep 25, 2019

cmsbuild commented Sep 25, 2019

cmsbuild commented Sep 25, 2019

smuzaffar commented Sep 25, 2019

cmsbuild commented Sep 25, 2019

cmsbuild commented Sep 25, 2019

fabiocos commented Sep 26, 2019

cmsbuild commented Sep 18, 2019 •

edited

Loading

cmsbuild commented Sep 25, 2019 •

edited

Loading