Adapt FWCore/PythonUtilities to python3 #28439

fabiocos · 2019-11-20T22:03:33Z

PR description:

Apply futurize to FWCore/PythonUtilities and adapt edmDumpEventContent to python3.

PR validation:

Tested locally both in the standard and python3 version of master.

cmsbuild · 2019-11-20T22:05:47Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-11-20T22:11:05Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28439/12849

This PR adds an extra 28KB to repository

cmsbuild · 2019-11-20T22:11:36Z

A new Pull Request was created by @fabiocos (Fabio Cossutti) for master.

It involves the following packages:

FWCore/PythonUtilities

@cmsbuild, @smuzaffar, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @wddgit this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

fabiocos · 2019-11-20T22:12:04Z

please test

cmsbuild · 2019-11-20T22:12:28Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3565/console Started: 2019/11/20 23:12

makortel · 2019-11-20T22:23:13Z

FWCore/PythonUtilities/python/LumiList.py

@@ -73,12 +77,12 @@ def __init__(self, filename = None, lumis = None, runsAndLumis = None, runs = No
        if isinstance(runsAndLumis, list):
            queued = {}
            for runLumiList in runsAndLumis:
-                for run, lumis in runLumiList.items():
+                for run, lumis in list(runLumiList.items()):


Here and in many cases below: why is a list created here? The iteration should work fine in both py2 and py3 with the return value of items() (or keys()). (or am I missing something?)

FWCore/PythonUtilities/scripts/edmDumpEventContent

makortel · 2019-11-20T22:28:13Z

FWCore/PythonUtilities/python/LumiList.py

@@ -242,7 +246,7 @@ def getLumis(self):
        Return the list of pairs representation
        """
        theList = []
-        runs = self.compactList.keys()
+        runs = list(self.compactList.keys())
        runs = sorted(run, key=int)


(possibly not for this PR) I wonder if this should be

runs = sorted(runs, key=int)

given that there is no local variable run.

cmsbuild · 2019-11-20T23:40:57Z

-1

Tested at: 41284cb

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8ff155/3565/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following workflows:
4.53 step2

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step2_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log

136.788 step2

runTheMatrix-results/136.788_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017/step2_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017.log

136.731 step2

runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step2_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log

136.85 step2

runTheMatrix-results/136.85_RunEGamma2018A+RunEGamma2018A+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM/step2_RunEGamma2018A+RunEGamma2018A+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM.log

4.22 step2

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

136.8311 step2

runTheMatrix-results/136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/step2_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017.log

136.7611 step2

runTheMatrix-results/136.7611_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM/step2_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM.log

140.56 step2

runTheMatrix-results/140.56_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18/step2_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18.log

140.53 step2

runTheMatrix-results/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

cmsbuild · 2019-11-20T23:41:00Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

cmsbuild · 2019-11-23T21:46:53Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-11-23T21:54:12Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28439/12893

This PR adds an extra 16KB to repository

cmsbuild · 2019-11-23T21:54:42Z

Pull request #28439 was updated. @cmsbuild, @smuzaffar, @Dr15Jones can you please check and sign again.

fabiocos · 2019-11-23T21:59:30Z

please test

cmsbuild · 2019-11-23T21:59:56Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3604/console Started: 2019/11/23 23:00

cmsbuild · 2019-11-23T23:10:05Z

+1
Tested at: 7678ff0
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8ff155/3604/summary.html

cmsbuild · 2019-11-23T23:10:08Z

Comparison job queued.

cmsbuild · 2019-11-24T00:27:43Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8ff155/3604/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2785613
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2785270
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

Dr15Jones · 2019-11-25T14:31:00Z

+1

cmsbuild · 2019-11-25T14:31:26Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2019-11-25T21:08:17Z

+1

apply futurize + fix edmDumpEventContent

41284cb

cmsbuild added this to the CMSSW_11_0_X milestone Nov 20, 2019

cmsbuild added code-checks-pending comparison-pending core-pending orp-pending pending-signatures tests-pending labels Nov 20, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 20, 2019

cmsbuild added tests-started and removed tests-pending labels Nov 20, 2019

makortel reviewed Nov 20, 2019

View reviewed changes

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Nov 20, 2019

fabiocos mentioned this pull request Nov 21, 2019

Explicitly copy scripts which should end up in cmssw/bin directory #28434

Merged

cmsbuild added code-checks-pending comparison-pending and removed code-checks-approved comparison-notrun tests-rejected labels Nov 21, 2019

cmsbuild added comparison-pending tests-pending labels Nov 23, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 23, 2019

cmsbuild added tests-started and removed tests-pending labels Nov 23, 2019

cmsbuild added tests-approved and removed tests-started labels Nov 23, 2019

cmsbuild added comparison-available and removed comparison-pending labels Nov 24, 2019

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Nov 25, 2019

cmsbuild added orp-approved and removed orp-pending labels Nov 25, 2019

cmsbuild merged commit b796e80 into cms-sw:master Nov 25, 2019

fabiocos deleted the fc-py3scripts branch December 18, 2019 13:44

makortel mentioned this pull request Jun 26, 2023

question about py2/py3 cms-sw/cmsdist#8570

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adapt FWCore/PythonUtilities to python3 #28439

Adapt FWCore/PythonUtilities to python3 #28439

fabiocos commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

fabiocos commented Nov 20, 2019

cmsbuild commented Nov 20, 2019 •

edited

makortel Nov 20, 2019

makortel Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 23, 2019

cmsbuild commented Nov 23, 2019

cmsbuild commented Nov 23, 2019

fabiocos commented Nov 23, 2019

cmsbuild commented Nov 23, 2019 •

edited

cmsbuild commented Nov 23, 2019

cmsbuild commented Nov 23, 2019

cmsbuild commented Nov 24, 2019

Dr15Jones commented Nov 25, 2019

cmsbuild commented Nov 25, 2019

fabiocos commented Nov 25, 2019

Adapt FWCore/PythonUtilities to python3 #28439

Adapt FWCore/PythonUtilities to python3 #28439

Conversation

fabiocos commented Nov 20, 2019

PR description:

PR validation:

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

fabiocos commented Nov 20, 2019

cmsbuild commented Nov 20, 2019 • edited

makortel Nov 20, 2019

Choose a reason for hiding this comment

makortel Nov 20, 2019

Choose a reason for hiding this comment

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 23, 2019

cmsbuild commented Nov 23, 2019

cmsbuild commented Nov 23, 2019

fabiocos commented Nov 23, 2019

cmsbuild commented Nov 23, 2019 • edited

cmsbuild commented Nov 23, 2019

cmsbuild commented Nov 23, 2019

cmsbuild commented Nov 24, 2019

Dr15Jones commented Nov 25, 2019

cmsbuild commented Nov 25, 2019

fabiocos commented Nov 25, 2019

cmsbuild commented Nov 20, 2019 •

edited

cmsbuild commented Nov 23, 2019 •

edited