remove python2 pip support and remaining python2 packages except for future #7112

davidlange6 · 2021-07-07T13:47:22Z

No description provided.

…future

cmsbuild · 2021-07-07T13:47:43Z

A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_12_0_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @iarspider can you please review it and eventually sign? Thanks.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

davidlange6 · 2021-07-07T13:48:27Z

please test

smuzaffar · 2021-07-07T13:56:35Z

@davidlange6 , we also need to update https://github.com/cms-sw/cmssw/blob/5244916401e27d97ce2dace3341db78c5c08098f/PhysicsTools/PythonAnalysis/test/BuildFile.xml#L29-L31 . I think just

<test name="testNumPy" command="python3 -c 'import numpy'"/>

should be enough now ( i.e drop the <iftool ...> condition)

davidlange6 · 2021-07-07T14:18:08Z

Yes, looks like the best way to go is to make that change after this is merged

…

On Jul 7, 2021, at 3:56 PM, Malik Shahzad Muzaffar ***@***.***> wrote: @davidlange6 , we also need to update https://github.com/cms-sw/cmssw/blob/5244916401e27d97ce2dace3341db78c5c08098f/PhysicsTools/PythonAnalysis/test/BuildFile.xml#L29-L31 . I think just <test name="testNumPy" command="python3 -c 'import numpy'"/> should be enough now ( i.e drop the <iftool ...> condition) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

cmsbuild · 2021-07-07T14:37:54Z

-1

Failed Tests: UnitTests RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16552/summary.html
COMMIT: 3b7bf5b
CMSSW: CMSSW_12_0_X_2021-07-06-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7112/16552/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

more changes for py3 in CRABClient #7108 @belforte: more changes for py3 in CRABClient
Update tag for L1Trigger-TrackFindingTracklet to V00-02-00 #7109 @cms-sw: Update tag for L1Trigger-TrackFindingTracklet to V00-02-00
move gdb to python3 #7107 @davidlange6: move gdb to python3

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16552/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16552/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test import-keras had ERRORS
---> test import-tensorflow had ERRORS
---> test testTheano had ERRORS

RelVals

AddOn Tests

fastsimcmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Wed Jul 7 16:35:11 2021-date Wed Jul 7 16:35:11 2021 s - exit: 256
pat1

----- Begin Fatal Exception 07-Jul-2021 16:35:12 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_0_X_2021-07-06-2300/src/PhysicsTools/PatAlgos/test/IntegrationTest_cfg.py
Exception Message:
 unknown python problem occurred.
ImportError: No module named six

At:
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_0_X_2021-07-06-2300/python/FWCore/ParameterSet/Config.py(6): <module>
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_0_X_2021-07-06-2300/python/PhysicsTools/PatAlgos/patTemplate_cfg.py(1): <module>
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_0_X_2021-07-06-2300/src/PhysicsTools/PatAlgos/test/IntegrationTest_cfg.py(2): <module>

----- End Fatal Exception -------------------------------------------------

hlt_data_Fake2cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_Fake2 --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016 --fileout file:RelVal_Raw_Fake2_DATA.root --filein /store/data/Run2016B/JetHT/RAW/v1/000/272/762/00000/C666CDE2-E013-E611-B15A-02163E011DBE.root : FAILED - time: date Wed Jul 7 16:35:15 2021-date Wed Jul 7 16:35:15 2021 s - exit: 256

Expand to see more addon errors ...

smuzaffar · 2021-07-08T06:09:38Z

please test
lets use a newer IB

davidlange6 · 2021-07-08T06:39:32Z

my guess is that none of these will be fixed. I need to finish my PR for replacing /usr/bin/env python with /usr/bin/env python3..

davidlange6 · 2021-07-08T06:40:42Z

well, but maybe I'm wrong. Keras/tensorflow etc have been python3 only for a while..

smuzaffar · 2021-07-08T06:44:44Z

Tensorflow for sure is py3 only since long

add back tensor flow - removed by mistake

davidlange6 · 2021-07-08T06:52:36Z

please abort

cmsbuild · 2021-07-08T06:52:42Z

Pull request #7112 was updated.

davidlange6 · 2021-07-08T06:52:59Z

please test

davidlange6 · 2021-07-08T06:53:25Z

sorry - my mistake..

davidlange6 · 2021-07-08T06:55:09Z

at least some (all?) of the other errors will be fixed by a PR merged (to move Configuration scripts to python3).

smuzaffar · 2021-07-13T17:45:17Z

test parameters:

full_cmssw = true

smuzaffar · 2021-07-13T17:45:27Z

please test

smuzaffar · 2021-07-13T17:46:00Z

@davidlange6 , lets see if cms-sw/cmssw#34471,cms-sw/cmssw#34472,cms-sw/cmssw#34473,cms-sw/cmssw#34474,cms-sw/cmssw#34475 set of PRs fixes the rest of the issues

smuzaffar · 2021-07-13T22:04:22Z

ok one unit test fails now ( src/FWCore/ParameterSet/test/TestFWCoreParameterSetDriver ) this should be fixed by cms-sw/cmssw#34477

smuzaffar · 2021-07-13T22:04:51Z

please test

cmsbuild · 2021-07-14T05:42:06Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16789/summary.html
COMMIT: 050ac0a
CMSSW: CMSSW_12_0_X_2021-07-13-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7112/16789/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16789/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16789/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreParameterSetDriver had ERRORS
---> test testhep_ml had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2786302
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2786279
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

smuzaffar · 2021-07-14T08:09:10Z

@davidlange6 , now TestFWCoreParameterSetDriver is failing in the same way as it is failing in PY3 IBs. Can you please check this unit tests in PY3 IBs? No idea why testhep_ml failed now, may be a glitch

smuzaffar · 2021-07-14T08:13:43Z

the framework test is being followed here cms-sw/cmssw#34477 (comment)

davidlange6 · 2021-07-14T08:23:10Z

I can have a look in ~2 hours - no problem

…

On Jul 14, 2021, at 10:09 AM, Malik Shahzad Muzaffar ***@***.***> wrote: @davidlange6 , now TestFWCoreParameterSetDriver is failing in the same way as it is failing in PY3 IBs. Can you please check this unit tests in PY3 IBs? No idea why testhep_ml failed now, may be a glitch — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

smuzaffar · 2021-07-16T03:44:14Z

please test

cmsbuild · 2021-07-16T10:26:34Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16876/summary.html
COMMIT: 050ac0a
CMSSW: CMSSW_12_0_X_2021-07-15-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7112/16876/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@gartung Utilities/StaticAnalyzers: Update generation of EventSetupRecord::get report. cmssw#34389
@bsunanda Run3-alca188 Add one AlCa to HLTPhysics stream cmssw#34486
@wddgit Improve MessageLogger Context Handling for ProcessBlock Transitions cmssw#34506
@makortel Use python3 in FWCore/ParameterSet unit tests cmssw#34493
@bsunanda Run3-hcx302 Update the analyzer code to test the showerlibrary and the script which combines all momentum bins to make the library cmssw#34500

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16876/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-508a55/16876/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2786302
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2786273
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

smuzaffar · 2021-07-16T10:36:36Z

looks good, lets get this in

remove python2 pip support and remaining python2 packages except for …

3b7bf5b

…future

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Jul 7, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 7, 2021

cmsbuild added tests-rejected and removed tests-started labels Jul 7, 2021

cmsbuild added tests-started and removed tests-rejected labels Jul 8, 2021

Update python_tools.spec

b563979

add back tensor flow - removed by mistake

cmsbuild added tests-pending and removed tests-started labels Jul 8, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 8, 2021

This was referenced Jul 13, 2021

CondTools/SiStrip: use python3 cms-sw/cmssw#34474

Merged

CMSCommonData: use python3 in unit test cms-sw/cmssw#34475

Merged

cmsbuild added requires-external tests-started and removed tests-rejected labels Jul 13, 2021

smuzaffar mentioned this pull request Jul 13, 2021

ParameterSet: Use python3 to run tests cms-sw/cmssw#34477

Closed

cmsbuild added tests-rejected and removed tests-started labels Jul 14, 2021

cmsbuild added tests-started and removed requires-external tests-rejected labels Jul 16, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 16, 2021

smuzaffar merged commit e8f3759 into cms-sw:IB/CMSSW_12_0_X/master Jul 16, 2021

This was referenced Jul 16, 2021

[Tensorflow] Update to version 2.5.0 #7141

Merged

[TOOLFILES] Check mergebility after merging with latest cmsdist #7110

Closed

[PY3] Update to python 3.9.6 #7138

Closed

qliphy mentioned this pull request Jul 17, 2021

Segfault in Herwig7GeneratorFilter cms-sw/cmssw#34529

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove python2 pip support and remaining python2 packages except for future #7112

remove python2 pip support and remaining python2 packages except for future #7112

davidlange6 commented Jul 7, 2021

cmsbuild commented Jul 7, 2021

davidlange6 commented Jul 7, 2021

smuzaffar commented Jul 7, 2021

davidlange6 commented Jul 7, 2021 via email

cmsbuild commented Jul 7, 2021

smuzaffar commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

smuzaffar commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

cmsbuild commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

smuzaffar commented Jul 13, 2021 •

edited

smuzaffar commented Jul 13, 2021

smuzaffar commented Jul 13, 2021

smuzaffar commented Jul 13, 2021

smuzaffar commented Jul 13, 2021

cmsbuild commented Jul 14, 2021

smuzaffar commented Jul 14, 2021

smuzaffar commented Jul 14, 2021

davidlange6 commented Jul 14, 2021 via email

smuzaffar commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

remove python2 pip support and remaining python2 packages except for future #7112

remove python2 pip support and remaining python2 packages except for future #7112

Conversation

davidlange6 commented Jul 7, 2021

cmsbuild commented Jul 7, 2021

davidlange6 commented Jul 7, 2021

smuzaffar commented Jul 7, 2021

davidlange6 commented Jul 7, 2021 via email

cmsbuild commented Jul 7, 2021

Unit Tests

RelVals

AddOn Tests

smuzaffar commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

smuzaffar commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

cmsbuild commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

davidlange6 commented Jul 8, 2021

smuzaffar commented Jul 13, 2021 • edited

smuzaffar commented Jul 13, 2021

smuzaffar commented Jul 13, 2021

smuzaffar commented Jul 13, 2021

smuzaffar commented Jul 13, 2021

cmsbuild commented Jul 14, 2021

Unit Tests

Comparison Summary

smuzaffar commented Jul 14, 2021

smuzaffar commented Jul 14, 2021

davidlange6 commented Jul 14, 2021 via email

smuzaffar commented Jul 16, 2021

cmsbuild commented Jul 16, 2021

Comparison Summary

smuzaffar commented Jul 16, 2021

smuzaffar commented Jul 13, 2021 •

edited