Update setuptools for py2 and py3 and pip for py3 #6557
Conversation
|
A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_11_3_X/master. @cmsbuild, @smuzaffar, @mrodozov can you please review it and eventually sign? Thanks. |
|
please test
… On Jan 16, 2021, at 1:47 PM, cmsbuild ***@***.***> wrote:
A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_11_3_X/master.
@cmsbuild, @smuzaffar, @mrodozov can you please review it and eventually sign? Thanks.
cms-bot commands are listed here
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fc36c8/12309/summary.html Comparison SummarySummary:
|
|
@davidlange6 , as this is going to build nearly all the pythoon packages, I would suggest to do the same for pip (for python3) i.e. a separate py3-pip package |
|
Pip seems not to have dropped py2 support yet; so hopefully there is no reason to split things before we drop py2.
I can try bumping it forward if you want. Did we run into problems upgrading pip before? We are 3 years behind now.
… On Jan 18, 2021, at 9:34 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
@davidlange6 , as this is going to build nearly all the pythoon packages, I would suggest to do the same for pip (for python3) i.e. a separate py3-pip package
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
IIRC, the newer pip also needed some extra packages and for couple of these there were cyclic dependencies. I hope those are fixed by now |
|
how about adding py3-setuptools dependency via https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_11_3_X/master/build-with-pip.file . We include py2-setuptools in build-with-pip.file , I would suggest to update that to include py3-setuptools. if you decided to add py3-pip then that can also go in build-with-pip |
|
Ha - meaning none of the pip/*.files need py*setuptools as a requirement.
… On Jan 18, 2021, at 9:50 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
how about adding py3-setuptools dependency via https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_11_3_X/master/build-with-pip.file . We include py2-setuptools in build-with-pip.file , I would suggest to update that to include py3-setuptools.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
right, as setuptool is very common dependency, so we had decided to include it via the common build file ( |
|
Ok, I started a test with 20.3.3, but it already failed with missing dependencies that seem never to have been supported in python2. Bof, ok, so splitting pip as to solve problems with python3.
(The new pip says that pip 21.0 will drop support for Python 2.7 in January 2021.)
… On Jan 18, 2021, at 9:44 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
IIRC, the newer pip also needed some extra packages and for couple of these there were cyclic dependencies. I hopw those are fixed by now
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
The first interesting error is
RpmInstallFailed: Failed to install package py3-scipy. Reason:
error: Failed dependencies:
libtatlas.so.3()(64bit) is needed by external+py3-scipy+1.5.4-cms2-1-1.x86_64
Atlas is needed by Scipy according to its documentation. Are we getting this via heplibs or something?
… On Jan 18, 2021, at 10:14 AM, David Lange ***@***.***> wrote:
Ok, I started a test with 20.3.3, but it already failed with missing dependencies that seem never to have been supported in python2. Bof, ok, so splitting pip as to solve problems with python3.
(The new pip says that pip 21.0 will drop support for Python 2.7 in January 2021.)
> On Jan 18, 2021, at 9:44 AM, Malik Shahzad Muzaffar ***@***.***> wrote:
>
>
> IIRC, the newer pip also needed some extra packages and for couple of these there were cyclic dependencies. I hopw those are fixed by now
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Pull request #6557 was updated. |
davidlange6
changed the title
|
ok, I tested up through python_tools except for tensorflow. I'll start tests here. |
|
please test |
|
Pull request #6557 was updated. |
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fc36c8/12538/summary.html Comparison SummarySummary:
|
|
@davidlange6 , this looks much better now. Comparison diff are due to messagelogger. |
|
Now to know what changed:)
On Jan 26, 2021 8:33 PM, Malik Shahzad Muzaffar <notifications@github.com> wrote:
@davidlange6<https://github.com/davidlange6> , this looks much better now. Comparison diff are due to messagelogger.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#6557 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQYYXHVIJJKMU6WSMILS34KHBANCNFSM4WFHUOYQ>.
|
|
please test for slc7_aarch64_gcc9 |
|
please test for slc7_ppc64le_gcc9 |
|
please test |
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fc36c8/12556/summary.html Comparison SummaryThe workflows 140.53 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons Summary:
|
|
-1 Failed Tests: UnitTests RelVals Unit TestsI found errors in the following unit tests: ---> test test_PrepareInputDb had ERRORS ---> test test_MpsWorkFlow had ERRORS ---> test testUnits had ERRORS ---> test testDD4hepFilteredViewFirstChild had ERRORS and more ... RelVals
|
|
+externals |
cmsbuild
added
externals-approved
fully-signed
and removed
externals-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_3_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2) |
|
-1 Failed Tests: UnitTests Unit TestsI found errors in the following unit tests: ---> test TestFWCoreServicesDriver had ERRORS ---> test PhiTest had ERRORS ---> test TestConfigDP had ERRORS |
Separate the py2 and py3 versions of setuptools to allow us to follow the last year of development after py2 support was dropped. These are now separate spec files.
This pr includes #6556 as all of those packages would be rebuilt anyway.
There are some new dependencies included. One to note is a number of new dependencies on wheel, which others have encountered when bumping setuptools. I just added a build requires in each case.
From discussion in this PR, pip is now updated for python3. While current pip releases nominally support python2, I quickly ran into dependency issues that I didn't want to solve for an obsolete stack. So it is updated for python3 only.
The one oddity is with scipy (as commented below). I don't think there is any regression, just a newly known dependency.