[PIP] force install pip packages even if there are already installed

[smuzaffar]

CMS docker container has some pip packages already installed [a] which causes some of these packages to be picked up from system instead of building and distributing them in cmssw env. This change should force build/install these packages.

[a]

bash-4.2$ pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                      Version
---------------------------- -------
backports.ssl-match-hostname 3.5.0.1
chardet                      2.2.1
iniparse                     0.4
ipaddress                    1.0.16
javapackages                 1.0.0
kerberos                     1.1
kitchen                      1.1.1
lxml                         3.2.1
pip                          19.2.1
pycurl                       7.19.0
pygobject                    3.22.0
pygpgme                      0.3
pyliblzma                    0.5.3
pyparsing                    1.5.6
python-dateutil              1.5
pyxattr                      0.5.1
requests                     2.6.0
requests-kerberos            0.7.0
setuptools                   0.9.8
six                          1.9.0
urlgrabber                   3.10
urllib3                      1.10.2
yum-metadata-parser          1.1.4

[cmsbuild]

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_11_0_X/master.

@cmsbuild, @smuzaffar, @mrodozov can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

[cmsbuild]

Pull request #5243 was updated.

[smuzaffar]

please test

unfortunately this will rebuild all pip based packages

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2650/console Started: 2019/09/24 17:12
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2652/console Started: 2019/09/24 17:18

[cmsbuild]

-1

Tested at: 461f3fc

• Build:

I found compilation error when building:

File "/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/py2-pip/9.0.3-nmpfii/lib/python3.6/site-packages/pip/req/req_install.py", line 878, in install
spinner=spinner,
File "/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/py2-pip/9.0.3-nmpfii/lib/python3.6/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
% (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/python3/3.6.4-pafccj/bin/python3 -u -c "import setuptools, tokenize;__file__='/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/pip-build-iprxbwng/numpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" --hdf5=/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/hdf5/1.8.17-pafccj --bzip2=/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/bz2lib/1.0.6-pafccj install --record /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/pip-752buz62-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/pip-build-iprxbwng/numpy/
error: Bad exit status from /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.bBrgm6 (%build)


RPM build errors:
Bad exit status from /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.bBrgm6 (%build)



You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a18204/2652/summary.html

[smuzaffar]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2654/console Started: 2019/09/24 21:12
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2655/console Started: 2019/09/24 21:16
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2657/console Started: 2019/09/24 23:46

[cmsbuild]

Pull request #5243 was updated.

[cmsbuild]

-1

Tested at: 2c276af

• Build:

I found compilation error when building:

++ which gcc
+ CC='/build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/gcc -fPIC'
+ PLATF_CONF_OPTS='--enable-shared --disable-static'
+ rm -f ./Config/config.sub ./Config/config.guess
+ curl -L -k -s -o ./Config/config.sub 'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD'
error: Bad exit status from /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.5qgco1 (%build)


RPM build errors:
Bad exit status from /build/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.5qgco1 (%build)



You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a18204/2655/summary.html

[cmsbuild]

-1

Tested at: 2c276af

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a18204/2657/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

• Unit Tests:

I found errors in the following unit tests:

---> test testPyDevParameterSet had ERRORS

[cmsbuild]

Comparison job queued.

[smuzaffar]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2659/console Started: 2019/09/25 06:57

[cmsbuild]

-1

Tested at: 2c276af

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a18204/2659/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

• Unit Tests:

I found errors in the following unit tests:

---> test TestRunnerFWCoreTFWLiteSelector had ERRORS

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a18204/2659/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2958092
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2957750
• DQMHistoTests: Total skipped: 341
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 145 log files, 15 edm output root files, 34 DQM output files

[smuzaffar]

+externals
lets get this in to fix the broken pip packages

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_0_X/master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)