Python: improvements for pip based packages

[smuzaffar]

• improved build-with-pip rules
  • automatic patching support
  • use correct pip for downloading sources
  • create tmp directory used by pip install under packages' BUILD tree
  • disable pip cache to avoid usage of ~/.cache/pip
  • automatic relocate site-package files to use correct python
  • automatic relocate script under bin if package is build for only one flavor of python
• bug fix: make sure pip generated files are readable. We have seen that on aarch64, sometime the directories are only readable by user
• Allow to have separate py2- and py3 packages. For py3- packages add ;python>'3.0' and for py2- package use ;python_version<'3.0' in requirement file
• clean up many python pip based packages to use new relocation and patching improvements
• updated setuptools to version 41.2.0
• use package BUILD area for bazel cache directory.
• properly relocate pip and tensorflow python files.
• Updated few python packages to their newer version ( see python package updates #5219 )

[cmsbuild]

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_11_0_X/master.

@cmsbuild, @smuzaffar, @mrodozov can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

[cmsbuild]

Pull request #5240 was updated.

[cmsbuild]

Pull request #5240 was updated.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e8e753/2618/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2958092
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2957750
• DQMHistoTests: Total skipped: 341
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 145 log files, 15 edm output root files, 34 DQM output files

[cmsbuild]

Pull request #5240 was updated.

[smuzaffar]

+externals

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2627/console Started: 2019/09/23 22:41
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2628/console Started: 2019/09/23 22:50

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_0_X/master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[smuzaffar]

this has been successfully built for amd64, aarch64 and ppc64le. lets get this in IB

[cmsbuild]

-1

Tested at: 823cc54

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e8e753/2628/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

• Unit Tests:

I found errors in the following unit tests:

---> test test_MpsWorkFlow had ERRORS
---> test ExpressionEvaluatorUnitTest had ERRORS

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e8e753/2628/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2958092
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2957750
• DQMHistoTests: Total skipped: 341
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 145 log files, 15 edm output root files, 34 DQM output files

[fabiocos]

@smuzaffar is the cause of unit test failures clear? I do not see any in the IB though...