Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openblas, Python package cleanup #2915

Merged
merged 7 commits into from Mar 25, 2017

Conversation

davidlange6
Copy link
Contributor

Python updates: migration towards pip and removal of python not in cmssw-tool-conf (all unchanged for 4+ years)

otherwise, openblas included, which should replace the need/desire for atlas libraries. numpy updated. scipy moved to pip, and should pick up the lin. algebra libraries using numpy's configuration. I still need to confirm that.

more checking needed before integrating. but lets see if there are issues at the cmssw level.

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_9_1_X/gcc530.

@cmsbuild, @smuzaffar, @iahmad-khan, @davidlange6 can you please review it and eventually sign? Thanks.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.

@davidlange6
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 23, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/18647/console


%define pip_name pysqlite
Requires: sqlite py2-setuptools
%define PipPreBuild tar xfz pysqlite-%{realversion}.tar.gz;rm pysqlite-%{realversion}.tar.gz;echo "include_dirs=$SQLITE_ROOT/include">>pysqlite-%{realversion}/setup.cfg;tar cfz pysqlite-%{realversion}.tar.gz pysqlite-%{realversion}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not completely safe btw. seems that the tarball gets left around when doing multiple attempts at building.

@cmsbuild
Copy link
Contributor

-1

Tested at: a306037

  • Build:

I found an error when building:

+ sed -ideleteme 's|#!.*/bin/python|#!/usr/bin/env python|' /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/bin/f2py '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py'
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py: No such file or directory
error: Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.sJr54a (%install)


RPM build errors:
Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.sJr54a (%install)



You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2915/18647/summary.html

@smuzaffar
Copy link
Contributor

actual error is

Trying to install the rpm package external+py2-nose+1.3.7-cms just built.
Checking local path dependency for rpm package external+py2-nose+1.3.7-cms just build.
Error: Local path dependency found for external+py2-nose+1.3.7-cms
  /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/slc6_amd64_gcc530/external/python/2.7.11-oenich2/bin/python

@davidlange6
Copy link
Contributor Author

davidlange6 commented Mar 23, 2017 via email

@cmsbuild
Copy link
Contributor

Pull request #2915 was updated.

@davidlange6
Copy link
Contributor Author

davidlange6 commented Mar 23, 2017 via email

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 23, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/18649/console

@cmsbuild
Copy link
Contributor

-1

Tested at: 263988e

  • Build:

I found an error when building:

+ sed -ideleteme 's|#!.*/bin/python|#!/usr/bin/env python|' /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/bin/f2py '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py'
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/0b3a9a88751c837f74829ddff3a2b8d2/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py: No such file or directory
error: Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.a8I3hI (%install)


RPM build errors:
Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.a8I3hI (%install)



You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2915/18649/summary.html

@cmsbuild
Copy link
Contributor

-1

Tested at: 0e98a78

  • Build:

I found an error when building:

+ sed -ideleteme 's|#!.*/bin/python|#!/usr/bin/env python|' /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/bin/f2py '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py'
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py: No such file or directory
error: Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.7ufwiy (%install)


RPM build errors:
Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.7ufwiy (%install)



You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2915/18665/summary.html

@davidlange6
Copy link
Contributor Author

please test

(the test results are gone?)
i checked numpy in my area, it looks like it is set up correctly

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 24, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/18676/console

@cmsbuild
Copy link
Contributor

-1

Tested at: 0e98a78

  • Build:

I found an error when building:

+ sed -ideleteme 's|#!.*/bin/python|#!/usr/bin/env python|' /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/bin/f2py '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py'
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py: No such file or directory
error: Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.UjvdJX (%install)


RPM build errors:
Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.UjvdJX (%install)



You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2915/18676/summary.html

@davidlange6
Copy link
Contributor Author

davidlange6 commented Mar 25, 2017 via email

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 25, 2017

@cmsbuild
Copy link
Contributor

-1

Tested at: 0e98a78

  • Build:

I found an error when building:

+ sed -ideleteme 's|#!.*/bin/python|#!/usr/bin/env python|' /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/bin/f2py '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py' '/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py'
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/EGG-INFO/scripts/f2py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/core/tests/test_arrayprint.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/from_template.py: No such file or directory
sed: can't read /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/BUILDROOT/41cfad3ae9fb2eb48557307da2eac2d3/opt/cmssw/slc6_amd64_gcc530/external/py2-numpy/1.12.1/lib/python*/site-packages/numpy-*/numpy/distutils/conv_template.py: No such file or directory
error: Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.g78bK8 (%install)


RPM build errors:
Bad exit status from /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/testBuildDir/tmp/rpm-tmp.g78bK8 (%install)



You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2915/18695/summary.html

@davidlange6
Copy link
Contributor Author

strange, will try again to reproduce locally...(but means I won't merge tonight as planned.)

@smuzaffar
Copy link
Contributor

I restarted the tests in jenkins and it looks good now ... externals are build and now cmssw is being tested.

@smuzaffar
Copy link
Contributor

smuzaffar commented Mar 25, 2017

looks good now, only one unit test failed and all relval tests passed.
I am merging it now for next IB (which will start in 30 mins)

@smuzaffar smuzaffar merged commit 6c87677 into cms-sw:IB/CMSSW_9_1_X/gcc530 Mar 25, 2017
@cmsbuild
Copy link
Contributor

-1

Tested at: 0e98a78

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2915/18698/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

  • Unit Tests:

I found errors in the following unit tests:

---> test runtestPythonAnalysis had ERRORS

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

@davidlange6
Copy link
Contributor Author

indeed, my local test did finish ok. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants