Python via pip in cmsdist #2678

davidlange6 · 2016-11-25T11:22:46Z

Introduce the ability to build python packages from pip. More testing is certainly needed, however current implementation checks for any dependencies being installed by pip and fails in the case that there are (so still a one to one mapping of python package to spec file).

Two examples are added (py2-configparser and py2-entrypoints) that confirm that dependencies are handled correctly (both are part of the stack of dependencies that jupiter depends on).

The main outstanding issue is to retain the source somehow..

Otherwise, I add rootpy (which I will eventually change to install via pip...)

…me only dependency)

cmsbuild · 2016-11-25T11:23:16Z

A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_9_0_X/gcc530.

@cmsbuild, @smuzaffar, @iahmad-khan, @davidlange6 can you please review it and eventually sign? Thanks.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.

…rking it seems)

…er still has to find out that there are missing dependencies or not...

smuzaffar · 2016-11-26T11:05:02Z

build-with-pip.file

+%build
+
+pip list
+pip install --no-deps -t %{i}/$PYTHON_LIB_SITE_PACKAGES  %{my_name}==%{realversion}


@davidlange6 , try the following. It allows pip to use our py2 tools

export PYTHONUSERBASE=%i pip list pip install --user %{my_name}==%{realversion}

and pip sysy

+ pip install --user entrypoints==0.2.2 Collecting entrypoints==0.2.2 Using cached entrypoints-0.2.2-py2.py3-none-any.whl Requirement already satisfied: configparser>=3.5; python_version == "2.7" in /var/build/muz/pip/w/slc6_amd64_gcc530/external/py2-configparser/3.5.0/lib/python2.7/site-packages (from entrypoints==0.2.2) Installing collected packages: entrypoints Successfully installed entrypoints-0.2.2

…d PKGTOOLS)

davidlange6 · 2016-11-28T10:30:09Z

Needs cms-sw/pkgtools#134
@smuzaffar -can you offer more advice for cleaning this up?

smuzaffar · 2016-11-28T15:36:28Z

@davidlange6 , looks good to me.
Looks like this is going to grow the PYTHONPATH a lot. Should we introduce a meta package for example py2-pip-packages.spec which basically depend on all the py2-* pip based packages and all it does is to rsync py2 packages in on single directory and we make cmssw only depend on this mate package. This way we will have only one directory (for these pip based packages) in PYTHONPATH?

cmsbuild · 2016-11-29T09:35:46Z

Pull request #2678 was updated.

smuzaffar · 2016-11-29T20:51:33Z

please test
pkgtools changes are now in, so test should work now

cmsbuild · 2016-11-29T20:51:48Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/cmsdist-test-pr/420/console

cmsbuild · 2016-11-29T20:55:40Z

-1

Tested at: b0ab4d4

Build:

I found an error when building:

+ eval PKGTOOLS/cmsBuild -i /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/testBuildDir --repository cms.week1 --arch slc6_amd64_gcc530 -j 24 build py2-rootpy-toolfile cmssw-tool-conf
++ PKGTOOLS/cmsBuild -i /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/testBuildDir --repository cms.week1 --arch slc6_amd64_gcc530 -j 24 build py2-rootpy-toolfile cmssw-tool-conf
Bootstrapping cmsBuild area.
Done. Setup log can be found in /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/testBuildDir/tmp/bootstrap.log.
/build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/CMSDIST/py2-notebook.spec
ERROR! File not found: /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/CMSDIST/py2-notebook.spec

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2678/420/summary.html

davidlange6 · 2016-11-29T20:58:55Z

ah - must be me… will look tomorrow..

…

On Nov 29, 2016, at 9:55 PM, cmsbuild ***@***.***> wrote: -1 Tested at: b0ab4d4 • Build: I found an error when building: + eval PKGTOOLS/cmsBuild -i /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/testBuildDir --repository cms.week1 --arch slc6_amd64_gcc530 -j 24 build py2-rootpy-toolfile cmssw-tool-conf ++ PKGTOOLS/cmsBuild -i /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/testBuildDir --repository cms.week1 --arch slc6_amd64_gcc530 -j 24 build py2-rootpy-toolfile cmssw-tool-conf Bootstrapping cmsBuild area. Done. Setup log can be found in /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/testBuildDir/tmp/bootstrap.log. /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/CMSDIST/py2-notebook.spec ERROR! File not found: /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/CMSDIST/py2-notebook.spec You can see the results of the tests here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2678/420/summary.html — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

smuzaffar · 2016-11-29T20:59:12Z

ok, was going to remind you

cmsbuild · 2016-11-30T14:10:04Z

Pull request #2678 was updated.

davidlange6 · 2016-11-30T14:11:10Z

please test

…

On Nov 30, 2016, at 3:10 PM, cmsbuild ***@***.***> wrote: Pull request #2678 was updated. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

cmsbuild · 2016-11-30T14:11:27Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/cmsdist-test-pr/424/console

cmsbuild · 2016-11-30T15:55:34Z

-1

Tested at: bd3a03e

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2678/424/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

Unit Tests:

I found errors in the following unit tests:

---> test runtestPythonAnalysis had ERRORS

cmsbuild · 2016-11-30T15:55:37Z

Comparison job queued.

cmsbuild · 2016-11-30T16:58:42Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2678/424/summary.html

The workflows 1003.0, 1001.0, 1000.0, 140.53, 136.731, 4.22 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

davidlange6 · 2016-11-30T17:03:22Z

Interface change that my unit tests need to adapt to..

…

On Nov 30, 2016, at 4:55 PM, cmsbuild ***@***.***> wrote: -1 Tested at: bd3a03e You can see the results of the tests here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2678/424/summary.html I found follow errors while testing this PR Failed tests: UnitTests • Unit Tests: I found errors in the following unit tests: ---> test runtestPythonAnalysis had ERRORS — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

cmsbuild · 2016-11-30T17:12:45Z

The tests are being triggered in jenkins.
Using cmssw from cms-sw/cmssw#16823
https://cmssdt.cern.ch/jenkins/job/cmsdist-cmssw-test-pr/80/console

cmsbuild · 2016-11-30T18:54:38Z

+1
Tested at: bd3a03e
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-16823/80/summary.html

cmsbuild · 2016-11-30T18:54:43Z

Comparison job queued.

davidlange6 added 5 commits November 23, 2016 16:18

bumping histogrammar version, add rootpy and pip (pip will be buildti…

b432b0c

…me only dependency)

remove local paths

ef721b6

a working pip spec file and some examples using pip to install python

ad2bc46

use -t instead of --root

ae5b127

and a working pip spec

2439ca2

davidlange6 changed the title ~~Pipping1~~ Python via pip in cmsdist Nov 25, 2016

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Nov 25, 2016

davidlange6 added 2 commits November 25, 2016 15:28

fix install path, clean up logs (but dependency check is still not wo…

8f696df

…rking it seems)

use no-deps to install, so this will do what we want, but the develop…

0d0c4fa

…er still has to find out that there are missing dependencies or not...

smuzaffar reviewed Nov 26, 2016

View reviewed changes

add source caching and use Shahzads recipe for building (needs update…

76d63a7

…d PKGTOOLS)

davidlange6 added 2 commits November 29, 2016 10:35

add pre/post/build options

918add6

get rootpy using pip

b0ab4d4

cmsbuild added tests-started and removed tests-pending labels Nov 29, 2016

cmsbuild added tests-rejected and removed tests-started labels Nov 29, 2016

comment out a change still in the works

bd3a03e

cmsbuild added tests-pending and removed tests-rejected labels Nov 30, 2016

cmsbuild added tests-started and removed tests-pending labels Nov 30, 2016

cmsbuild added tests-rejected and removed tests-started labels Nov 30, 2016

davidlange6 mentioned this pull request Nov 30, 2016

update interface to matplotlib with new histogrammar version cms-sw/cmssw#16823

Merged

cmsbuild added tests-started and removed tests-rejected labels Nov 30, 2016

cmsbuild added tests-approved and removed tests-started labels Nov 30, 2016

davidlange6 merged commit 13e84ca into cms-sw:IB/CMSSW_9_0_X/gcc530 Dec 1, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python via pip in cmsdist #2678

Python via pip in cmsdist #2678

davidlange6 commented Nov 25, 2016

cmsbuild commented Nov 25, 2016

smuzaffar Nov 26, 2016

davidlange6 commented Nov 28, 2016

smuzaffar commented Nov 28, 2016

cmsbuild commented Nov 29, 2016

smuzaffar commented Nov 29, 2016

cmsbuild commented Nov 29, 2016 •

edited

cmsbuild commented Nov 29, 2016

davidlange6 commented Nov 29, 2016 via email

smuzaffar commented Nov 29, 2016

cmsbuild commented Nov 30, 2016

davidlange6 commented Nov 30, 2016 via email

cmsbuild commented Nov 30, 2016 •

edited

cmsbuild commented Nov 30, 2016

cmsbuild commented Nov 30, 2016

cmsbuild commented Nov 30, 2016

davidlange6 commented Nov 30, 2016 via email

cmsbuild commented Nov 30, 2016 •

edited

cmsbuild commented Nov 30, 2016

cmsbuild commented Nov 30, 2016

Python via pip in cmsdist #2678

Python via pip in cmsdist #2678

Conversation

davidlange6 commented Nov 25, 2016

cmsbuild commented Nov 25, 2016

smuzaffar Nov 26, 2016

Choose a reason for hiding this comment

davidlange6 commented Nov 28, 2016

smuzaffar commented Nov 28, 2016

cmsbuild commented Nov 29, 2016

smuzaffar commented Nov 29, 2016

cmsbuild commented Nov 29, 2016 • edited

cmsbuild commented Nov 29, 2016

davidlange6 commented Nov 29, 2016 via email

smuzaffar commented Nov 29, 2016

cmsbuild commented Nov 30, 2016

davidlange6 commented Nov 30, 2016 via email

cmsbuild commented Nov 30, 2016 • edited

cmsbuild commented Nov 30, 2016

cmsbuild commented Nov 30, 2016

cmsbuild commented Nov 30, 2016

davidlange6 commented Nov 30, 2016 via email

cmsbuild commented Nov 30, 2016 • edited

cmsbuild commented Nov 30, 2016

cmsbuild commented Nov 30, 2016

cmsbuild commented Nov 29, 2016 •

edited

cmsbuild commented Nov 30, 2016 •

edited

cmsbuild commented Nov 30, 2016 •

edited