Add scikit-learn to CMSSW environment #2387

gkasieczka · 2016-07-13T09:26:21Z

Dear Colleagues,

I would like to request adding scikit-learn [1] to the CMSSW enviroment. scikit-learn is a widely used Python package for machine learning. It is also increasingly used in CMS, especially in b-tagging.

As pointed out by Valentin ([2]) spec-files and RPMs already exist in a branch of cmsdist [3].

The spec for scikit-learn is here:
https://github.com/cms-sw/cmsdist/blob/comp_gcc493/py2-scikit-learn.spec

I would however propose to move to the latest version of scikit learn (0.17.1).

The key dependencies are numpy and scipy (both have spec files in [4] as well).

I would also suggest to include Pandas [4,5] which offers much convenience for data handling as input to scikit.

Thank you!

[1] http://scikit-learn.org/stable/
[2] https://hypernews.cern.ch/HyperNews/CMS/get/swDevelopment/3355/1.html
[3] https://github.com/cms-sw/cmsdist/tree/comp_gcc493
[4] http://pandas.pydata.org/
[5] https://github.com/cms-sw/cmsdist/blob/comp_gcc493/py2-pandas.spec

cmsbuild · 2016-07-13T09:26:49Z

A new Issue was created by @gkasieczka .

@davidlange6, @smuzaffar, @Degano, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are list here cms-sw/cmssw#13029

jpata · 2016-07-13T09:30:55Z

Dear BTV conveners (@acaudron @pvmulder @mverzett @imarches @carolinecollard, ...)
Dear Tau conveners (@veelken, ...)

would you lend your support to having a recent version scikit-learn available with CMSSW? That would simplify training efforts for various MVA-s.

davidlt · 2016-07-13T09:36:47Z

@davidlange6 your ACK will be required here.

mverzett · 2016-07-13T09:41:07Z

@jpata, that would be indeed a nice addition indeed. I am currently running everything on a custom anaconda build because of this.

It would be also nice if the package is kept up-to-date.

blinkseb · 2016-07-13T09:41:24Z

Can we also please update the version of matplotlib we ship in CMSSW? It's currently very old, and it's the perfect tool to plot results from scikit :)

davidlt · 2016-07-13T09:47:55Z

We did have 1.4.1 version of matplotlib in 2015, but that had to reverted due to some issues. Sadly commit message does not indicate what they were thus will be hard to follow up. @Degano do you recall what were the issues?

Other than that, all of this would be first tested in DEVEL IBs before proposing final changes to stable IBs.

Keeping it up do date will require someone maintaining it, or at least someone checking for new versions/options and making CMSDIST issues to update.

davidlt · 2016-07-13T11:34:49Z

@iahmad-khan no, it's at 1.2.1: https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_8_1_X/gcc530next/py2-matplotlib.spec

vkuznet · 2016-07-13T12:51:17Z

I maintain ML stack within comp_gcc493 repository. And, right now building recent version of numpy, scipy, pandas, scikit-learn, xgboost, vw, etc. All of them are in use of package I'm responsible for, DCAFPilot.

mbluj · 2016-07-13T13:14:28Z

Hello,

as Tau conveners were pinged I am answering: yes, we are interested in having scikit-learn available with external packages of CMSSW which will make it easier studies towards replacement our TMVA based tools by ones based on scikit-learn.
Best,
Michal

iahmad-khan · 2016-08-01T09:22:51Z

Matplotlib and numpy related changes have been merged in #2391

iahmad-khan · 2016-08-01T14:24:43Z

Merged in #2426

veelken · 2016-08-04T08:52:08Z

Dear all,

can you let me know in which CMSSW version (or IB) scikit-learn will be included by default.
One of the tau ID experts in the Tau POG would like to start using it.

Thank you very much,

Christian

davidlange6 · 2016-08-04T09:00:19Z

at this moment, its in the 81x integration build. It will be part of 810pre10 and then it can be tested and back ported to 80x if desired (I would desire it).

david

On Aug 4, 2016, at 10:52 AM, Christian Veelken <notifications@github.com mailto:notifications@github.com> wrote:

Dear all,

can you let me know in which CMSSW version (or IB) scikit-learn will be included by default.
One of the tau ID experts in the Tau POG would like to start using it.

Thank you very much,

Christian

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/2387#issuecomment-237492913, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEzyww8PURxR063dzN7bv-V2V2ObHi8cks5qcag5gaJpZM4JLOcz.

jpata · 2016-08-25T09:00:41Z

Hi @davidlange6 , pinging also @gkasieczka

scikit-learn doesn't seem to be there in the 810_pre10, perhaps we want to reopen:

$ find /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_0_pre10 -path "*scikit*" | wc -l
0

Or am I looking in the wrong place?

gkasieczka · 2016-08-25T09:56:02Z

Hi,

I would haved expected it to show up in:
/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external
(where it isn't either).

smuzaffar · 2016-08-25T10:04:28Z

@gkasieczka , it is only available in CMSSW_8_1_DEVEL_X IBs

scram p CMSSW_8_1_DEVEL_X_2016-08-24-2300
cd CMSSW_8_1_DEVEL_X_2016-08-24-2300
cmsenv
python -c 'import sklearn; print "OK"'

can you please check and confirm if it works as expected? then we can include it in normal 80X IBs.

davidlange6 · 2016-08-25T10:04:32Z

ah - thanks for checking - it seems we only put this into the DEVEL branches and not the mainstream. Not sure why. Will fix asap.

On Aug 25, 2016, at 11:56 AM, gkasieczka notifications@github.com wrote:

Hi,

I would haved expected it to show up in:
/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external
(where it isn't either).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

gkasieczka · 2016-08-25T11:33:56Z

@smuzaffar Yes, the tests works. However, the DEVEL branch still take it from /cvmfs/cms-ib.cern.ch/ which is not guaranteed to be available on grid sites.

@davidlange6 Thank you!

aknayak · 2016-08-31T16:25:05Z

Hi, I checked that scikit-learn is available in CMSSW_8_1_0_pre11. But, I did not find root_numpy. Is it not needed? I think, it is needed to convert root trees to read in sklearn. Is it possible to add it? Thanks.

aknayak · 2016-10-06T10:32:20Z

Hi, there is a problem in 810_pre12 as given below. It works ok in pre11. problem seems to be due to scipy. How to find out which version of sklean and scipy included in pre11 and pre12?

import sklearn
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/init.py", line 57, in
from .base import clone
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/base.py", line 11, in
from .utils.fixes import signature
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/utils/init.py", line 11, in
from .validation import (as_float_array,
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/utils/validation.py", line 16, in
from ..utils.fixes import signature
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/utils/fixes.py", line 324, in
from scipy.sparse.linalg import lsqr as sparse_lsqr
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/sparse/linalg/init.py", line 109, in
from .isolve import *
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/init.py", line 8, in
from .lgmres import lgmres
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/lgmres.py", line 8, in
from scipy.linalg import get_blas_funcs
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/linalg/init.py", line 172, in
from .misc import *
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/linalg/misc.py", line 6, in
from .lapack import get_lapack_funcs
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/linalg/lapack.py", line 356, in
from scipy.linalg import _flapack
ImportError: /cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/linalg/flapack.so: undefined symbol: sgegv

davidlange6 · 2016-10-06T10:50:34Z

interesting - we definitely need a unit test...

you can look at

scram tool list
scram tool info

On Oct 6, 2016, at 2:32 PM, aknayak notifications@github.com wrote:

Hi, there is a problem in 810_pre12 as given below. It works ok in pre11. problem seems to be due to scipy. How to find out which version of sklean and scipy included in pre11 and pre12?

import sklearn
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/init.py", line 57, in
from .base import clone
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/base.py", line 11, in
from .utils.fixes import signature
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/utils/init.py", line 11, in
from .validation import (as_float_array,
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/utils/validation.py", line 16, in
from ..utils.fixes import signature
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scikit-learn/0.17.1-giojec/lib/python2.7/site-packages/sklearn/utils/fixes.py", line 324, in
from scipy.sparse.linalg import lsqr as sparse_lsqr
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/sparse/linalg/init.py", line 109, in
from .isolve import *
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/init.py", line 8, in
from .lgmres import lgmres
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/lgmres.py", line 8, in
from scipy.linalg import get_blas_funcs
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/linalg/init.py", line 172, in
from .misc import *
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/linalg/misc.py", line 6, in
from .lapack import get_lapack_funcs
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/linalg/lapack.py", line 356, in
from scipy.linalg import flapack
ImportError: /cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/py2-scipy/0.16.1-giojec/lib/python2.7/site-packages/scipy/linalg/_flapack.so: undefined symbol: sgegv

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

davidlt · 2016-10-06T11:04:14Z

Correct. In order to make sure things are working:

Tests are needed to make sure all components are working correctly together (they aren't according to this issue)
Tests are needed to ensure that result continues to be valid while we are updating compilers and other libraries
Tests must be part of CMSSW IB otherwise no one will notice failures

In this particular case, I guess, py2-scipy was built against system lapack library and in CMSSW we use our own, which does not provide sgegv symbol. Our CMSSW_8_1_0_pre12/external/slc6_amd64_gcc530/lib/liblapack.so.3 does not have this symbol. Looks like this particular symbol has been deprecated in lapack library.

Well, we don't run any tests while building RPMs thus we never know if package is working.

davidlange6 · 2016-10-06T11:12:16Z

so the problem is that we build against the wrong lapack library - @iahmad-khan - can you have a look at using the one we distribute while building py2-scipy?

iahmad-khan · 2016-10-06T11:42:55Z

i am looking into it

""In lapack 3.6.0 some deprecated functions which are used in scipy were dropped. Have to use lapack 3.5.0 or lower in order to get things to work back or upgrade scipy. ""

iahmad-khan · 2016-10-06T11:55:05Z

SciPy 0.16.1 [released 2015-10-24] , that is in use now , is too old , the newest is SciPy 0.18.0 [ released 2016-07-25] . i am going to upgrade and test.

davidlange6 · 2016-10-06T11:55:49Z

ok - so likely my fault for thinking to update lapack.
#2527
my notes for why are lacking - hopefully this does not break something else (I think its a problem for something else not yet introduced, but thats ok)

On Oct 6, 2016, at 3:42 PM, Ijaz ahmad khan notifications@github.com wrote:

i am looking into it

""In lapack 3.6.0 some deprecated functions which are used in scipy were dropped. Have to use lapack 3.5.0 or lower in order to get things to work back or upgrade scipy. ""

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

iahmad-khan · 2016-10-06T11:58:25Z

@davidlange6 , ok , so should i downgrade lapack or upgrade scipy?

davidlange6 · 2016-10-06T12:00:30Z

best to upgrade scipy if that is an option. yes.

On Oct 6, 2016, at 3:58 PM, Ijaz ahmad khan notifications@github.com wrote:

@davidlange6 , ok , so should i downgrade lapack or upgrade scipy?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

iahmad-khan · 2016-10-06T12:00:39Z

Ok

davidlt · 2016-10-06T12:00:41Z

Add latest version of scipy to DEVEL IBs.

iahmad-khan · 2016-10-06T12:01:07Z

ok , [SciPy 0.18.0 released 2016-07-25] is the latest.

davidlt · 2016-10-06T12:04:16Z

@aknayak if you want this working in future release, you (or someone else) will need to provide some QA to validate this machinery.

aknayak · 2016-10-06T12:26:17Z

I can do a quick test, let me know the version where it is included.

cmsbuild added the pending-assignment label Jul 13, 2016

iahmad-khan mentioned this issue Jul 13, 2016

upgrades matplotlib to 1.5.0 #2391

Merged

ghost assigned iahmad-khan Aug 1, 2016

iahmad-khan mentioned this issue Aug 1, 2016

adds scikit-learn to externals and updates scipy #2426

Merged

iahmad-khan closed this as completed Aug 1, 2016

jpata mentioned this issue Aug 25, 2016

switch to non-IB as soon as possible jpata/tthbb13#124

Open

iahmad-khan mentioned this issue Sep 1, 2016

adds root_numpy #2481

Merged

iahmad-khan mentioned this issue Oct 6, 2016

upgrades py2-scipy to 0.18.1 #2562

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scikit-learn to CMSSW environment #2387

Add scikit-learn to CMSSW environment #2387

gkasieczka commented Jul 13, 2016

cmsbuild commented Jul 13, 2016

jpata commented Jul 13, 2016

davidlt commented Jul 13, 2016

mverzett commented Jul 13, 2016

blinkseb commented Jul 13, 2016

davidlt commented Jul 13, 2016

davidlt commented Jul 13, 2016

vkuznet commented Jul 13, 2016

mbluj commented Jul 13, 2016

iahmad-khan commented Aug 1, 2016

iahmad-khan commented Aug 1, 2016

veelken commented Aug 4, 2016

davidlange6 commented Aug 4, 2016

jpata commented Aug 25, 2016

gkasieczka commented Aug 25, 2016

smuzaffar commented Aug 25, 2016

davidlange6 commented Aug 25, 2016

gkasieczka commented Aug 25, 2016

aknayak commented Aug 31, 2016

aknayak commented Oct 6, 2016

davidlange6 commented Oct 6, 2016

davidlt commented Oct 6, 2016

davidlange6 commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

davidlange6 commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

davidlange6 commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

davidlt commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

davidlt commented Oct 6, 2016

aknayak commented Oct 6, 2016

Add scikit-learn to CMSSW environment #2387

Add scikit-learn to CMSSW environment #2387

Comments

gkasieczka commented Jul 13, 2016

cmsbuild commented Jul 13, 2016

jpata commented Jul 13, 2016

davidlt commented Jul 13, 2016

mverzett commented Jul 13, 2016

blinkseb commented Jul 13, 2016

davidlt commented Jul 13, 2016

davidlt commented Jul 13, 2016

vkuznet commented Jul 13, 2016

mbluj commented Jul 13, 2016

iahmad-khan commented Aug 1, 2016

iahmad-khan commented Aug 1, 2016

veelken commented Aug 4, 2016

davidlange6 commented Aug 4, 2016

jpata commented Aug 25, 2016

gkasieczka commented Aug 25, 2016

smuzaffar commented Aug 25, 2016

davidlange6 commented Aug 25, 2016

gkasieczka commented Aug 25, 2016

aknayak commented Aug 31, 2016

aknayak commented Oct 6, 2016

davidlange6 commented Oct 6, 2016

davidlt commented Oct 6, 2016

davidlange6 commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

davidlange6 commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

davidlange6 commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

davidlt commented Oct 6, 2016

iahmad-khan commented Oct 6, 2016

davidlt commented Oct 6, 2016

aknayak commented Oct 6, 2016