Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use Rapids in PyCharm #174

Closed
daddydrac opened this issue Jan 31, 2019 · 22 comments
Closed

How to use Rapids in PyCharm #174

daddydrac opened this issue Jan 31, 2019 · 22 comments
Labels
0 - Backlog In queue waiting for assignment doc Documentation help wanted Extra attention is needed

Comments

@daddydrac
Copy link

daddydrac commented Jan 31, 2019

How do you set up PyCharm to use rapids?

@daddydrac daddydrac added ? - Needs Triage Need team to review and classify doc Documentation labels Jan 31, 2019
@datametrician
Copy link

@joehoeller I don't believe anyone on the RAPIDS team uses PyCharm. If you want to experiment and provide docs on this that would be amazing!

@datametrician datametrician added help wanted Extra attention is needed 0 - Backlog In queue waiting for assignment and removed ? - Needs Triage Need team to review and classify labels Jan 31, 2019
@daddydrac
Copy link
Author

Rapids then is a concept. Nothing I can build a system or an app with. It’s an immature product is what you’re telling me, and not quite ready for prime time.

@daddydrac
Copy link
Author

I’m closing this out. I’ll come back when it’s ready to be used to build software.

@mrocklin
Copy link

mrocklin commented Feb 1, 2019

@joehoeller I'm curious how do you set up PyCharm to use Pandas?

The same procedure should also work with RAPIDS.

@daddydrac
Copy link
Author

Fair comeback. In all seriousness I did that, and it doesn’t come up at all in settings > project interpreter > Anaconda

@daddydrac
Copy link
Author

daddydrac commented Feb 1, 2019

Also, I get this: The conda install line is taken right from the instructions on the cuml repo, where it says, "cuML can be installed using the rapidsai conda channel".

`conda install -c nvidia -c rapidsai -c conda-forge -c pytorch -c defaults cuml
Collecting package metadata: done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  • cuml -> libcuml=0.5.0

Current channels:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.
`

@daddydrac
Copy link
Author

daddydrac commented Feb 1, 2019

Because none of the automated installs worked (conda install *), I decided to build from source. During the test phase, these 3 failed:


[  FAILED  ] 3 tests, listed below:
[  FAILED  ] KmeansTests/KmeansTestF.Fit/0, where GetParam() = 16-byte object <02-00 00-00 CD-CC 4C-3D 04-00 00-00 02-00 00-00>
[  FAILED  ] KmeansTests/KmeansTestD.Fit/0, where GetParam() = 24-byte object <02-00 00-00 00-00 00-00 9A-99 99-99 99-99 A9-3F 04-00 00-00 02-00 00-00>
[  FAILED  ] PcaTests/PcaTestDataVecF.Result/0, where GetParam() = 

I also ran into these errors after running py.test:

================================ ERRORS ====================================
__________________ ERROR collecting cuML/test/test_dbscan.py ___________________
ImportError while importing test module '/home/joe/Desktop/rapids/cuml/python/cuML/test/test_dbscan.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
cuML/test/test_dbscan.py:17: in <module>
    from cuml import DBSCAN as cuDBSCAN
E   ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
_______________ ERROR collecting cuML/test/test_kalman_filter.py _______________
ImportError while importing test module '/home/joe/Desktop/rapids/cuml/python/cuML/test/test_kalman_filter.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
cuML/test/test_kalman_filter.py:17: in <module>
    from cuml import KalmanFilter
E   ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
__________________ ERROR collecting cuML/test/test_kmeans.py ___________________
ImportError while importing test module '/home/joe/Desktop/rapids/cuml/python/cuML/test/test_kmeans.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
cuML/test/test_kmeans.py:20: in <module>
    import cuml
E   ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
_______________ ERROR collecting cuML/test/test_linear_model.py ________________
ImportError while importing test module '/home/joe/Desktop/rapids/cuml/python/cuML/test/test_linear_model.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
cuML/test/test_linear_model.py:17: in <module>
    from cuml import LinearRegression as cuLinearRegression
E   ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
____________________ ERROR collecting cuML/test/test_pca.py ____________________
ImportError while importing test module '/home/joe/Desktop/rapids/cuml/python/cuML/test/test_pca.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
cuML/test/test_pca.py:17: in <module>
    from cuml import PCA as cuPCA
E   ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
___________________ ERROR collecting cuML/test/test_tsvd.py ____________________
ImportError while importing test module '/home/joe/Desktop/rapids/cuml/python/cuML/test/test_tsvd.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
cuML/test/test_tsvd.py:16: in <module>
    from cuml import TruncatedSVD as cuTSVD
E   ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 6 error in 0.16 seconds ============================

Here is some addt'l errata, not sure it will help:

Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

from cuml import KNN as cumlKNN
ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

>>> 

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

NVIDIA-SMI 410.48                 Driver Version: 410.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2080 Ti...  Off  | 00000000:41:00.0  On |                  N/A |
| 41%   26C    P8     7W / 260W |    650MiB / 10986MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1501      G   /usr/lib/xorg/Xorg                            40MiB |
|    0      1740      G   /usr/bin/gnome-shell                          58MiB |
|    0      2732      G   /usr/lib/xorg/Xorg                           341MiB |
|    0      2862      G   /usr/bin/gnome-shell                          79MiB |
|    0      3550      G   ...quest-channel-token=1174352793230323240   127MiB |
+-----------------------------------------------------------------------------+

Maybe you can point me to what I need to do to fix that?

@mrocklin
Copy link

mrocklin commented Feb 1, 2019 via email

@daddydrac
Copy link
Author

daddydrac commented Feb 1, 2019

@mrocklin - please see all the errors, one in particular is:

from cuml import KNN as cumlKNN
ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

  • Do you have a contact URL for upstream?

@daddydrac daddydrac reopened this Feb 1, 2019
@mrocklin
Copy link

mrocklin commented Feb 1, 2019

Do you have a contact URL for upstream?

Nope. You'll have to find out how to report things to either PyCharm or conda yourself as appropriate. I suspect that this is particular to your setup though. Sorry I can't be of more help. Good luck!

@daddydrac
Copy link
Author

It’s just Ubuntu 18, Anaconda 5/Python 3.7, & CUDA 10.

@kkraus14
Copy link
Contributor

kkraus14 commented Feb 1, 2019

It looks like cuBLAS isn't installed / recognized based on what I'm seeing but I would defer to @dantegd or @cjnolet here.

@dantegd
Copy link
Member

dantegd commented Feb 1, 2019

I think @kkraus14 is right it looks like an issue finding cuBLAS during runtime of the python package (not the c++ libcuml). I have a couple of ideas of what could be the issue, but can’t say for sure until I’ve seen the details of the environment. @joehoeller could you post the full output of the c++ tests, your environment variables and conda packages (with the conda environment you used for building)? Thanks.

@daddydrac
Copy link
Author

@dantegd, on my lunch break now, but when I get off from my FT i'll be sure to do that this evening. Thank you for pointing this out, much appreciated.

@daddydrac
Copy link
Author

@dantegd

Env var's:

CLUTTER_IM_MODULE=xim
CONDA_SHLVL=1
CONDA_EXE=/home/joe/anaconda3/bin/conda
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_MENU_PREFIX=gnome-
LANG=en_US.UTF-8
DISPLAY=:1
GNOME_SHELL_SESSION_MODE=ubuntu
COLORTERM=truecolor
USERNAME=joe
CONDA_PREFIX=/home/joe/anaconda3
XDG_VTNR=2
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
XDG_SESSION_ID=3
USER=joe
DESKTOP_SESSION=ubuntu
QT4_IM_MODULE=xim
TEXTDOMAINDIR=/usr/share/locale/
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/e46940b2_6ee4_4b1c_bec1_037fbb3aa3e3
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
PWD=/home/joe
HOME=/home/joe
CONDA_PYTHON_EXE=/home/joe/anaconda3/bin/python
TEXTDOMAIN=im-config
SSH_AGENT_PID=7956
QT_ACCESSIBILITY=1
LIBVIRT_DEFAULT_URI=qemu:///system
XDG_SESSION_TYPE=x11
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share:/usr/share:/var/lib/snapd/desktop
XDG_SESSION_DESKTOP=ubuntu
GJS_DEBUG_OUTPUT=stderr
CONDA_PROMPT_MODIFIER=(base) 
GTK_MODULES=gail:atk-bridge
WINDOWPATH=2
TERM=xterm-256color
SHELL=/bin/bash
VTE_VERSION=5202
QT_IM_MODULE=xim
XMODIFIERS=@im=ibus
IM_CONFIG_PHASE=2
XDG_CURRENT_DESKTOP=ubuntu:GNOME
GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
GNOME_TERMINAL_SERVICE=:1.107
XDG_SEAT=seat0
SHLVL=1
GDMSESSION=ubuntu
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
LOGNAME=joe
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
XDG_RUNTIME_DIR=/run/user/1000
XAUTHORITY=/run/user/1000/gdm/Xauthority
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
PATH=/usr/local/cuda-10.0/bin:/home/joe/anaconda3/bin:/home/joe/anaconda3/condabin:/home/joe/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/joe/.local/bin
CONDA_DEFAULT_ENV=base
GJS_DEBUG_TOPICS=JS ERROR;JS LOG
SESSION_MANAGER=local/uptrend:@/tmp/.ICE-unix/7859,unix/uptrend:/tmp/.ICE-unix/7859
LESSOPEN=| /usr/bin/lesspipe %s
GTK_IM_MODULE=ibus
_=/usr/bin/printenv

@daddydrac
Copy link
Author

@dantegd - $PATH vars:

echo $PATH
/usr/local/cuda-10.0/bin:/home/joe/anaconda3/bin:/home/joe/anaconda3/condabin:/home/joe/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/joe/.local/bin

@daddydrac
Copy link
Author

@dantegd List of Anaconda packages:

conda list
# packages in environment at /home/joe/anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0  
alabaster                 0.7.11                   py37_0  
anaconda                  5.3.1                    py37_0  
anaconda-client           1.7.2                    py37_0  
anaconda-navigator        1.9.2                    py37_0  
anaconda-project          0.8.2                    py37_0  
appdirs                   1.4.3            py37h28b3542_0  
arrow                     0.12.1                py37_1002    conda-forge
arrow-cpp                 0.12.0           py37h0e61e49_0    conda-forge
asn1crypto                0.24.0                   py37_0  
astroid                   2.0.4                    py37_0  
astropy                   3.0.4            py37h14c3975_0  
atomicwrites              1.2.1                    py37_0  
attrs                     18.2.0           py37h28b3542_0  
automat                   0.7.0                    py37_0  
babel                     2.6.0                    py37_0  
backcall                  0.1.0                    py37_0  
backports                 1.0                      py37_1  
backports.shutil_get_terminal_size 1.0.0                    py37_2  
beautifulsoup4            4.6.3                    py37_0  
binaryornot               0.4.4                      py_1    conda-forge
bitarray                  0.8.3            py37h14c3975_0  
bkcharts                  0.2                      py37_0  
blas                      1.0                         mkl  
blaze                     0.11.3                   py37_0  
bleach                    2.1.4                    py37_0  
blosc                     1.14.4               hdbcaa40_0  
bokeh                     0.13.0                   py37_0  
boost-cpp                 1.68.0            h11c811c_1000    conda-forge
boto                      2.49.0                   py37_0  
bottleneck                1.2.1            py37h035aef0_1  
bzip2                     1.0.6                h14c3975_5  
ca-certificates           2018.12.5                     0    anaconda
cairo                     1.14.12              h8948797_3  
certifi                   2018.11.29               py37_0    anaconda
cffi                      1.11.5           py37he75722e_1  
chardet                   3.0.4                    py37_1  
click                     6.7                      py37_0  
cloudpickle               0.5.5                    py37_0  
clyent                    1.2.2                    py37_1  
colorama                  0.3.9                    py37_0  
conda                     4.6.2                    py37_0  
conda-build               3.15.1                   py37_0  
conda-env                 2.6.0                         1  
constantly                15.1.0           py37h28b3542_0  
contextlib2               0.5.5                    py37_0  
cookiecutter              1.6.0                 py37_1000    conda-forge
cryptography              2.5              py37hb7f436b_0    conda-forge
cuda100                   1.0                           0    pytorch
cudatoolkit               9.0                  h13b8566_0  
cudf                      0.5.0                   py37_18    rapidsai/label/cuda10.0
cudf-cuda100              0.5.0                    pypi_0    pypi
cudnn                     7.3.1                 cuda9.0_0  
cuml                      0.6.0.dev0+6.gd39cf85          pypi_0    pypi
cuml-cuda100              0.5.0                    pypi_0    pypi
curl                      7.26.0                        1    anaconda
cycler                    0.10.0                   py37_0  
cython                    0.29.2           py37he6710b0_0    anaconda
cytoolz                   0.9.0.1          py37h14c3975_1  
dask                      0.19.1                   py37_0  
dask-core                 0.19.1                   py37_0  
datashape                 0.5.4                    py37_1  
dbus                      1.13.2               h714fa37_1  
decorator                 4.3.0                    py37_0  
defusedxml                0.5.0                    py37_1  
distributed               1.23.1                   py37_0  
docutils                  0.14                     py37_0  
entrypoints               0.2.3                    py37_2  
et_xmlfile                1.0.1                    py37_0  
expat                     2.2.6                he6710b0_0  
faiss-gpu                 1.5.0           py37_cuda10.0_1  [cuda100]  pytorch
fastcache                 1.0.2            py37h14c3975_2  
filelock                  3.0.8                    py37_0  
flask                     1.0.2                    py37_1  
flask-cors                3.0.6                    py37_0  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_1  
fribidi                   1.0.5                h7b6447c_0  
future                    0.17.1                py37_1000    conda-forge
get_terminal_size         1.0.0                haa9412d_0  
gevent                    1.3.6            py37h7b6447c_0  
glib                      2.56.2               hd408876_0  
glob2                     0.6                      py37_0  
gmp                       6.1.2                h6c8ec71_1  
gmpy2                     2.0.8            py37h10f8cd9_2  
graphite2                 1.3.12               h23475e2_2  
greenlet                  0.4.15           py37h7b6447c_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.8.0            py37h989c5e5_3  
harfbuzz                  1.8.8                hffaf4a1_0  
hdf5                      1.10.2               hba1933b_1  
heapdict                  1.0.0                    py37_2  
html5lib                  1.0.1                    py37_0  
hyperlink                 18.0.0                   py37_0  
icu                       58.2                 h9c2bf20_1  
idna                      2.7                      py37_0  
imageio                   2.4.1                    py37_0  
imagesize                 1.1.0                    py37_0  
incremental               17.5.0                   py37_0  
intel-openmp              2019.0                      118  
ipykernel                 4.9.0                    py37_1  
ipython                   6.5.0                    py37_0  
ipython_genutils          0.2.0                    py37_0  
ipywidgets                7.4.1                    py37_0  
isort                     4.3.4                    py37_0  
itsdangerous              0.24                     py37_1  
jbig                      2.1                  hdba287a_0  
jdcal                     1.4                      py37_0  
jedi                      0.12.1                   py37_0  
jeepney                   0.3.1                    py37_0  
jinja2                    2.10                     py37_0  
jinja2-time               0.2.0                      py_2    conda-forge
jpeg                      9b                   h024ee3a_2  
jsonschema                2.6.0                    py37_0  
jupyter                   1.0.0                    py37_7  
jupyter_client            5.2.3                    py37_0  
jupyter_console           5.2.0                    py37_1  
jupyter_core              4.4.0                    py37_0  
jupyterlab                0.34.9                   py37_0  
jupyterlab_launcher       0.13.1                   py37_0  
keyring                   13.2.1                   py37_0  
kiwisolver                1.0.1            py37hf484d3e_0  
krb5                      1.16.2            hc83ff2d_1000    conda-forge
lazy-object-proxy         1.3.1            py37h14c3975_2  
libcudf                   0.5.0               cuda10.0_18    rapidsai/label/cuda10.0
libcudf_cffi              0.5.0           cuda10.0_py37_18    rapidsai/label/cuda10.0
libcurl                   7.63.0            h01ee5af_1000    conda-forge
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libpng                    1.6.35               hbc83047_0    anaconda
libprotobuf               3.6.1             hdbcaa40_1000    conda-forge
libsodium                 1.0.16               h1bed415_0  
libssh2                   1.8.0             h1ad7b7a_1003    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.9                he85c1e1_2  
libtool                   2.4.6                h544aabb_3  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.8                h26e45fe_1  
libxslt                   1.1.32               h1312cb7_0  
llvmlite                  0.27.0           py37hd408876_0  
locket                    0.2.0                    py37_1  
lxml                      4.2.5            py37hefd8a0e_0  
lzo                       2.10                 h49e0be7_2  
markupsafe                1.0              py37h14c3975_1  
matplotlib                2.2.3            py37hb69df0a_0  
mccabe                    0.6.1                    py37_1  
mistune                   0.8.3            py37h14c3975_1  
mkl                       2018.0.3                      1  
mkl-service               1.1.2            py37h90e4bf4_5  
mkl_fft                   1.0.6            py37h7dd41cf_0    anaconda
mkl_random                1.0.1            py37h4414c95_1    anaconda
more-itertools            4.3.0                    py37_0  
mpc                       1.1.0                h10f8cd9_1  
mpfr                      4.0.1                hdf1c602_3  
mpmath                    1.0.0                    py37_2  
msgpack-python            0.5.6            py37h6bb024c_1  
multipledispatch          0.6.0                    py37_0  
navigator-updater         0.2.1                    py37_0  
nbconvert                 5.4.0                    py37_1  
nbformat                  4.4.0                    py37_0  
nccl                      1.3.5                 cuda9.0_0  
ncurses                   6.1                  hf484d3e_0  
networkx                  2.1                      py37_0  
ninja                     1.8.2            py37h6bb024c_1  
nltk                      3.3.0                    py37_0  
nose                      1.3.7                    py37_2  
notebook                  5.6.0                    py37_0  
numba                     0.42.0           py37h962f231_0  
numexpr                   2.6.8            py37hd89afb7_0  
numpy                     1.15.4           py37h1d66e8a_0    anaconda
numpy-base                1.15.4           py37h81de0dd_0    anaconda
numpydoc                  0.8.0                    py37_0  
nvstrings                 0.2.0           cuda10.0_py37_0    nvidia/label/cuda10.0
nvstrings-cuda100         0.2.0.post2              pypi_0    pypi
odo                       0.5.1                    py37_0  
olefile                   0.46                     py37_0  
openpyxl                  2.5.6                    py37_0  
openssl                   1.0.2p               h14c3975_0    anaconda
packaging                 17.1                     py37_0  
pandas                    0.24.0           py37he6710b0_0  
pandoc                    1.19.2.1             hea2e7c5_1  
pandocfilters             1.4.2                    py37_1  
pango                     1.42.4               h049681c_0  
parquet-cpp               1.5.1                         4    conda-forge
parso                     0.3.1                    py37_0  
partd                     0.3.8                    py37_0  
patchelf                  0.9                  hf484d3e_2  
path.py                   11.1.0                   py37_0  
pathlib2                  2.3.2                    py37_0  
patsy                     0.5.0                    py37_0  
pcre                      8.42                 h439df22_0  
pep8                      1.7.1                    py37_0  
pexpect                   4.6.0                    py37_0  
pickleshare               0.7.4                    py37_0  
pillow                    5.2.0            py37heded4f4_0  
pip                       19.0.1                   pypi_0    pypi
pixman                    0.34.0               hceecf20_3  
pkginfo                   1.4.2                    py37_1  
pluggy                    0.7.1            py37h28b3542_0  
ply                       3.11                     py37_0  
poyo                      0.4.2                      py_0    conda-forge
prometheus_client         0.3.1            py37h28b3542_0  
prompt_toolkit            1.0.15                   py37_0  
psutil                    5.4.7            py37h14c3975_0  
ptyprocess                0.6.0                    py37_0  
py                        1.6.0                    py37_0  
pyarrow                   0.11.1                   pypi_0    pypi
pyasn1                    0.4.4            py37h28b3542_0  
pyasn1-modules            0.2.2                    py37_0  
pycodestyle               2.4.0                    py37_0  
pycosat                   0.6.3            py37h14c3975_0  
pycparser                 2.19                     pypi_0    pypi
pycrypto                  2.6.1            py37h14c3975_9  
pycurl                    7.43.0.2         py37hb7f436b_0  
pyflakes                  2.0.0                    py37_0  
pygments                  2.2.0                    py37_0  
pyhamcrest                1.9.0                    pypi_0    pypi
pylint                    2.1.1                    py37_0  
pyodbc                    4.0.24           py37he6710b0_0  
pyopenssl                 18.0.0                   py37_0  
pyparsing                 2.2.0                    py37_1  
pyqt                      5.9.2            py37h05f1152_2  
pysocks                   1.6.8                    py37_0  
pytables                  3.4.4            py37ha205bf6_0  
pytest                    3.8.0                    py37_0  
pytest-arraydiff          0.2              py37h39e3cac_0  
pytest-astropy            0.4.0                    py37_0  
pytest-doctestplus        0.1.3                    py37_0  
pytest-openfiles          0.3.0                    py37_0  
pytest-remotedata         0.3.0                    py37_0  
python                    3.7.1             hd21baee_1000    conda-forge
python-dateutil           2.7.5                    py37_0  
pytorch                   1.0.0           py3.7_cuda10.0.130_cudnn7.4.1_1  [cuda100]  pytorch
pytz                      2018.9                   py37_0  
pywavelets                1.0.0            py37hdd07704_0  
pyyaml                    3.13             py37h14c3975_0  
pyzmq                     17.1.2           py37h14c3975_0  
qt                        5.9.6                h8703b6f_2  
qtawesome                 0.4.4                    py37_0  
qtconsole                 4.4.1                    py37_0  
qtpy                      1.5.0                    py37_0  
readline                  7.0                  h7b6447c_5  
requests                  2.19.1                   py37_0  
rope                      0.11.0                   py37_0  
ruamel_yaml               0.15.46          py37h14c3975_0  
scikit-image              0.14.0           py37hf484d3e_1  
scikit-learn              0.19.2           py37h4989274_0  
scipy                     1.1.0            py37hfa4b5c9_1  
seaborn                   0.9.0                    py37_0  
secretstorage             3.1.0                    py37_0  
send2trash                1.5.0                    py37_0  
service_identity          17.0.0           py37h28b3542_0  
setuptools                40.6.3                   py37_0  
simplegeneric             0.8.1                    py37_2  
singledispatch            3.4.0.3                  py37_0  
sip                       4.19.8           py37hf484d3e_0  
six                       1.12.0                   py37_0  
snappy                    1.1.7                hbae5bb6_3  
snowballstemmer           1.2.1                    py37_0  
sortedcollections         1.0.1                    py37_0  
sortedcontainers          2.0.5                    py37_0  
sphinx                    1.7.9                    py37_0  
sphinxcontrib             1.0                      py37_1  
sphinxcontrib-websupport  1.1.0                    py37_1  
spyder                    3.3.1                    py37_1  
spyder-kernels            0.2.6                    py37_0  
sqlalchemy                1.2.11           py37h7b6447c_0  
sqlite                    3.25.3               h7b6447c_0    anaconda
statsmodels               0.9.0            py37h035aef0_0  
sympy                     1.2                      py37_0  
tblib                     1.3.2                    py37_0  
terminado                 0.8.1                    py37_1  
testpath                  0.3.1                    py37_0  
thrift-cpp                0.12.0            h23e226f_1000    conda-forge
tk                        8.6.8                hbc83047_0  
toolz                     0.9.0                    py37_0  
torchvision               0.2.1                      py_2    pytorch
tornado                   5.1              py37h14c3975_0  
tqdm                      4.26.0           py37h28b3542_0  
traitlets                 4.3.2                    py37_0  
twisted                   18.7.0           py37h14c3975_1  
unicodecsv                0.14.1                   py37_0  
unixodbc                  2.3.7                h14c3975_0  
urllib3                   1.23                     py37_0  
wcwidth                   0.1.7                    py37_0  
webencodings              0.5.1                    py37_1  
werkzeug                  0.14.1                   py37_0  
wheel                     0.32.3                   py37_0  
whichcraft                0.5.2                      py_1    conda-forge
widgetsnbextension        3.4.1                    py37_0  
wrapt                     1.10.11          py37h14c3975_2  
xlrd                      1.1.0                    py37_1  
xlsxwriter                1.1.0                    py37_0  
xlwt                      1.3.0                    py37_0  
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
zeromq                    4.2.5                hf484d3e_1  
zict                      0.1.3                    py37_0  
zlib                      1.2.11               h7b6447c_3    anaconda
zope                      1.0                      py37_1  
zope.interface            4.5.0            py37h14c3975_0  

@daddydrac
Copy link
Author

daddydrac commented Feb 2, 2019

@dantegd - Something doesn't feel right here, thoughts?

cuda100                   1.0                           0    pytorch
cudatoolkit               9.0                  h13b8566_0  
cudf                      0.5.0                   py37_18    rapidsai/label/cuda10.0
cudf-cuda100              0.5.0                    pypi_0    pypi
cudnn                     7.3.1                 cuda9.0_0  
cuml                      0.6.0.dev0+6.gd39cf85          pypi_0    pypi
cuml-cuda100              0.5.0                    pypi_0    pypi

When I am running:

(base) joe@ml-dev-box:~/NVIDIA_CUDA-10.0_Samples $ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

@dantegd
Copy link
Member

dantegd commented Feb 2, 2019

@joehoeller I haven't had time to reproduce yet, but f I'm not mistaken the cudatoolkit installed by conda is the issue (which you well identified). Since it installed the toolkit 9.0 it is conflicting/hiding the system one (10.0), so that is the reason that the needed cuda 10 cublas is not being found. So my first thought is to uninstall the conda cudatoolkit package to solve the issue (might need to rebuild). If that's not enough let us know to look more into it to recreate the environment and see if there are any potential issues in our build instructions/files. Thanks!

@kkraus14
Copy link
Contributor

kkraus14 commented Feb 6, 2019

@joehoeller I haven't had time to reproduce yet, but f I'm not mistaken the cudatoolkit installed by conda is the issue (which you well identified). Since it installed the toolkit 9.0 it is conflicting/hiding the system one (10.0), so that is the reason that the needed cuda 10 cublas is not being found. So my first thought is to uninstall the conda cudatoolkit package to solve the issue (might need to rebuild). If that's not enough let us know to look more into it to recreate the environment and see if there are any potential issues in our build instructions/files. Thanks!

The conda cudatoolkit package doesn't ship the nvcc compiler while it does ship the libraries, so it's likely that while nvcc is reporting CUDA 10 that the libraries being found are from the conda package on CUDA 9.0. @joehoeller it would be great if you could run numba -s and dump the output here for us to confirm.

@fondaing fondaing added this to Needs prioritizing in Other Issue Triage via automation Feb 21, 2019
@robocopnixon robocopnixon moved this from Needs prioritizing to Backlog in Other Issue Triage Feb 22, 2019
@daddydrac
Copy link
Author

daddydrac commented Mar 30, 2019

Yeah, built container here: https://github.com/joehoeller/computer-vision-container/
Still have same problem, also note that all the other frameworks work in the CUDA 10 container.

cc/ @mt-jones per our LinkedIn convo as well

After I launch container and goto demo, I get:


import numpy as np
import pandas as pd
import cudf
import os

from sklearn.neighbors import NearestNeighbors as skKNN
from cuml.neighbors.nearest_neighbors import NearestNeighbors as cumlKNN

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-1-05ab86581f23> in <module>
      1 import numpy as np
      2 import pandas as pd
----> 3 import cudf
      4 import os
      5 

/usr/local/lib/python3.6/dist-packages/cudf/__init__.py in <module>
      1 # Copyright (c) 2018, NVIDIA CORPORATION.
      2 
----> 3 from cudf import dataframe
      4 from cudf import datasets
      5 from cudf.dataframe import DataFrame, from_pandas, merge

/usr/local/lib/python3.6/dist-packages/cudf/dataframe/__init__.py in <module>
----> 1 from cudf.dataframe import (buffer, dataframe, series,
      2     index, numerical, datetime, categorical, string)
      3 
      4 from cudf.dataframe.dataframe import DataFrame, from_pandas, merge
      5 from cudf.dataframe.index import (Index, GenericIndex,

/usr/local/lib/python3.6/dist-packages/cudf/dataframe/buffer.py in <module>
      1 import numpy as np
      2 
----> 3 from librmm_cffi import librmm as rmm
      4 
      5 from cudf.utils import cudautils, utils

/usr/local/lib/python3.6/dist-packages/librmm_cffi/__init__.py in <module>
     43             return path
     44 
---> 45     librmm_api = ffi.dlopen(_get_lib_name())
     46     librmm = _RMMWrapper(ffi, librmm_api)
     47 

OSError: cannot load library 'librmm.so': librmm.so: cannot open shared object file: No such file or directory

@daddydrac
Copy link
Author

daddydrac commented Apr 1, 2019

@kkraus14 ->

`
$ numba -s
System info:

Time Stamp
2019-04-01 13:24:42.697640

Hardware Information
Machine : x86_64
CPU Name : skylake
Number of accessible CPU cores : 12
Listed accessible CPUs cores : 0-11
CFS restrictions : None

OS Information
Platform : Linux-4.18.0-15-generic-x86_64-with-debian-buster-sid
Release : 4.18.0-15-generic
System Name : Linux
Version : #16~18.04.1-Ubuntu SMP Thu Feb 7 14:06:04 UTC 2019
OS specific info : debianbuster/sid
glibc info : glibc 2.10

Python Information
Python Compiler : GCC 7.3.0
Python Implementation : CPython
Python Version : 3.7.1
Python Locale : en_US UTF-8

LLVM information
LLVM version : 6.0.0

CUDA Information
Found 1 CUDA devices
id 0 b'GeForce RTX 2080 Ti' [SUPPORTED]
compute capability: 7.5
pci device id: 0
pci bus id: 1
Summary:
1/1 devices are supported
CUDA driver version : 10010
CUDA libraries:
Finding cublas
ERROR: can't locate lib
Finding cusparse
ERROR: can't locate lib
Finding cufft
ERROR: can't locate lib
Finding curand
ERROR: can't locate lib
Finding nvvm
ERROR: can't locate lib
finding libdevice for compute_20... ERROR: can't open libdevice for compute_20
finding libdevice for compute_30... ERROR: can't open libdevice for compute_30
finding libdevice for compute_35... ERROR: can't open libdevice for compute_35
finding libdevice for compute_50... ERROR: can't open libdevice for compute_50

ROC Information
ROC available : False
Error initialising ROC due to : No ROC toolchains found.
No HSA Agents found, encountered exception when searching:
Error at driver init:
NUMBA_HSA_DRIVER /opt/rocm/lib/libhsa-runtime64.so is not a valid file path. Note it must be a filepath of the .so/.dll/.dylib or the driver:

SVML Information
SVML state, config.USING_SVML : False
SVML library found and loaded : False
llvmlite using SVML patched LLVM : True
SVML operational : False

Threading Layer Information
TBB Threading layer available : False
+--> Disabled due to : Unknown import problem.
OpenMP Threading layer available : True
Workqueue Threading layer available : True

Numba Environment Variable Information
None set.

Conda Information
conda_build_version : 3.17.6
conda_env_version : 4.5.12
platform : linux-64
python_version : 3.7.1.final.0
root_writable : False

Current Conda Env
_ipyw_jlab_nb_ext_conf 0.1.0 py37_0
alabaster 0.7.12 py37_0
anaconda 2018.12 py37_0
anaconda-client 1.7.2 py37_0
anaconda-navigator 1.9.6 py37_0
anaconda-project 0.8.2 py37_0
asn1crypto 0.24.0 py37_0
astroid 2.1.0 py37_0
astropy 3.1 py37h7b6447c_0
atomicwrites 1.2.1 py37_0
attrs 18.2.0 py37h28b3542_0
babel 2.6.0 py37_0
backcall 0.1.0 py37_0
backports 1.0 py37_1
backports.os 0.1.1 py37_0
backports.shutil_get_terminal_size 1.0.0 py37_2
beautifulsoup4 4.6.3 py37_0
bitarray 0.8.3 py37h14c3975_0
bkcharts 0.2 py37_0
blas 1.0 mkl
blaze 0.11.3 py37_0
bleach 3.0.2 py37_0
blosc 1.14.4 hdbcaa40_0
bokeh 1.0.2 py37_0
boto 2.49.0 py37_0
bottleneck 1.2.1 py37h035aef0_1
bzip2 1.0.6 h14c3975_5
ca-certificates 2018.03.07 0
cairo 1.14.12 h8948797_3
certifi 2018.11.29 py37_0
cffi 1.11.5 py37he75722e_1
chardet 3.0.4 py37_1
click 7.0 py37_0
cloudpickle 0.6.1 py37_0
clyent 1.2.2 py37_1
colorama 0.4.1 py37_0
conda 4.5.12 py37_0
conda-build 3.17.6 py37_0
conda-env 2.6.0 1
conda-verify 3.1.1 py37_0
contextlib2 0.5.5 py37_0
cryptography 2.4.2 py37h1ba5d50_0
curl 7.63.0 hbc83047_1000
cycler 0.10.0 py37_0
cython 0.29.2 py37he6710b0_0
cytoolz 0.9.0.1 py37h14c3975_1
dask 1.0.0 py37_0
dask-core 1.0.0 py37_0
datashape 0.5.4 py37_1
dbus 1.13.2 h714fa37_1
decorator 4.3.0 py37_0
defusedxml 0.5.0 py37_1
distributed 1.25.1 py37_0
docutils 0.14 py37_0
entrypoints 0.2.3 py37_2
et_xmlfile 1.0.1 py37_0
expat 2.2.6 he6710b0_0
fastcache 1.0.2 py37h14c3975_2
filelock 3.0.10 py37_0
flask 1.0.2 py37_1
flask-cors 3.0.7 py37_0
fontconfig 2.13.0 h9420a91_0
freetype 2.9.1 h8a8886c_1
fribidi 1.0.5 h7b6447c_0
future 0.17.1 py37_0
get_terminal_size 1.0.0 haa9412d_0
gevent 1.3.7 py37h7b6447c_1
glib 2.56.2 hd408876_0
glob2 0.6 py37_1
gmp 6.1.2 h6c8ec71_1
gmpy2 2.0.8 py37h10f8cd9_2
graphite2 1.3.12 h23475e2_2
greenlet 0.4.15 py37h7b6447c_0
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb453b48_1
h5py 2.8.0 py37h989c5e5_3
harfbuzz 1.8.8 hffaf4a1_0
hdf5 1.10.2 hba1933b_1
heapdict 1.0.0 py37_2
html5lib 1.0.1 py37_0
icu 58.2 h9c2bf20_1
idna 2.8 py37_0
imageio 2.4.1 py37_0
imagesize 1.1.0 py37_0
importlib_metadata 0.6 py37_0
intel-openmp 2019.1 144
ipykernel 5.1.0 py37h39e3cac_0
ipython 7.2.0 py37h39e3cac_0
ipython_genutils 0.2.0 py37_0
ipywidgets 7.4.2 py37_0
isort 4.3.4 py37_0
itsdangerous 1.1.0 py37_0
jbig 2.1 hdba287a_0
jdcal 1.4 py37_0
jedi 0.13.2 py37_0
jeepney 0.4 py37_0
jinja2 2.10 py37_0
jpeg 9b h024ee3a_2
jsonschema 2.6.0 py37_0
jupyter 1.0.0 py37_7
jupyter_client 5.2.4 py37_0
jupyter_console 6.0.0 py37_0
jupyter_core 4.4.0 py37_0
jupyterlab 0.35.3 py37_0
jupyterlab_server 0.2.0 py37_0
keyring 17.0.0 py37_0
kiwisolver 1.0.1 py37hf484d3e_0
krb5 1.16.1 h173b8e3_7
lazy-object-proxy 1.3.1 py37h14c3975_2
libarchive 3.3.3 h5d8350f_5
libcurl 7.63.0 h20c2e04_1000
libedit 3.1.20170329 h6b74fdf_2
libffi 3.2.1 hd88cf55_4
libgcc-ng 8.2.0 hdf63c60_1
libgfortran-ng 7.3.0 hdf63c60_0
liblief 0.9.0 h7725739_1
libpng 1.6.35 hbc83047_0
libsodium 1.0.16 h1bed415_0
libssh2 1.8.0 h1ba5d50_4
libstdcxx-ng 8.2.0 hdf63c60_1
libtiff 4.0.9 he85c1e1_2
libtool 2.4.6 h7b6447c_5
libuuid 1.0.3 h1bed415_2
libxcb 1.13 h1bed415_1
libxml2 2.9.8 h26e45fe_1
libxslt 1.1.32 h1312cb7_0
llvmlite 0.26.0 py37hd408876_0
locket 0.2.0 py37_1
lxml 4.2.5 py37hefd8a0e_0
lz4-c 1.8.1.2 h14c3975_0
lzo 2.10 h49e0be7_2
markupsafe 1.1.0 py37h7b6447c_0
matplotlib 3.0.2 py37h5429711_0
mccabe 0.6.1 py37_1
mistune 0.8.4 py37h7b6447c_0
mkl 2019.1 144
mkl-service 1.1.2 py37he904b0f_5
mkl_fft 1.0.6 py37hd81dba3_0
mkl_random 1.0.2 py37hd81dba3_0
more-itertools 4.3.0 py37_0
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.1 hdf1c602_3
mpmath 1.1.0 py37_0
msgpack-python 0.5.6 py37h6bb024c_1
multipledispatch 0.6.0 py37_0
navigator-updater 0.2.1 py37_0
nbconvert 5.4.0 py37_1
nbformat 4.4.0 py37_0
ncurses 6.1 he6710b0_1
networkx 2.2 py37_1
nltk 3.4 py37_1
nose 1.3.7 py37_2
notebook 5.7.4 py37_0
numba 0.41.0 py37h962f231_0
numexpr 2.6.8 py37h9e4a6bb_0
numpy 1.15.4 py37h7e9f1db_0
numpy-base 1.15.4 py37hde5b4d6_0
numpydoc 0.8.0 py37_0
odo 0.5.1 py37_0
olefile 0.46 py37_0
openpyxl 2.5.12 py37_0
openssl 1.1.1a h7b6447c_0
packaging 18.0 py37_0
pandas 0.23.4 py37h04863e7_0
pandoc 1.19.2.1 hea2e7c5_1
pandocfilters 1.4.2 py37_1
pango 1.42.4 h049681c_0
parso 0.3.1 py37_0
partd 0.3.9 py37_0
patchelf 0.9 he6710b0_3
path.py 11.5.0 py37_0
pathlib2 2.3.3 py37_0
patsy 0.5.1 py37_0
pcre 8.42 h439df22_0
pep8 1.7.1 py37_0
pexpect 4.6.0 py37_0
pickleshare 0.7.5 py37_0
pillow 5.3.0 py37h34e0f95_0
pip 18.1 py37_0
pixman 0.34.0 hceecf20_3
pkginfo 1.4.2 py37_1
pluggy 0.8.0 py37_0
ply 3.11 py37_0
prometheus_client 0.5.0 py37_0
prompt_toolkit 2.0.7 py37_0
psutil 5.4.8 py37h7b6447c_0
ptyprocess 0.6.0 py37_0
py 1.7.0 py37_0
py-lief 0.9.0 py37h7725739_1
pycodestyle 2.4.0 py37_0
pycosat 0.6.3 py37h14c3975_0
pycparser 2.19 py37_0
pycrypto 2.6.1 py37h14c3975_9
pycurl 7.43.0.2 py37h1ba5d50_0
pyflakes 2.0.0 py37_0
pygments 2.3.1 py37_0
pylint 2.2.2 py37_0
pyodbc 4.0.25 py37he6710b0_0
pyopenssl 18.0.0 py37_0
pyparsing 2.3.0 py37_0
pyqt 5.9.2 py37h05f1152_2
pysocks 1.6.8 py37_0
pytables 3.4.4 py37ha205bf6_0
pytest 4.0.2 py37_0
pytest-arraydiff 0.3 py37h39e3cac_0
pytest-astropy 0.5.0 py37_0
pytest-doctestplus 0.2.0 py37_0
pytest-openfiles 0.3.1 py37_0
pytest-remotedata 0.3.1 py37_0
python 3.7.1 h0371630_7
python-dateutil 2.7.5 py37_0
python-libarchive-c 2.8 py37_6
pytz 2018.7 py37_0
pywavelets 1.0.1 py37hdd07704_0
pyyaml 3.13 py37h14c3975_0
pyzmq 17.1.2 py37h14c3975_0
qt 5.9.7 h5867ecd_1
qtawesome 0.5.3 py37_0
qtconsole 4.4.3 py37_0
qtpy 1.5.2 py37_0
readline 7.0 h7b6447c_5
requests 2.21.0 py37_0
rope 0.11.0 py37_0
ruamel_yaml 0.15.46 py37h14c3975_0
scikit-image 0.14.1 py37he6710b0_0
scikit-learn 0.20.1 py37hd81dba3_0
scipy 1.1.0 py37h7c811a0_2
seaborn 0.9.0 py37_0
secretstorage 3.1.0 py37_0
send2trash 1.5.0 py37_0
setuptools 40.6.3 py37_0
simplegeneric 0.8.1 py37_2
singledispatch 3.4.0.3 py37_0
sip 4.19.8 py37hf484d3e_0
six 1.12.0 py37_0
snappy 1.1.7 hbae5bb6_3
snowballstemmer 1.2.1 py37_0
sortedcollections 1.0.1 py37_0
sortedcontainers 2.1.0 py37_0
sphinx 1.8.2 py37_0
sphinxcontrib 1.0 py37_1
sphinxcontrib-websupport 1.1.0 py37_1
spyder 3.3.2 py37_0
spyder-kernels 0.3.0 py37_0
sqlalchemy 1.2.15 py37h7b6447c_0
sqlite 3.26.0 h7b6447c_0
statsmodels 0.9.0 py37h035aef0_0
sympy 1.3 py37_0
tblib 1.3.2 py37_0
terminado 0.8.1 py37_1
testpath 0.4.2 py37_0
tk 8.6.8 hbc83047_0
toolz 0.9.0 py37_0
tornado 5.1.1 py37h7b6447c_0
tqdm 4.28.1 py37h28b3542_0
traitlets 4.3.2 py37_0
unicodecsv 0.14.1 py37_0
unixodbc 2.3.7 h14c3975_0
urllib3 1.24.1 py37_0
wcwidth 0.1.7 py37_0
webencodings 0.5.1 py37_1
werkzeug 0.14.1 py37_0
wheel 0.32.3 py37_0
widgetsnbextension 3.4.2 py37_0
wrapt 1.10.11 py37h14c3975_2
wurlitzer 1.0.2 py37_0
xlrd 1.2.0 py37_0
xlsxwriter 1.1.2 py37_0
xlwt 1.3.0 py37_0
xz 5.2.4 h14c3975_4
yaml 0.1.7 had09818_2
zeromq 4.2.5 hf484d3e_1
zict 0.1.3 py37_0
zlib 1.2.11 h7b6447c_3
zstd 1.3.7 h0b5b093_0

`

@cjnolet cjnolet closed this as completed May 4, 2019
Other Issue Triage automation moved this from Backlog to Closed May 4, 2019
Salonijain27 pushed a commit to Salonijain27/cuml that referenced this issue Jan 22, 2020
[REVIEW] Move RF Demo from notebooks to notebooks-extended
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0 - Backlog In queue waiting for assignment doc Documentation help wanted Extra attention is needed
Projects
Development

No branches or pull requests

6 participants