[BUG] PCA doesn't work with sparse matrices #5475

Intron7 · 2023-06-22T11:57:23Z

Describe the bug
The PCA function doesn't work with sparse matrices.
Steps/Code to reproduce bug

import cupy as cp 
from cuml.decomposition import PCA
from cupyx.scipy import sparse
from cuml.testing.utils import array_equal

X = sparse.random(100000,5000,density=0.05,format="csr",dtype=cp.float32, random_state=0)

pca_func = PCA(
    n_components=50, random_state=0, output_type="cupy"
)
X_pca = pca_func.fit_transform(X)

X_dense = X.toarray()
pca_func_dense = PCA(
    n_components=50, random_state=0, output_type="cupy"
)
X_pca_dense = pca_func_dense.fit_transform(X_dense)

assert array_equal(X_pca_dense, X_pca, 1e-3, with_sign=False)

Output

Traceback (most recent call last):
  File "cupy_backends/cuda/libs/cusparse.pyx", line 1544, in cupy_backends.cuda.libs.cusparse.check_status
  File "cupy_backends/cuda/libs/cusparse.pyx", line 1534, in cupy_backends.cuda.libs.cusparse.CuSparseError.__init__
KeyError: 11
Exception ignored in: 'cupy_backends.cuda.libs.cusparse.spGEMM_workEstimation'
Traceback (most recent call last):
  File "cupy_backends/cuda/libs/cusparse.pyx", line 1544, in cupy_backends.cuda.libs.cusparse.check_status
  File "cupy_backends/cuda/libs/cusparse.pyx", line 1534, in cupy_backends.cuda.libs.cusparse.CuSparseError.__init__
KeyError: 11
Traceback (most recent call last):
  File "/sc-scratch/sc-scratch-rosen-neurogenetics/git/rapids_singlecell_tester/notebooks/pca_sparse.py", line 19, in <module>
    assert array_equal(X_pca_dense, X_pca, 1e-3, with_sign=False)
AssertionError

Expected behavior
To run PCA with sparse matrix without the error and get the same result.

Environment details (please complete the following information):

Click here to see environment details


Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
Linux Distro/Architecture: [Ubuntu 20.04 amd64, Rocky Linux 8.5 amd 64]
GPU Model/Driver: [A100 driver 525.78.01 & A100 470.161.03]
CUDA: [12, 11.4]
Method of cuDF & cuML install: [conda]

name: rapids-23.06

channels:


nvidia
rapidsai
conda-forge
defaults

dependencies:
_libgcc_mutex=0.1=conda_forge
_openmp_mutex=4.5=2_gnu
aiohttp=3.8.4=py310h2372a71_1
aiosignal=1.3.1=pyhd8ed1ab_0
anyio=3.7.0=pyhd8ed1ab_1
aom=3.5.0=h27087fc_0
appdirs=1.4.4=pyh9f0ad1d_0
argon2-cffi=21.3.0=pyhd8ed1ab_0
argon2-cffi-bindings=21.2.0=py310h5764c6d_3
asttokens=2.2.1=pyhd8ed1ab_0
async-timeout=4.0.2=pyhd8ed1ab_0
attrs=23.1.0=pyh71513ae_1
aws-c-auth=0.6.28=hccec9ca_5
aws-c-cal=0.5.27=hf85dbcb_0
aws-c-common=0.8.20=hd590300_0
aws-c-compression=0.2.17=h4b87b72_0
aws-c-event-stream=0.3.0=hc5de78f_6
aws-c-http=0.7.8=h412fb1b_4
aws-c-io=0.13.26=h0d05201_0
aws-c-mqtt=0.8.13=ha5d9b87_2
aws-c-s3=0.3.4=h95e21fb_5
aws-c-sdkutils=0.1.10=h4b87b72_0
aws-checksums=0.1.16=h4b87b72_0
aws-crt-cpp=0.20.2=h5289e1f_9
aws-sdk-cpp=1.10.57=h8101662_14
backcall=0.2.0=pyh9f0ad1d_0
backports=1.0=pyhd8ed1ab_3
backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
beautifulsoup4=4.12.2=pyha770c72_0
bleach=6.0.0=pyhd8ed1ab_0
blosc=1.21.4=h0f2a231_0
bokeh=2.4.3=pyhd8ed1ab_3
boost-cpp=1.78.0=h5adbc97_2
branca=0.6.0=pyhd8ed1ab_0
brotli=1.0.9=h166bdaf_8
brotli-bin=1.0.9=h166bdaf_8
brotlipy=0.7.0=py310h5764c6d_1005
brunsli=0.1=h9c3ff4c_0
bzip2=1.0.8=h7f98852_4
c-ares=1.19.1=hd590300_0
c-blosc2=2.9.2=hb4ffafa_0
ca-certificates=2023.5.7=hbcca054_0
cachetools=5.3.0=pyhd8ed1ab_0
cairo=1.16.0=ha61ee94_1014
certifi=2023.5.7=pyhd8ed1ab_0
cffi=1.15.1=py310h255011f_3
cfitsio=4.2.0=hd9d235c_0
charls=2.4.2=h59595ed_0
charset-normalizer=3.1.0=pyhd8ed1ab_0
click=8.1.3=unix_pyhd8ed1ab_2
click-plugins=1.1.1=py_0
cligj=0.7.2=pyhd8ed1ab_1
cloudpickle=2.2.1=pyhd8ed1ab_0
colorama=0.4.6=pyhd8ed1ab_0
colorcet=3.0.1=pyhd8ed1ab_0
comm=0.1.3=pyhd8ed1ab_0
contourpy=1.1.0=py310hd41b1e2_0
cryptography=41.0.1=py310h75e40e8_0
cubinlinker=0.3.0=py310hfdf336d_0
cucim=23.06.00=cuda11_py310_230607_gfdc657b_0
cuda-profiler-api=11.8.86=0
cuda-python=11.8.2=py310h01a121a_0
cuda-version=11.8=h70ddcb2_2
cudatoolkit=11.8.0=h37601d7_11
cudf=23.06.00=cuda11_py310_230607_gf881d40c63_0
cudf_kafka=23.06.00=py310_230607_gf881d40c63_0
cugraph=23.06.02=cuda11_py310_230613_gdb9d3c12_0
cuml=23.06.00=cuda11_py310_230607_ga381e03f2_0
cupy=12.0.0=py310h9216885_4
curl=8.1.2=h409715c_0
cusignal=23.06.00=py39_230607_g22c7120_0
cuspatial=23.06.00=py310_230607_g7b3284af_0
custreamz=23.06.00=py310_230607_gf881d40c63_0
cuxfilter=23.06.00=py310_230607_g862c7d1_0
cycler=0.11.0=pyhd8ed1ab_0
cyrus-sasl=2.1.27=h9033bb2_6
cytoolz=0.12.0=py310h5764c6d_1
dask=2023.3.2=pyhd8ed1ab_0
dask-core=2023.3.2=pyhd8ed1ab_0
dask-cuda=23.06.00=py310_230607_gfd3ab2d_0
dask-cudf=23.06.00=cuda11_py310_230607_gf881d40c63_0
datashader=0.15.0=pyhd8ed1ab_0
datashape=0.5.4=py_1
dav1d=1.2.1=hd590300_0
debugpy=1.6.7=py310heca2aa9_0
decorator=5.1.1=pyhd8ed1ab_0
defusedxml=0.7.1=pyhd8ed1ab_0
distributed=2023.3.2.1=pyhd8ed1ab_0
dlpack=0.5=h9c3ff4c_0
entrypoints=0.4=pyhd8ed1ab_0
exceptiongroup=1.1.1=pyhd8ed1ab_0
executing=1.2.0=pyhd8ed1ab_0
expat=2.5.0=hcb278e6_1
fastrlock=0.8=py310hd8f1fbe_3
fiona=1.9.1=py310ha325b7b_0
flit-core=3.9.0=pyhd8ed1ab_0
fmt=9.1.0=h924138e_0
folium=0.14.0=pyhd8ed1ab_0
font-ttf-dejavu-sans-mono=2.37=hab24e00_0
font-ttf-inconsolata=3.000=h77eed37_0
font-ttf-source-code-pro=2.038=h77eed37_0
font-ttf-ubuntu=0.83=hab24e00_0
fontconfig=2.14.2=h14ed4e7_0
fonts-conda-ecosystem=1=0
fonts-conda-forge=1=0
fonttools=4.40.0=py310h2372a71_0
freetype=2.12.1=hca18f0e_1
freexl=1.0.6=h166bdaf_1
frozenlist=1.3.3=py310h5764c6d_0
fsspec=2023.6.0=pyh1a96a4e_0
gdal=3.6.2=py310hc1b7723_3
gdk-pixbuf=2.42.10=h05c8ddd_0
geopandas=0.13.2=pyhd8ed1ab_1
geopandas-base=0.13.2=pyha770c72_1
geos=3.11.1=h27087fc_0
geotiff=1.7.1=h7157cca_5
gettext=0.21.1=h27087fc_0
gflags=2.2.2=he1b5a44_1004
giflib=5.2.1=h0b41bf4_3
glib=2.76.3=hfc55251_0
glib-tools=2.76.3=hfc55251_0
glog=0.6.0=h6f12383_0
gmock=1.13.0=ha770c72_1
gtest=1.13.0=h00ab1b0_1
hdf4=4.2.15=h9772cbc_5
hdf5=1.12.2=nompi_h4df4325_101
holoviews=1.15.4=pyhd8ed1ab_0
icu=70.1=h27087fc_0
idna=3.4=pyhd8ed1ab_0
imagecodecs=2023.1.23=py310ha3ed6a1_0
imageio=2.31.1=pyh24c5eb1_0
importlib-metadata=6.7.0=pyha770c72_0
importlib_metadata=6.7.0=hd8ed1ab_0
importlib_resources=5.12.0=pyhd8ed1ab_0
ipykernel=6.23.2=pyh210e3f2_0
ipython=8.14.0=pyh41d4057_0
ipywidgets=8.0.6=pyhd8ed1ab_0
jbig=2.1=h7f98852_2003
jedi=0.18.2=pyhd8ed1ab_0
jinja2=3.1.2=pyhd8ed1ab_1
joblib=1.2.0=pyhd8ed1ab_0
jpeg=9e=h0b41bf4_3
json-c=0.16=hc379101_0
jsonschema=4.17.3=pyhd8ed1ab_0
jupyter-server-proxy=4.0.0=pyhd8ed1ab_0
jupyter_client=8.2.0=pyhd8ed1ab_0
jupyter_core=5.3.1=py310hff52083_0
jupyter_events=0.6.3=pyhd8ed1ab_0
jupyter_server=2.6.0=pyhd8ed1ab_0
jupyter_server_terminals=0.4.4=pyhd8ed1ab_1
jupyterlab_pygments=0.2.2=pyhd8ed1ab_0
jupyterlab_widgets=3.0.7=pyhd8ed1ab_1
jxrlib=1.1=h7f98852_2
kealib=1.5.0=ha7026e8_0
keyutils=1.6.1=h166bdaf_0
kiwisolver=1.4.4=py310hbf28c38_1
krb5=1.20.1=h81ceb04_0
lazy_loader=0.2=pyhd8ed1ab_0
lcms2=2.15=hfd0df8a_0
ld_impl_linux-64=2.40=h41732ed_0
lerc=4.0.0=h27087fc_0
libabseil=20230125.2=cxx17_h59595ed_2
libaec=1.0.6=hcb278e6_1
libarrow=11.0.0=hc00ebf5_25_cpu
libavif=0.11.1=h8182462_2
libblas=3.9.0=17_linux64_openblas
libbrotlicommon=1.0.9=h166bdaf_8
libbrotlidec=1.0.9=h166bdaf_8
libbrotlienc=1.0.9=h166bdaf_8
libcblas=3.9.0=17_linux64_openblas
libcrc32c=1.1.2=h9c3ff4c_0
libcublas=11.11.3.6=0
libcublas-dev=11.11.3.6=0
libcucim=23.06.00=cuda11_230607_gfdc657b_0
libcudf=23.06.00=cuda11_230607_gf881d40c63_0
libcudf_kafka=23.06.00=230607_gf881d40c63_0
libcufft=10.9.0.58=0
libcufile=1.4.0.31=0
libcufile-dev=1.4.0.31=0
libcugraph=23.06.02=cuda11_230613_gdb9d3c12_0
libcugraph_etl=23.06.02=cuda11_230613_gdb9d3c12_0
libcugraphops=23.06.00=cuda11_230607_g77d012ac_0
libcuml=23.06.00=cuda11_230607_ga381e03f2_0
libcumlprims=23.06.00=cuda11_230607_g7081940_0
libcurand=10.3.0.86=0
libcurand-dev=10.3.0.86=0
libcurl=8.1.2=h409715c_0
libcusolver=11.4.1.48=0
libcusolver-dev=11.4.1.48=0
libcusparse=11.7.5.86=0
libcusparse-dev=11.7.5.86=0
libcuspatial=23.06.00=cuda11_230607_g7b3284af_0
libdeflate=1.17=h0b41bf4_0
libedit=3.1.20191231=he28a2e2_2
libev=4.33=h516909a_1
libevent=2.1.12=hf998b51_1
libexpat=2.5.0=hcb278e6_1
libffi=3.4.2=h7f98852_5
libgcc-ng=13.1.0=he5830b7_0
libgdal=3.6.2=h10cbb15_3
libgfortran-ng=13.1.0=h69a702a_0
libgfortran5=13.1.0=h15d22d2_0
libglib=2.76.3=hebfc3b9_0
libgomp=13.1.0=he5830b7_0
libgoogle-cloud=2.11.0=hac9eb74_1
libgrpc=1.54.2=hb20ce57_2
libiconv=1.17=h166bdaf_0
libkml=1.3.0=h37653c0_1015
libkvikio=23.06.00=cuda11_230607_gd3b823c_0
liblapack=3.9.0=17_linux64_openblas
libllvm14=14.0.6=hcd5def8_3
libnetcdf=4.8.1=nompi_h261ec11_106
libnghttp2=1.52.0=h61bc06f_0
libnsl=2.0.0=h7f98852_0
libntlm=1.4=h7f98852_1002
libnuma=2.0.16=h0b41bf4_1
libopenblas=0.3.23=pthreads_h80387f5_0
libpng=1.6.39=h753d276_0
libpq=15.2=hb675445_0
libprotobuf=3.21.12=h3eb15da_0
libraft=23.06.01=cuda11_230612_g9147c907_0
libraft-headers=23.06.01=cuda11_230612_g9147c907_0
libraft-headers-only=23.06.01=cuda11_230612_g9147c907_0
librdkafka=1.9.2=ha5a0de0_2
librmm=23.06.00=cuda11_230607_gacaf3f5e_0
librttopo=1.1.0=ha49c73b_12
libsodium=1.0.18=h36c2ea0_1
libspatialindex=1.9.3=h9c3ff4c_4
libspatialite=5.0.1=h7c8129e_22
libsqlite=3.42.0=h2797004_0
libssh2=1.11.0=h0841786_0
libstdcxx-ng=13.1.0=hfd8a6a1_0
libthrift=0.18.1=h8fd135c_2
libtiff=4.5.0=h6adf6a1_2
libutf8proc=2.8.0=h166bdaf_0
libuuid=2.38.1=h0b41bf4_0
libuv=1.44.2=h166bdaf_0
libwebp=1.2.4=h1daa5a0_1
libwebp-base=1.2.4=h166bdaf_0
libxcb=1.13=h7f98852_1004
libxgboost=1.7.5dev.rapidsai23.06=cuda11_0
libxml2=2.10.3=hca2bb57_4
libzip=1.9.2=hc929e4a_1
libzlib=1.2.13=hd590300_5
libzopfli=1.0.3=h9c3ff4c_0
llvmlite=0.40.0=py310h1b8f574_0
locket=1.0.0=pyhd8ed1ab_0
lz4=4.3.2=py310h0cfdcf0_0
lz4-c=1.9.4=hcb278e6_0
mapclassify=2.5.0=pyhd8ed1ab_1
markdown=3.4.3=pyhd8ed1ab_0
markupsafe=2.1.3=py310h2372a71_0
matplotlib-base=3.7.1=py310he60537e_0
matplotlib-inline=0.1.6=pyhd8ed1ab_0
mistune=3.0.0=pyhd8ed1ab_0
msgpack-python=1.0.5=py310hdf3cbec_0
multidict=6.0.4=py310h1fa729e_0
multipledispatch=0.6.0=py_0
munch=3.0.0=pyhd8ed1ab_0
munkres=1.1.4=pyh9f0ad1d_0
nbclient=0.8.0=pyhd8ed1ab_0
nbconvert-core=7.6.0=pyhd8ed1ab_0
nbformat=5.9.0=pyhd8ed1ab_0
nccl=2.18.3.1=h12f7317_0
ncurses=6.4=hcb278e6_0
nest-asyncio=1.5.6=pyhd8ed1ab_0
networkx=3.1=pyhd8ed1ab_0
nodejs=18.15.0=h8d033a5_0
nspr=4.35=h27087fc_0
nss=3.89=he45b914_0
numba=0.57.0=py310h0f6aa51_2
numpy=1.24.3=py310ha4c1d20_0
nvtx=0.2.5=py310h1fa729e_0
openjpeg=2.5.0=hfec8fc6_2
openslide=3.4.1=h7773abc_6
openssl=3.1.1=hd590300_1
orc=1.8.4=h2f23424_0
overrides=7.3.1=pyhd8ed1ab_0
packaging=23.1=pyhd8ed1ab_0
pandas=1.5.3=py310h9b08913_1
pandocfilters=1.5.0=pyhd8ed1ab_0
panel=0.14.1=pyhd8ed1ab_0
param=1.13.0=pyh1a96a4e_0
parso=0.8.3=pyhd8ed1ab_0
partd=1.4.0=pyhd8ed1ab_0
pcre2=10.40=hc3806b6_0
pexpect=4.8.0=pyh1a96a4e_2
pickleshare=0.7.5=py_1003
pillow=9.4.0=py310h023d228_1
pip=23.1.2=pyhd8ed1ab_0
pixman=0.40.0=h36c2ea0_0
pkgutil-resolve-name=1.3.10=pyhd8ed1ab_0
platformdirs=3.6.0=pyhd8ed1ab_0
pooch=1.7.0=pyha770c72_3
poppler=22.12.0=h091648b_1
poppler-data=0.4.12=hd8ed1ab_0
postgresql=15.2=h3248436_0
proj=9.1.0=h8ffa02c_1
prometheus_client=0.17.0=pyhd8ed1ab_0
prompt-toolkit=3.0.38=pyha770c72_0
prompt_toolkit=3.0.38=hd8ed1ab_0
protobuf=4.21.12=py310heca2aa9_0
psutil=5.9.5=py310h1fa729e_0
pthread-stubs=0.4=h36c2ea0_1001
ptxcompiler=0.8.1=py310h01a121a_0
ptyprocess=0.7.0=pyhd3deb0d_0
pure_eval=0.2.2=pyhd8ed1ab_0
py-xgboost=1.7.5dev.rapidsai23.06=cuda11_py310_0
pyarrow=11.0.0=py310he6bfd7f_25_cpu
pycparser=2.21=pyhd8ed1ab_0
pyct=0.4.6=py_0
pyct-core=0.4.6=py_0
pydeck=0.5.0=pyh9f0ad1d_0
pyee=8.1.0=pyhd8ed1ab_0
pygments=2.15.1=pyhd8ed1ab_0
pylibcugraph=23.06.02=cuda11_py310_230613_gdb9d3c12_0
pylibraft=23.06.01=cuda11_py310_230612_g9147c907_0
pynvml=11.4.1=pyhd8ed1ab_0
pyopenssl=23.2.0=pyhd8ed1ab_1
pyparsing=3.1.0=pyhd8ed1ab_0
pyppeteer=1.0.2=pyhd8ed1ab_0
pyproj=3.4.0=py310hb1338dc_2
pyrsistent=0.19.3=py310h1fa729e_0
pysocks=1.7.1=pyha2e5f31_6
python=3.10.11=he550d4f_0_cpython
python-confluent-kafka=1.9.2=py310h5764c6d_2
python-dateutil=2.8.2=pyhd8ed1ab_0
python-fastjsonschema=2.17.1=pyhd8ed1ab_0
python-json-logger=2.0.7=pyhd8ed1ab_0
python_abi=3.10=3_cp310
pytz=2023.3=pyhd8ed1ab_0
pyviz_comms=2.3.2=pyhd8ed1ab_0
pywavelets=1.4.1=py310h0a54255_0
pyyaml=6.0=py310h5764c6d_5
pyzmq=25.1.0=py310h5bbb5d0_0
raft-dask=23.06.01=cuda11_py310_230612_g9147c907_0
rapids=23.06.02=cuda11_py310_230613_g9b052fc_0
rapids-xgboost=23.06.02=cuda11_py310_230613_g9b052fc_0
rdma-core=28.9=h59595ed_1
re2=2023.03.02=h8c504da_0
readline=8.2=h8228510_1
requests=2.31.0=pyhd8ed1ab_0
rfc3339-validator=0.1.4=pyhd8ed1ab_0
rfc3986-validator=0.1.1=pyh9f0ad1d_0
rmm=23.06.00=cuda11_py310_230607_gacaf3f5e_0
rtree=1.0.1=py310hbdcdc62_1
s2n=1.3.45=h06160fa_0
scikit-image=0.20.0=py310h9b08913_1
scikit-learn=1.2.2=py310hf7d194e_2
scipy=1.10.1=py310ha4c1d20_3
send2trash=1.8.2=pyh41d4057_0
setuptools=67.7.2=pyhd8ed1ab_0
shapely=2.0.1=py310h8b84c32_0
simpervisor=1.0.0=pyhd8ed1ab_0
six=1.16.0=pyh6c4a22f_0
snappy=1.1.10=h9fff704_0
sniffio=1.3.0=pyhd8ed1ab_0
sortedcontainers=2.4.0=pyhd8ed1ab_0
soupsieve=2.3.2.post1=pyhd8ed1ab_0
spdlog=1.11.0=h9b3ece8_1
sqlite=3.42.0=h2c6b66d_0
stack_data=0.6.2=pyhd8ed1ab_0
streamz=0.6.4=pyh6c4a22f_0
tblib=1.7.0=pyhd8ed1ab_0
terminado=0.17.1=pyh41d4057_0
threadpoolctl=3.1.0=pyh8a188c0_0
tifffile=2023.4.12=pyhd8ed1ab_0
tiledb=2.13.2=hd532e3d_0
tinycss2=1.2.1=pyhd8ed1ab_0
tk=8.6.12=h27826a3_0
toolz=0.12.0=pyhd8ed1ab_0
tornado=6.3.2=py310h2372a71_0
tqdm=4.65.0=pyhd8ed1ab_1
traitlets=5.9.0=pyhd8ed1ab_0
treelite=3.2.0=py310h1be96d9_0
typing-extensions=4.6.3=hd8ed1ab_0
typing_extensions=4.6.3=pyha770c72_0
typing_utils=0.1.0=pyhd8ed1ab_0
tzcode=2023c=h0b41bf4_0
tzdata=2023c=h71feb2d_0
ucx=1.14.1=h4a2ce2d_2
ucx-proc=1.0.0=gpu
ucx-py=0.32.00=py310_230607_gded9ea2_0
unicodedata2=15.0.0=py310h5764c6d_0
urllib3=1.26.15=pyhd8ed1ab_0
wcwidth=0.2.6=pyhd8ed1ab_0
webencodings=0.5.1=py_1
websocket-client=1.6.0=pyhd8ed1ab_0
websockets=10.4=py310h5764c6d_1
wheel=0.40.0=pyhd8ed1ab_0
widgetsnbextension=4.0.7=pyhd8ed1ab_0
xarray=2023.5.0=pyhd8ed1ab_0
xerces-c=3.2.4=h55805fa_1
xgboost=1.7.5dev.rapidsai23.06=cuda11_py310_0
xorg-kbproto=1.0.7=h7f98852_1002
xorg-libice=1.1.1=hd590300_0
xorg-libsm=1.2.4=h7391055_0
xorg-libx11=1.8.4=h0b41bf4_0
xorg-libxau=1.0.11=hd590300_0
xorg-libxdmcp=1.1.3=h7f98852_0
xorg-libxext=1.3.4=h0b41bf4_2
xorg-libxrender=0.9.10=h7f98852_1003
xorg-renderproto=0.11.1=h7f98852_1002
xorg-xextproto=7.3.0=h0b41bf4_1003
xorg-xproto=7.0.31=h7f98852_1007
xyzservices=2023.5.0=pyhd8ed1ab_1
xz=5.2.6=h166bdaf_0
yaml=0.2.5=h7f98852_2
yarl=1.9.2=py310h2372a71_0
zeromq=4.3.4=h9c3ff4c_1
zfp=1.0.0=h27087fc_3
zict=3.0.0=pyhd8ed1ab_0
zipp=3.15.0=pyhd8ed1ab_0
zlib=1.2.13=hd590300_5
zlib-ng=2.0.7=h0b41bf4_0
zstd=1.5.2=h3eb15da_6
pip:

treelite-runtime==3.2.0

prefix: /home/dleonpe/miniconda3/envs/rapids-23.06

Additional context
Has also been prevalent in older rapids versions.

The text was updated successfully, but these errors were encountered:

lowener · 2023-07-05T14:36:40Z

You are likely running into the error of CUSPARSE_STATUS_INSUFFICIENT_RESOURCES (see here).
On my 48GB GPU I was not able to run the sparse PCA with a density of 0.05, but by lowering the density to less than 0.02 I was able to run it.
By quickly running a few sparse PCA I saw that for a matrix of that dimension, a density of 0.02 leads to a 26GB memory allocation. This is related to the covariance matrix computation.

cuml/python/cuml/decomposition/pca.pyx

Line 365 in 93f7ddc

covariance, self.mean_, _ = cov(X, X, return_mean=True)

So you can either lower the dimension of the matrix or lower it's density so that it fits your GPU memory.

Intron7 · 2023-07-05T14:43:07Z

Ok thank you. I'm just confused because the dense version works with way bigger matrices. Usually sparse matrices are used to save VRAM.

flying-sheep · 2023-07-06T09:25:38Z

@lowener why was this closed? Doesn’t it seem like a bug that dense PCA needs less memory than sparse PCA?

Intron7 · 2023-07-06T12:23:10Z

I think it might be a bug with cuSPARSE. I created a replacement implementation for the cov function with a custom cupy kernel for csr matrices based on a spGEMM paper. It performs really well. I can create a PR.

lowener · 2023-07-06T14:43:53Z

@Intron7 This would be a great new feature that would be very helpful for many users! Your PR would be very welcomed!
CuSparse has recently released more algorithm (as of this February) for SpGEMM that are using less memory. (see here)
So I created this issue cupy/cupy#7699 to add them in cupy.
@flying-sheep I closed this issue too early. I'll reopen it.

## Fixed bug with Sparse PCA function. #5475 I added a new `cov_sparse` function that replaces the `cov` function for `_sparse_fit`. It used a custom kernel to calculate the `gram_matrix` within `cov_sparse` . Authors: - Severin Dicks (https://github.com/Intron7) - Simon Adorf (https://github.com/csadorf) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Simon Adorf (https://github.com/csadorf) URL: #5493

Intron7 added ? - Needs Triage Need team to review and classify bug Something isn't working labels Jun 22, 2023

lowener closed this as completed Jul 5, 2023

lowener reopened this Jul 6, 2023

Intron7 mentioned this issue Jul 6, 2023

Sparse pca patch #5493

Merged

csadorf linked a pull request Jul 7, 2023 that will close this issue

Sparse pca patch #5493

Merged

Intron7 added a commit to Intron7/cuml that referenced this issue Jul 10, 2023

added hint to issue rapidsai#5475

e8df915

csadorf removed the ? - Needs Triage Need team to review and classify label Jul 13, 2023

rapids-bot bot closed this as completed in #5493 Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] PCA doesn't work with sparse matrices #5475

[BUG] PCA doesn't work with sparse matrices #5475

Intron7 commented Jun 22, 2023 •

edited by csadorf

Loading

lowener commented Jul 5, 2023 •

edited

Loading

Intron7 commented Jul 5, 2023

flying-sheep commented Jul 6, 2023

Intron7 commented Jul 6, 2023 •

edited

Loading

lowener commented Jul 6, 2023 •

edited

Loading

[BUG] PCA doesn't work with sparse matrices #5475

[BUG] PCA doesn't work with sparse matrices #5475

Comments

Intron7 commented Jun 22, 2023 • edited by csadorf Loading

lowener commented Jul 5, 2023 • edited Loading

Intron7 commented Jul 5, 2023

flying-sheep commented Jul 6, 2023

Intron7 commented Jul 6, 2023 • edited Loading

lowener commented Jul 6, 2023 • edited Loading

Intron7 commented Jun 22, 2023 •

edited by csadorf

Loading

lowener commented Jul 5, 2023 •

edited

Loading

Intron7 commented Jul 6, 2023 •

edited

Loading

lowener commented Jul 6, 2023 •

edited

Loading