Error in py_call_impl(callable, dots$args, dots$keywords) : ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

[rrydbirk]

Unless my input (data) has equal row and column numbers, I receive this error. Am I doing something wrong?

dim(data)
[1] 8910 41

traceback()
4: stop(structure(list(message = "ValueError: Buffer has wrong number of dimensions (expected 1, got 2)",
call = py_call_impl(callable, dots$args, dots$keywords),
cppstack = structure(list(file = "", line = -1L, stack = c("/d0/home/rasmusr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/libs/reticulate.so(Rcpp::exception::exception(char const*, bool)+0x78) [0x7f66e414eea8]",
"/d0/home/rasmusr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/libs/reticulate.so(Rcpp::stop(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0x27) [0x7f66e414ef17]",
"/d0/home/rasmusr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/libs/reticulate.so(py_call_impl(PyObjectRef, Rcpp::Vector<19, Rcpp::PreserveStorage>, Rcpp::Vector<19, Rcpp::PreserveStorage>)+0x52d) [0x7f66e416109d]",
"/d0/home/rasmusr/R/x86_64-pc-linux-gnu-library/4.0/reticulate/libs/reticulate.so(_reticulate_py_call_impl+0x95) [0x7f66e4149f15]",
"/usr/lib/R/lib/libR.so(+0xf9290) [0x7f6715177290]", "/usr/lib/R/lib/libR.so(+0x13940c) [0x7f67151b740c]",
"/usr/lib/R/lib/libR.so(Rf_eval+0x180) [0x7f67151c1670]",
"/usr/lib/R/lib/libR.so(+0x14548f) [0x7f67151c348f]", "/usr/lib/R/lib/libR.so(Rf_applyClosure+0x1c7) [0x7f67151c4257]",
"/usr/lib/R/lib/libR.so(+0x13a909) [0x7f67151b8909]", "/usr/lib/R/lib/libR.so(Rf_eval+0x180) [0x7f67151c1670]",
"/usr/lib/R/lib/libR.so(+0x14548f) [0x7f67151c348f]", "/usr/lib/R/lib/libR.so(Rf_applyClosure+0x1c7) [0x7f67151c4257]",
"/usr/lib/R/lib/libR.so(+0x13a909) [0x7f67151b8909]", "/usr/lib/R/lib/libR.so(Rf_eval+0x180) [0x7f67151c1670]",
"/usr/lib/R/lib/libR.so(+0x14548f) [0x7f67151c348f]", "/usr/lib/R/lib/libR.so(Rf_applyClosure+0x1c7) [0x7f67151c4257]",
"/usr/lib/R/lib/libR.so(Rf_eval+0x353) [0x7f67151c1843]",
"/usr/lib/R/lib/libR.so(+0x148613) [0x7f67151c6613]", "/usr/lib/R/lib/libR.so(Rf_eval+0x572) [0x7f67151c1a62]",
"/usr/lib/R/lib/libR.so(Rf_ReplIteration+0x23a) [0x7f67151f539a]",
"/usr/lib/R/lib/libR.so(+0x177761) [0x7f67151f5761]", "/usr/lib/R/lib/libR.so(run_Rmainloop+0x48) [0x7f67151f5818]",
"/usr/lib/rstudio-server/bin/rsession(+0x887166) [0x557db1872166]",
"/usr/lib/rstudio-server/bin/rsession(+0x8632ad) [0x557db184e2ad]",
"/usr/lib/rstudio-server/bin/rsession(+0x7c666) [0x557db1067666]",
"/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f6713456b97]",
"/usr/lib/rstudio-server/bin/rsession(+0x9fe4a) [0x557db108ae4a]"
)), class = "Rcpp_stack_trace")), class = c("Rcpp::exception",
"C++Error", "error", "condition")))
3: py_call_impl(callable, dots$args, dots$keywords)
2: densMAP$fit_transform(data) at densmap.R#54
1: densMAP(data, verbose = T)

sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] magrittr_1.5 optparse_1.6.6 reticulate_1.16-9000

loaded via a namespace (and not attached):
[1] fastmatch_1.1-0 plyr_1.8.6 igraph_1.2.5 splines_4.0.2 BiocParallel_1.22.0
[6] GenomeInfoDb_1.24.2 ggplot2_3.3.2 scater_1.16.2 urltools_1.7.3 digest_0.6.25
[11] htmltools_0.5.0 GOSemSim_2.14.0 viridis_0.5.1 GO.db_3.11.4 memoise_1.1.0
[16] limma_3.44.3 annotate_1.66.0 graphlayouts_0.7.0 matrixStats_0.56.0 sccore_0.1
[21] enrichplot_1.8.1 prettyunits_1.1.1 colorspace_1.4-1 blob_1.2.1 rappdirs_0.3.1
[26] ggrepel_0.8.2 pagoda2_0.1.1 xfun_0.15 dplyr_1.0.0 crayon_1.3.4
[31] RCurl_1.98-1.2 jsonlite_1.7.0 scatterpie_0.1.4 genefilter_1.70.0 brew_1.0-6
[36] survival_3.2-3 glue_1.4.1 polyclip_1.10-0 gtable_0.3.0 zlibbioc_1.34.0
[41] XVector_0.28.0 DelayedArray_0.14.0 BiocSingular_1.4.0 Rook_1.1-1 SingleCellExperiment_1.10.1
[46] BiocGenerics_0.34.0 scales_1.1.1 DOSE_3.14.0 DBI_1.1.0 edgeR_3.30.3
[51] Rcpp_1.0.5 viridisLite_0.3.0 xtable_1.8-4 progress_1.2.2 gridGraphics_0.5-0
[56] dqrng_0.2.1 bit_1.1-15.2 europepmc_0.4 rsvd_1.0.3 stats4_4.0.2
[61] httr_1.4.1 getopt_1.20.3 fgsea_1.14.0 RColorBrewer_1.1-2 ellipsis_0.3.1
[66] pkgconfig_2.0.3 XML_3.99-0.4 farver_2.0.3 locfit_1.5-9.4 ggplotify_0.0.5
[71] tidyselect_1.1.0 rlang_0.4.7 reshape2_1.4.4 later_1.1.0.1 AnnotationDbi_1.50.1
[76] munsell_0.5.0 tools_4.0.2 downloader_0.4 generics_0.0.2 RSQLite_2.2.0
[81] ggridges_0.5.2 stringr_1.4.0 fastmap_1.0.1 yaml_2.2.1 knitr_1.29
[86] bit64_0.9-7 tidygraph_1.2.0 purrr_0.3.4 dendextend_1.13.4 ggraph_2.0.3
[91] mime_0.9 scran_1.16.0 DO.db_2.9 xml2_1.3.2 compiler_4.0.2
[96] rstudioapi_0.11 beeswarm_0.2.3 tibble_3.0.3 statmod_1.4.34 tweenr_1.0.1
[101] geneplotter_1.66.0 stringi_1.4.6 lattice_0.20-41 Matrix_1.2-18 vctrs_0.3.1
[106] pillar_1.4.6 lifecycle_0.2.0 BiocManager_1.30.10 triebeard_0.3.0 BiocNeighbors_1.6.0
[111] data.table_1.12.8 cowplot_1.0.0 bitops_1.0-6 irlba_2.3.3 httpuv_1.5.4
[116] conos_1.3.0 GenomicRanges_1.40.0 qvalue_2.20.0 R6_2.4.1 promises_1.1.1
[121] gridExtra_2.3 vipor_0.4.5 IRanges_2.22.2 MASS_7.3-51.6 SummarizedExperiment_1.18.1
[126] rjson_0.2.20 DESeq2_1.28.1 cacoa_0.1 S4Vectors_0.26.1 GenomeInfoDbData_1.2.3
[131] parallel_4.0.2 hms_0.5.3 clusterProfiler_3.16.0 grid_4.0.2 tidyr_1.1.0
[136] DelayedMatrixStats_1.10.1 rvcheck_0.1.8 ggforce_0.3.2 base64enc_0.1-3 Biobase_2.48.0
[141] shiny_1.5.0 tinytex_0.24 ggbeeswarm_0.6.0

[anashwin]

That's odd ... will look into it shortly, thanks!

[anashwin]

Hi @rrydbirk so sorry that it took me so long to get to this. I've been looking into the issue to see if the entire R wrapper was compromised because of an update we made, but that does not seem to be the case. Do you mean that it only works for you when you input a square matrix? Would you be able to send me (at least a subsample) of the input data you are trying to use?

[woshijiaomu]

@ashwinn226 I just run the example codes in python3, It seems that the number of rows in the input array cannot exceed 4000, otherwise densmap will report the error,
import densmap
from sklearn.datasets import fetch_openml
from sklearn.utils import resample
digits = fetch_openml(name='mnist_784')
subsample, subsample_labels = resample(digits.data, digits.target, n_samples=7000, stratify=digits.target, random_state=1)
emb, ro, re = densmap.densMAP().fit_transform(subsample[0:3000,:])
emb, ro, re = densmap.densMAP().fit_transform(subsample[0:4000,:])
until this line，the program reports no errors
emb, ro, re = densmap.densMAP().fit_transform(subsample[0:4100,:])
Traceback (most recent call last):
File "", line 1, in
File "/Users/lvdekang/densvis-master/densmap/densmap/densmap_.py", line 2027, in fit_transform
self.fit(X, y)
File "/Users/lvdekang/densvis-master/densmap/densmap/densmap_.py", line 1794, in fit
self._search_graph.transpose()
File "/Users/lvdekang/miniconda3/envs/site/lib/python3.7/site-packages/scipy/sparse/lil.py", line 437, in transpose
return self.tocsr(copy=copy).transpose(axes=axes, copy=False).tolil(copy=False)
File "/Users/lvdekang/miniconda3/envs/site/lib/python3.7/site-packages/scipy/sparse/lil.py", line 462, in tocsr
_csparsetools.lil_get_lengths(self.rows, indptr[1:])
File "_csparsetools.pyx", line 109, in scipy.sparse._csparsetools.lil_get_lengths
ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

[jrdliu]

Hi there, I think I'm having the same issue as well.

If this is useful to anyone, I found that one possible workaround is to change 2 lines in densmap_.py (1737 and 1852 in the version I'm currently using) that seem to specifically test whether the data has 4,096 or more rows:

if X.shape[0] < 4096: (at 1737)
if y.shape[0] < 4096: (at 1852)

I changed them to some arbitrarily high number greater than the number of rows in my matrix/data frame and it seemed to run in 1 minute or less for my 5000 x 10 data matrix.

I did have to delete/refresh the precompiled versions of densmap_.py in the 'pycache' folder for it to take effect, though.

[anashwin]

Wow great find ... I have no memory of putting those lines in there, will look into what's going on.

[brianhie]

Also facing this issue, looks like behavior is caused by differences between implementations of nearest neighbors search between small and large datasets.

[anashwin]

Thanks all for patience. There was some issue with scipy's LIL sparse matrix type, and I just switched to CSR matrices, and this seems to have fixed the issue.