Error writing sub arrays

[jkanche]

Hi @eddelbuettel I'm using a combination of scripts from my earlier issue and from this page - https://docs.tiledb.com/main/solutions/tiledb-embedded/api-usage/writing-arrays/writing-in-dense-subarrays

Hopefully I'm doing something wrong here

library(tiledb)

uri <- "test.tdb"
if (dir.exists(uri)) unlink(uri, recursive=TRUE, force=TRUE)

## Generate a matrix
n <- 5L
k <- 4L
mat <- matrix(1:(n*k) * 0.12345, nrow=n, ncol=k)
print(mat)

dom <- tiledb_domain(dims = c(tiledb_dim("rows", c(1L, n), n, "INT32"),
                              tiledb_dim("cols", c(1L, k), k, "INT32")))
schema <- tiledb_array_schema(dom, attrs=tiledb_attr("vals", type="FLOAT64"))
tiledb_array_create(uri, schema)

arr <- tiledb_array(uri, is.sparse=FALSE)
I <- c(1:3)
J <- c(2:3)

print(mat[I, J])
arr[I, J] <- mat[I, J]

Fails to write in chunks and expects full matrix

> arr[I, J] <- mat[I, J]
Error in libtiledb_query_submit(qryptr) : 
  [TileDB::Writer] Error: Buffer sizes check failed; Invalid number of cells given for attribute 'vals' (6 != 20)

[jkanche]

sessionInfo

> sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] tiledb_0.9.2

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6        lattice_0.20-44   zoo_1.8-9         digest_0.6.27     grid_4.1.0        magrittr_2.0.1   
 [7] rlang_0.4.11      DT_0.18           shinyjs_2.0.0     tools_4.1.0       bit64_4.0.5       RcppCCTZ_0.2.9   
[13] nanotime_0.3.2    htmlwidgets_1.5.3 bit_4.0.4         compiler_4.1.0    htmltools_0.5.1.1

[eddelbuettel]

Thanks for the bug report. I can replicate the issue and will take a look.

[eddelbuettel]

I probably lack a subarray assignment somewhere in the dense case in the code for the higher-level tiledb_array assignment, and will look into fixing this.

In the meantime, you can obtain the desired behaviour via the more fundamental operations for the underlying library. Here I assign 'scaled by a factor of 10' just to check I hit the desired array elements:

qry <- tiledb_query(arr, "WRITE")
qry <- tiledb_query_set_buffer(qry, "vals", 10*as.array(mat[I,J]))
qry <- tiledb_query_set_subarray(qry, c(1L, 3L, 2L, 3L))
qry <- tiledb_query_set_layout(qry, "ROW_MAJOR")
qry <- tiledb_query_submit(qry)
qry <- tiledb_query_finalize(qry)

[jkanche]

running into issues when I use this to write a matrix in chunks -

> I <- c(1:3)
> J <- c(3:4)
> print(mat[I, J])
        [,1]    [,2]
[1,] 1.35795 1.97520
[2,] 1.48140 2.09865
[3,] 1.60485 2.22210
> # arr[I, J] <- mat[I, J]
> qry <- tiledb_query(arr, "WRITE")
> qry <- tiledb_query_set_buffer(qry, "vals", as.array(mat[I,J]))
> qry <- tiledb_query_set_subarray(qry, c(1L, 3L, 3L, 4L))
> qry <- tiledb_query_set_layout(qry, "COL_MAJOR")
> qry <- tiledb_query_submit(qry)
> qry <- tiledb_query_finalize(qry)
> I <- c(4:5)
> J <- c(1:4)
> print(mat[I, J])
        [,1]    [,2]    [,3]    [,4]
[1,] 0.49380 1.11105 1.72830 2.34555
[2,] 0.61725 1.23450 1.85175 2.46900
> # arr[I, J] <- mat[I, J]
> # qry <- tiledb_query(arr, "WRITE")
> qry <- tiledb_query_set_buffer(qry, "vals", as.array(mat[I,J]))
> qry <- tiledb_query_set_subarray(qry, c(4L, 5L, 1L, 4L))
> qry <- tiledb_query_set_layout(qry, "COL_MAJOR")
> qry <- tiledb_query_submit(qry)
> qry <- tiledb_query_finalize(qry)
> arr2 <- tiledb_array(uri, is.sparse=FALSE, as.matrix=TRUE)
> res2 <- arr2[]
> print(res2)
     [,1] [,2]    [,3]    [,4]
[1,]  NaN  NaN 1.35795 1.97520
[2,]  NaN  NaN 1.48140 2.09865
[3,]  NaN  NaN 1.60485 2.22210
[4,]  NaN  NaN     NaN     NaN
[5,]  NaN  NaN     NaN     NaN

I'm expecting except for the block 1:3, 1:2 everything else should be filled in the matrix

[eddelbuettel]

I think I have a fix for initial case. I will need to look at what you posted since.

With the fix I have here, if I write in two chunks (your initial one, another non-overlapping one just to illustrate) via (this as extensions of your initial example)

arr <- tiledb_array(uri, is.sparse=FALSE, as.matrix=TRUE)
I <- c(1:3)
J <- c(2:3)
arr[I, J] <- mat[I, J]

I <- 4:5
J <- 3:4
arr[I,J] <- mat[I, J]

print(arr[1:5,1:4])    # show all

I get

     [,1]    [,2]    [,3]    [,4]
[1,]  NaN 0.74070 1.35795     NaN
[2,]  NaN 0.86415 1.48140     NaN
[3,]  NaN 0.98760 1.60485     NaN
[4,]  NaN     NaN 1.72830 2.34555
[5,]  NaN     NaN 1.85175 2.46900

which seems right.

[eddelbuettel]

I think the fix I just committed also covers your second case.

Full Script

library(tiledb)

uri <- "test.tdb"
if (dir.exists(uri)) unlink(uri, recursive=TRUE, force=TRUE)

## Generate a matrix
n <- 5L
k <- 4L
mat <- matrix(1:(n*k) * 0.12345, nrow=n, ncol=k)
print(mat)

dom <- tiledb_domain(dims = c(tiledb_dim("rows", c(1L, n), n, "INT32"),
                              tiledb_dim("cols", c(1L, k), k, "INT32")))
schema <- tiledb_array_schema(dom, attrs=tiledb_attr("vals", type="FLOAT64"))
tiledb_array_create(uri, schema)

arr <- tiledb_array(uri, is.sparse=FALSE)
I <- c(1:3)
J <- c(3:4)
print(mat[I, J])
arr[I, J] <- mat[I, J]

I <- 4:5
J <- 1:4
arr[I,J] <- mat[I, J]

arr2 <- tiledb_array(uri, as.matrix=TRUE)
print(arr2[])
print(arr2[1:5,1:4])

Output

$ Rscript issue255-2.R 
        [,1]    [,2]    [,3]    [,4]
[1,] 0.12345 0.74070 1.35795 1.97520
[2,] 0.24690 0.86415 1.48140 2.09865
[3,] 0.37035 0.98760 1.60485 2.22210
[4,] 0.49380 1.11105 1.72830 2.34555
[5,] 0.61725 1.23450 1.85175 2.46900
[1] "test.tdb"
        [,1]    [,2]
[1,] 1.35795 1.97520
[2,] 1.48140 2.09865
[3,] 1.60485 2.22210
        [,1]    [,2]    [,3]    [,4]
[1,]     NaN     NaN 1.35795 1.97520
[2,]     NaN     NaN 1.48140 2.09865
[3,]     NaN     NaN 1.60485 2.22210
[4,] 0.49380 1.11105 1.72830 2.34555
[5,] 0.61725 1.23450 1.85175 2.46900
        [,1]    [,2]    [,3]    [,4]
[1,]     NaN     NaN 1.35795 1.97520
[2,]     NaN     NaN 1.48140 2.09865
[3,]     NaN     NaN 1.60485 2.22210
[4,] 0.49380 1.11105 1.72830 2.34555
[5,] 0.61725 1.23450 1.85175 2.46900
$ 

[jkanche]

awesome! Thank you for the quick changes. Will this hit CRAN anytime soon ?

[eddelbuettel]

Thanks you for the detailed report! I think I may just add your examples as bases for some quick unit tests, then it can be PRed and we can probably look into CRAN "soon" thereafter, with at least one ball up in the air for TileDB Embedded 2.3.0 which should materialize around the same time.

Can we help you with a formal-enough release candidate prior to CRAN? Is merged-to-main branch good enough or would you need a formal tarball on CRAN. (Not suggesting to short-circuit things, processes exist for a reason, simply scoping out if there is middle ground...)

[jkanche]

I prefer a formal update on CRAN because our build system syncs nightly with CRAN. We could change it to the master branch on GitHub, but It might break other packages that use/depend on tiledb.

[eddelbuettel]

(I'm in favour of that approach too. We'll work on getting this up there 'soon enough'. Another fix or two and polish or another have come in since 0.9.2, and CRAN has no open issues with the package either so should all progress...)

[eddelbuettel]

Now merged in master with a test based on this example -- thanks again for raising the issue. We will try to get this to CRAN in the next couple of days.