show method for >2D arrays load the whole 2d slices (than necessary) in the middle steps

[Liubuntu]

Is it expected to return just a few rows and a few columns? (e.g., first and last 2 rows, ...)

da <- DelayedArray(array(1, dim=c(20, 5, 4)))
trace(extract_array, quote(print(str(index))), exit=quote(print(returnValue())))
## Tracing function "extract_array" in environment
## <namespace:DelayedArray>
## Warning: Tracing only in the namespace; to untrace you will need:
## untrace("extract_array", where = getNamespace("DelayedArray"))
## [1] "extract_array"
## attr(,"package")
## [1] "DelayedArray"
> da
## Tracing extract_array(x, index) on entry 
## List of 3
##  $ : int(0) 
##  $ : int(0) 
##  $ : int(0) 
## NULL
## Tracing extract_array(x@seed, index) on entry 
## List of 3
##  $ : int(0) 
##  $ : int(0) 
##  $ : int(0) 
## NULL
## Tracing extract_array(x@seed, index) on exit 
## <0 x 0 x 0 array of double>


## Tracing extract_array(x, index) on exit 
## <0 x 0 x 0 array of double>


## <20 x 5 x 4> DelayedArray object of type "double":
## Tracing extract_array(x, index) on entry 
## List of 3
##  $ : int(0) 
##  $ : int(0) 
##  $ : int(0) 
## NULL
## Tracing extract_array(x@seed, index) on entry 
## List of 3
##  $ : int(0) 
##  $ : int(0) 
##  $ : int(0) 
## NULL
## Tracing extract_array(x@seed, index) on exit 
## <0 x 0 x 0 array of double>

## ,,1
## Tracing extract_array(x, expand_Nindex_RangeNSBS(Nindex)) on entry 
## List of 3
##  $ : NULL
##  $ : NULL
##  $ : int 1
## NULL
## Tracing extract_array(x@seed, index) on entry 
## List of 3
##  $ : NULL
##  $ : NULL
##  $ : int 1
## NULL
## Tracing extract_array(x@seed, index) on exit 
## , , 1
##       [,1] [,2] [,3] [,4] [,5]
##  [1,]    1    1    1    1    1
##  [2,]    1    1    1    1    1
##  [3,]    1    1    1    1    1
##  [4,]    1    1    1    1    1
##  [5,]    1    1    1    1    1
##  [6,]    1    1    1    1    1
##  [7,]    1    1    1    1    1
##  [8,]    1    1    1    1    1
##  [9,]    1    1    1    1    1
## [10,]    1    1    1    1    1
## [11,]    1    1    1    1    1
## [12,]    1    1    1    1    1
## [13,]    1    1    1    1    1
## [14,]    1    1    1    1    1
## [15,]    1    1    1    1    1
## [16,]    1    1    1    1    1
## [17,]    1    1    1    1    1
## [18,]    1    1    1    1    1
## [19,]    1    1    1    1    1
## [20,]    1    1    1    1    1

## Tracing extract_array(x, expand_Nindex_RangeNSBS(Nindex)) on exit 
## , , 1

##       [,1] [,2] [,3] [,4] [,5]
##  [1,]    1    1    1    1    1
##  [2,]    1    1    1    1    1
##  [3,]    1    1    1    1    1
##  [4,]    1    1    1    1    1
##  [5,]    1    1    1    1    1
##  [6,]    1    1    1    1    1
##  [7,]    1    1    1    1    1
##  [8,]    1    1    1    1    1
##  [9,]    1    1    1    1    1
## [10,]    1    1    1    1    1
## [11,]    1    1    1    1    1
## [12,]    1    1    1    1    1
## [13,]    1    1    1    1    1
## [14,]    1    1    1    1    1
## [15,]    1    1    1    1    1
## [16,]    1    1    1    1    1
## [17,]    1    1    1    1    1
## [18,]    1    1    1    1    1
## [19,]    1    1    1    1    1
## [20,]    1    1    1    1    1

## Tracing extract_array(x, expand_Nindex_RangeNSBS(Nindex)) on entry 
## List of 2
##  $ : int [1:4] 1 2 19 20
##  $ : NULL
## NULL
## Tracing extract_array(x, expand_Nindex_RangeNSBS(Nindex)) on exit 
##      [,1] [,2] [,3] [,4] [,5]
## [1,]    1    1    1    1    1
## [2,]    1    1    1    1    1
## [3,]    1    1    1    1    1
## [4,]    1    1    1    1    1
##       [,1] [,2] [,3] [,4] [,5]
##  [1,]    1    1    1    1    1
##  [2,]    1    1    1    1    1
##   ...    .    .    .    .    .
## [19,]    1    1    1    1    1
## [20,]    1    1    1    1    1

## ...

## ,,4
## Tracing extract_array(x, expand_Nindex_RangeNSBS(Nindex)) on entry 
## List of 3
##  $ : NULL
##  $ : NULL
##  $ : int 4
## NULL
## Tracing extract_array(x@seed, index) on entry 
## List of 3
##  $ : NULL
##  $ : NULL
##  $ : int 4
## NULL
## Tracing extract_array(x@seed, index) on exit 
## , , 1

##       [,1] [,2] [,3] [,4] [,5]
##  [1,]    1    1    1    1    1
##  [2,]    1    1    1    1    1
##  [3,]    1    1    1    1    1
##  [4,]    1    1    1    1    1
##  [5,]    1    1    1    1    1
##  [6,]    1    1    1    1    1
##  [7,]    1    1    1    1    1
##  [8,]    1    1    1    1    1
##  [9,]    1    1    1    1    1
## [10,]    1    1    1    1    1
## [11,]    1    1    1    1    1
## [12,]    1    1    1    1    1
## [13,]    1    1    1    1    1
## [14,]    1    1    1    1    1
## [15,]    1    1    1    1    1
## [16,]    1    1    1    1    1
## [17,]    1    1    1    1    1
## [18,]    1    1    1    1    1
## [19,]    1    1    1    1    1
## [20,]    1    1    1    1    1

## Tracing extract_array(x, expand_Nindex_RangeNSBS(Nindex)) on exit 
## , , 1

##       [,1] [,2] [,3] [,4] [,5]
##  [1,]    1    1    1    1    1
##  [2,]    1    1    1    1    1
##  [3,]    1    1    1    1    1
##  [4,]    1    1    1    1    1
##  [5,]    1    1    1    1    1
##  [6,]    1    1    1    1    1
##  [7,]    1    1    1    1    1
##  [8,]    1    1    1    1    1
##  [9,]    1    1    1    1    1
## [10,]    1    1    1    1    1
## [11,]    1    1    1    1    1
## [12,]    1    1    1    1    1
## [13,]    1    1    1    1    1
## [14,]    1    1    1    1    1
## [15,]    1    1    1    1    1
## [16,]    1    1    1    1    1
## [17,]    1    1    1    1    1
## [18,]    1    1    1    1    1
## [19,]    1    1    1    1    1
## [20,]    1    1    1    1    1

## Tracing extract_array(x, expand_Nindex_RangeNSBS(Nindex)) on entry 
## List of 2
##  $ : int [1:4] 1 2 19 20
##  $ : NULL
## NULL
## Tracing extract_array(x, expand_Nindex_RangeNSBS(Nindex)) on exit 
##      [,1] [,2] [,3] [,4] [,5]
## [1,]    1    1    1    1    1
## [2,]    1    1    1    1    1
## [3,]    1    1    1    1    1
## [4,]    1    1    1    1    1
##       [,1] [,2] [,3] [,4] [,5]
##  [1,]    1    1    1    1    1
##  [2,]    1    1    1    1    1
##   ...    .    .    .    .    .
## [19,]    1    1    1    1    1
## [20,]    1    1    1    1    1

[hpages]

No it's expected to return everything. See ?extract_array and the Implementing A DelayedArray Backend vignette.

[mtmorgan]

It seems like this will be costly -- to 'show' a 1e7 x 1e5 x 3 array it would load, several times, a 1e7 x 1e5 x x arrays...

[hpages]

Ah I see what you mean. Should be fixed in DelayedArray 0.7.28. Excellent catch, thanks!

[Liubuntu]

Thanks @hpages for the quick fix. The 3D array show method now works for my new VCFArray (with a backend of VcfFile).

[hpages]

No problem. Glad it works now. Note that the title of the issue was a little bit misleading since the question was whether or not extract_array(, index = list(NULL, NULL, ...)) should return all rows and all columns. The answer to that question is still yes. However I didn't immediately realize that the real issue you were having was that show() was not working properly on your VCFArray objects (I only realized this after Martin chimed in). It was not working properly because it was calling extract_array(, index = list(NULL, NULL, ...)) internally. This doesn't mean that there is a problem with extract_array(, index = list(NULL, NULL, ...)), only that show() should not call this. Hope this makes sense.

[Liubuntu]

Yes, the title is confusing because I was a little bit confused about the trace messages... The problem is truly the show method of >2D arrays, that when dealing with >2D arrays, the DelayedArray will do the slices of 2-d arrays, and this step was loading the whole slice, and now only the rows/cols that show method needs. Thanks for explaining and the quick fix (so much faster than we find the problem). 😄 Now I understand the back end logics better by the informative tracing message.