dplyr creates deep copies of data.tables

[matthieugomez]

From the documentation of dplyr, "mutate() never copies columns, except when you modify an existing column". But this is not true for data.tables.

library(pryr)
library(devtools)
install_github("hadley/lazyeval")
install_github("hadley/dplyr")
install_github("Rdatatable/data.table", build_vignettes=FALSE)
library(dplyr)
library(data.table)

N=1e8; K=100
set.seed(1)
DF <- data.frame(
  id = 1:N,
  v1 = sample(round(runif(100,max=100),4), N, TRUE)                       
)
object_size(DF)
1.2 GB
DF1 <- DF %>% mutate(y=mean(v1))
object_size(DF,DF1)
2 GB

While for a data.table object

N=1e8; K=100
set.seed(1)
DT <- data.table(
  id = 1:N,
  v1 = sample(round(runif(100,max=100),4), N, TRUE)                        
)
object_size(DT)
1.2 GB
DT1 <- DT %>% mutate(y=mean(v1))
object_size(DT,DT1)
3.2 GB

The fonction copy() inside the code for mutate seems to do a deep copy and not a shallow copy.
Maybe shallow copies can be used. I don't know the right way to do it but this seems to work:

setDF(DT)
DT1 <- DT
setDT(DT)
setDT(DT1)
DT1[,y:=mean(v1)]
object_size(DT,DT1)
2 GB

[hadley]

Oh hmmm, it seems to me like data.table::copy() should make a shallow copy, not a deep copy.

@arunsrinivasan any thoughts on the best way to do a shallow copy in dt?

[arunsrinivasan]

@hadley, we've shallow function in data.table, but it isn't exported yet. We'll need to add some extra stuff in there so that we can take care of sub-assign by reference properly on shallow copied data.tables. Once that's done and function exported, I'll write back here (or as a pull request?).

[hadley]

@arunsrinivasan a pull request would be awesome!

[hadley]

@arunsrinivasan any change on this? If not, I'll put off until the next release.

[arunsrinivasan]

Not yet, sorry. Likely in 1.9.8. Will write back.

[krlmlr]

@arunsrinivasan shallow() doesn't seem to be exported yet in the CRAN version (1.9.6). Any prospects?

Sorry: You said 1.9.8 -- I guess it's in the pipeline then?