Plans on adding group_map/group_modify/group_walk support?

[TobiRoby]

Love dtplyr! I am using the newest version (lazy evaluation) as often as possible.
Sadly there is no support for group_map, group_modify, group_walk functionality.

Are there any plans on adding them in the future?

Here is a failing example:

library(data.table)
library(dtplyr)
library(dplyr, warn.conflicts = FALSE)
library(stringi)

# specify size 
n <- 1e5

# generate data.table with random data
data_dt <- data.table(id=stri_rand_strings(n, 1, pattern = "[A-Z]"),
                      product=stri_rand_strings(n, 3, pattern = "[A-Z]"),
                      date=sample(seq(as.Date('2019/01/01'), as.Date('2019/04/01'), by="day"), n, replace=TRUE),
                      amount=sample(1:10000,n,replace=TRUE),
                      price=rnorm(n, mean = 100, sd = 20))

data_dt_lazy <- lazy_dt(data_dt)

lag_amount_per_group <- function(group, ...){
  group %>% 
    lazy_dt() %>%
    mutate(amount_lag7 = lag(amount, n = 7)) %>% 
    as.data.table()
}

result_dtplyr <- data_dt_lazy %>%
  arrange(date) %>%
  group_by(id) %>%
  group_modify(lag_amount_per_group) %>%
  as.data.table()

---

Error in UseMethod("group_split") :
no applicable method for 'group_split' applied to an object of class "c('dtplyr_step_group', 'dtplyr_step')"

---

Also a working example with native data.table and dplyr:

result_dtplyr <- data_dt %>%
  arrange(date) %>%
  group_by(id) %>%
  group_modify(lag_amount_per_group)

result_native_dt <- data_dt[order(date)][, lag_amount_per_group(.SD), keyby=.(id)]

[hadley]

group_walk() seems like it wouldn't be a good fit here, because side-effects and lazy computation are a weird match. I think the best it could do would be to create force computation with a warning. I think the same is probably true for group_map(), since it has to return a list, and making the results lazy would require some new data structure.

I think group_modify() could return a lazy_dt though:

library(data.table)
library(dplyr, warn.conflicts = FALSE)

dt <- data.table(g = c(1, 1, 2), x = 1:3, y = 2:4, z = 3:5)

myfun <- function(df, key) {
  data.frame(xy = df$x + df$y, yz = df$y + df$z)
}
dt %>% 
  as_tibble() %>% 
  group_by(g) %>% 
  group_modify(myfun)
#> # A tibble: 3 x 3
#> # Groups:   g [2]
#>       g    xy    yz
#>   <dbl> <int> <int>
#> 1     1     3     5
#> 2     1     5     7
#> 3     2     7     9

dt[, as.data.table(myfun(.SD)), key = g]
#>    g xy yz
#> 1: 1  3  5
#> 2: 1  5  7
#> 3: 2  7  9

^{Created on 2019-10-04 by the reprex package (v0.3.0)}

[hadley]

Implementing this will require a group_split() method that collects (with a warning), and a new lazy step_modify().

[hadley]

Need #110 before I implement group_split():

group_split.dtplyr_step <- function(x, ...) {
  warn("Forcing evaluation of lazy data table in group_split()")
  
  group_split(collect(x), ...)
}

[hadley]

That approach doesn't work because group_map() also needs group_keys(), and implementing the methods individual will be inefficient (it would have to collect the lazy_dt twice). Unfortunately, I can't currently provide a group_map() method because it's not generic (tidyverse/dplyr#4576).

Once that issue is fixed, I think group_map() should look something like this:

#' @export
#' @importFrom dplyr group_split
group_map.dtplyr_step <- function(.tbl, .f, ..., keep = FALSE) {
  dt <- as.data.table(.tbl)
  dt[, list(list(.f(.SD, .BY, ...))), by = .tbl$groups]$V1
}