Error creating an alias function for .N

[markfairbanks]

I am trying to create an alias function for .N, but run into an error that I think is caused by a data.table internal issue.

library(data.table)

test_df <- data.table(a = 1:3, b = c("a","a","b"))

get_n <- function() eval(quote(.N), parent.frame())

# Using get_n() alone works
test_df[, list(count = get_n()), by = b] # Works
#>    b count
#> 1: a     2
#> 2: b     1

# Using it with another summary function fails
test_df[, list(count = get_n(), avg_a = mean(a)), by = b]
#> Error in q[[2L]]: subscript out of bounds

[jangorecki]

Yes, it definitely looks like an internal issue, looking at traceback we can observe the following

traceback()
#3: .gforce_ok(jsub[[ii]]) at data.table.R#1701
#2: `[.data.table`(test_df, , list(count = get_n(), avg_a = mean(a)), 
#       by = b)
#1: test_df[, list(count = get_n(), avg_a = mean(a)), by = b]

Any reason why you cannot use .N directly? It may help us to find a proper solution for you.
For example if you would be ok with using a symbol as an alias, rather than a function call, then that would be easier.

[markfairbanks]

I have a package that replicates tidyverse syntax with a data.table/rlang backend. (I am aware of dtplyr, but dtplyr is missing a lot of tidyverse functionality. pivot/fill/etc.).

For me personally it's really easy to just use .N, but for users of the package that are coming from the tidyverse I am trying to replicate dplyr's n().

Here is the shell of the two functions:

library(rlang)
library(data.table)

n_fn <- function() {
  eval_tidy(expr(.N), env = caller_env())
}

summarize_fn <- function(.data, ..., by = NULL) {

  dots <- enexprs(...)
  by <- enexpr(by)

  eval_tidy(expr(
    .data[, list(!!!dots), by = !!by]
  ))
}

After making these I found the issue.

[markfairbanks]

I actually found a workaround this morning by wrapping each phrase in parentheses, so I don't know if this issue is very pressing.

summarize_fn <- function(.data, ..., by = NULL) {

  dots <- enexprs(...)
  by <- enexpr(by)
  
  # Workaround by adding parentheses to user input
  dots <- lapply(dots, function(.x) parse_expr(paste0("(", deparse(.x), ")")))
  
  eval_tidy(expr(
    .data[, list(!!!dots), by = !!by]
  ))
}

When I tested this in data.table it seems to make summary functions faster. Is there a reason for this?

library(data.table)
library(bench)
setDTthreads(4)

set.seed(123)
data_size <- 10000000
test_df <- data.table(a = sample(1:20, data_size, TRUE),
                      b = sample(c("a","a","b","c","d"), data_size, TRUE))

bench::mark(
  normal = test_df[, list(avg_a = mean(a)), by = b],
  parentheses = test_df[, list(avg_a = (mean(a))), by = b],
  check = TRUE,
  iterations = 30)
#> Warning: Some expressions had a GC in every iteration; so filtering is disabled.
#> # A tibble: 2 x 6
#>   expression       min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr>  <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 normal         150ms  161.2ms      5.80     299MB     7.34
#> 2 parentheses     78ms   91.9ms     10.7       84MB     3.56

This seems to occur with mean/.N/max/etc.

[jangorecki]

Using verbose=TRUE shows that different algorithm is being used in each of those cases.

---

And the computation speed of those, which is actually an off-topic in this issue seems to be faster when GForce is disabled. I am able to reproduce it.

> invisible(test_df[, list(avg_a = mean(a)), by = b, verbose=TRUE])
Detected that j uses these columns: a 
Finding groups using forderv ... forder.c received 10000000 rows and 1 columns
0.178s elapsed (0.306s cpu) 
Finding group sizes from the positions (can be avoided to save RAM) ... 0.000s elapsed (0.000s cpu) 
Getting back original order ... forder.c received a vector type 'integer' length 4
0.000s elapsed (0.000s cpu) 
lapply optimization is on, j unchanged as 'list(mean(a))'
GForce optimized j to 'list(gmean(a))'
Making each group and running j (GForce TRUE) ... gforce initial population of grp took 0.021
gforce assign high and low took 0.219
This gsum took (narm=FALSE) ... gather took ... 0.051s
0.077s
gforce eval took 0.077
0.317s elapsed (0.457s cpu) 

> invisible(test_df[, list(avg_a = (mean(a))), by = b, verbose=TRUE])
Detected that j uses these columns: a 
Finding groups using forderv ... forder.c received 10000000 rows and 1 columns
0.186s elapsed (0.298s cpu) 
Finding group sizes from the positions (can be avoided to save RAM) ... 0.000s elapsed (0.000s cpu) 
Getting back original order ... forder.c received a vector type 'integer' length 4
0.000s elapsed (0.001s cpu) 
lapply optimization is on, j unchanged as 'list((mean(a)))'
GForce is on, left j unchanged
Old mean optimization is on, left j unchanged.
Making each group and running j (GForce FALSE) ... 
  collecting discontiguous groups took 0.091s for 4 groups
  eval(j) took 0.049s for 4 calls
0.097s elapsed (0.154s cpu) 

There might be an open issue related to that already, not sure now. We have quite detailed timings reported here so we can see what exactly is being slower than the other. fyi @mattdowle

[jangorecki]

Regarding the question raised in the issue: alias of .N

We could eventually support .N() to be interpretted as .N, then we can easily substitute function call using new #4304 feature

DT[, get_n(), env=list(get_n=".N")]
# will get transformed into
#DT[, .N()]

Another thing I see in your code is to splice dots inline into list. This one I recently mentioned in #4304 (comment)

[markfairbanks]

That sounds great. Thanks for all the help