Lazy ordering of character vectors

[krlmlr]

because this is a rather expensive operation. Reduces run time for grouping a data frame with 1e5 unique strings from ~1.3 s to ~0.8 s. Also includes slight improvements for the CharacterVectorOrderer.

Fixes #2198.

[krlmlr]

So far the benchmarks cannot detect a difference, will wait until I have a benchmark for this code path.

[krlmlr]

Interestingly, the performance difference depends very much on the size of the alphabet; in some cases, this PR is slower. I'll work on adding examples with alphabet sizes of 4, 10, 26 and ~80 to the benchmark code.

Benchmark draft for alphabet size 4, where this PR performs much better than master:

set.seed(123)

create_ids <- function(N) {
  s <- paste(sample(c(letters[1:4], "|"), N, replace = TRUE), collapse = "")
  ss <- strsplit(s, "|", fixed = TRUE)[[1]]
  ss <- unique(ss)
  ss <- ss[nchar(ss) > 3]
  ss
}

N <- 1e7
ids <- create_ids(N)

benchmark <- function(ids, summarize) {
  force(ids)
  df <- data_frame(ids, n = 0)
  gc()
  if (summarize) {
    system.time(group_by(df, ids) %>% summarize(n = mean(n)))
  } else {
    system.time(group_by(df, ids))
  }
}

devtools::load_all()

NN <- 3e5
benchmark(ids, TRUE)
benchmark(sample(ids, NN, replace = FALSE), TRUE)
benchmark(sample(ids, NN, replace = TRUE), TRUE)

[krlmlr]

Alphabet size changes the expected length of common identifier parts. With fce0a9f a run time change can be observed with the existing tests, but this change assumes string equality if and only if the SEXP is equal; this will require that we use only UTF-8 for column data (#1885).

[lock]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/