Option to disable result caching in bench::mark()

[MarcusKlik]

thanks for providing an excellent package, bench's extended set of measurements as compared to other benchmarking packages are extremely useful.

I have a feature request for method bench::mark(), when using it to time a number of expressions that each require a significant amount of RAM, it would be convenient to have an option to disable the caching of results. Caching a potentially large number of large sized objects quickly eats the available memory which limits benchmarking of e.g. large vectors:

nr_of_ints <- 1e8

res <- bench::mark(
  integer(nr_of_ints),
  integer(nr_of_ints),
  max_iterations = 10)

# 2 times 400 MB
object.size(res)
#> 800014016 bytes

I realize that there is a workaround by providing a custom method that returns a small result:

nr_of_ints <- 1e8

fx <- function(nr_of_ints) {
  integer(nr_of_ints)
  TRUE
}

res <- bench::mark(
  fx(nr_of_ints),
  fx(nr_of_ints),
  max_iterations = 10)
#> Warning: Some expressions had a GC in every iteration; so filtering is
#> disabled.

# small
object.size(res)
#> 99464 bytes

With this workaround, the chance of introducing additional garbage collections during benchmarking increases (and these are also measured). So that seems like a less elegant solution :-)

Would it be an idea to skip caching the results when check = FALSE ?

thanks and all the best!

[jimhester]

The garbage collections are happening at the function boundary, you can avoid this by not creating a function.

res <- bench::mark(
  a = { integer(nr_of_ints); NULL },
  b = { integer(nr_of_ints); NULL },
  max_iterations = 10)

But also not keeping the results when check = FALSE might be ok

[MarcusKlik]

Hi @jimhester,

great, thanks for the workaround!

The option to disable result caching would be convenient but from your earlier comment I understand that the focus of bench::mark() is on smaller datasets and for those such an option is not really relevant, so I guess there is something to be said for both approaches...

[jimhester]

As of fce1e23 setting check = FALSE also disables the storage of the results.

[MarcusKlik]

great, thanks for adding the feature!