Progress bars

[hadley]

Would be nice to have support for progress bars in all map functions. This is a nice feature of plyr.

Could use https://github.com/gaborcsardi/progress, although we might need to ask @gaborcsardi to also provide a C API.

[gaborcsardi]

There is a header only C++ API, isn't that good? Although I have to say that is not very well tested and have less features. https://github.com/gaborcsardi/progress#c-api

[gaborcsardi]

It would also make sense to put the C++ part in another package, as it is completely independent.

[lionel-]

This will not work with mapping functions because we eval R functions from the C code :/ Any user interruption or R error will cause a long jump that bypasses all C++ destructors.

Thus if you have any data on the heap, you'll get leaks. For example all STL containers or even a simple std::string allocate memory dynamically so need to be destructed appropriately. See discussion in e2def88

[gaborcsardi]

Well, there is an R API and a C++ API. It seems reasonable that you would be able to use at least one of them. :)

[vixr]

+1 for this feature

[gaborcsardi]

FWIW I wanted to note that I am adding some new progress bar API, which has the nice feature of having (almost) zero overhead when the progress bars are not shown (e.g. non-interactive use), in addition to ease of use. This is how it will look:

progress %~~% lapply(seq, fun, ...)

If progress bars are turned off, then it simply runs the lapply. If progress bars are on, then it appends the progress bar ticks to fun.

I am saying this, because it would be great to use it for purrr functions as well.

[lionel-]

It'd probably be more purrr-like to have an adverb functional or function operator that takes mapping functionals and add progress bars to them. With a functional:

# ..f must be another functional that takes a .x and a .f
# ..f must have the usual purrr signature ..f(.x, .f, ...)
with_progress <- function(..x, ..f, .f, ...) {
  .f <- add_progress(.f, length(..x))
  ..f(..x, .f, ...)
}

mtcars %>% with_progress(map, as.character)

[lionel-]

A quick thought: I think it's natural to have adverb functionals when we're modifying another functional, like in the example above. But otoh it's natural to have adverb function operators when we're modifying a regular function, e.g. safely(), lift(), etc.

[gaborcsardi]

@lionel- Hmmm, maybe I misunderstand sg, but why not

mtcars %>% with_progress(map)(as.character)

then? Or is this what you mean in your second comment?

[lionel-]

Or is this what you mean in your second comment?

yes this is what I mean. Maybe @hadley has another opinion though.

[gaborcsardi]

Hmmm, actually I quite like this, no extra operator needed. Maybe I should do it with lapply as well. Unfortunately I cannot really do it with for loops. They have to use an operator:

with_pb %~~% for (i in 1:100) { }

[lionel-]

Maybe I should do it with lapply as well

lapply(), vapply() etc should work for free with this approach since they take a vector as first argument and a function as second :)

mtcars %>% with_progress(vapply, sum, numeric(1))

Unfortunately I cannot really do it with for loops. They have to use an operator:

There is some discussion about function-like looping in #168 and #135.

[hadley]

I think that progress bars are so useful, there should be minimal friction to use them in purrr. That makes me think that they should be an option (like plyr), or possibly even automatically display given some conditional (e.g. loop has run for 2 seconds and has at least two more to go, like dplyr).

[lionel-]

How about automatically displaying them unless (a) this is not an interactive session (b) a global option is set to disable them?

This is a case where it makes sense to have a global option since this is a side effect for user convenience that shouldn't have an impact on the return value. Also it'll still be possible to use withr::with_options() in case it's important to control the option on a case by case basis (though I don't see when that would be useful). I think that's preferable to a plyr-like option that would clutter the function signatures.

Also it's still nice to have a functional to add progress bars to lapply() etc.

[hadley]

Yes, agreed about non-interactive use + global option to turn off. That's what dplyr has too.