progressor(steps, props)

[HenrikBengtsson]

Add support for realitive sizes of progression steps, e.g.

progress <- progressor(steps = 6L, props = c(1/3,3/3,2/3))
progress <- progressor(steps = 4L, props = c(1,3,2))  ## automatically rescaled

Alternatively,

progress <- progressor(step_sizes = c(1L, 3L, 2L))

where steps = length(step_sizes) by default.

It would be neat if props can be modified along the way, e.g. if one of the steps end up finding a shortcut (because of input data) and only needing, say, 50% of its processing "time".

[HenrikBengtsson]

Allow for named steps and step sizes.

[HenrikBengtsson]

Support for controlling step sizes will be useful when for instance processing a set of elements in chunks, where step sizes are proportional to the chunk size. For example, for y <- future_lapply(x, FUN), the elements in x will be processed in nbrOfWorkers() chunks.

[martinju]

Hi! I would very much like to see this feature. Any timeline for when this might be added?

[HenrikBengtsson]

Sorry, no progress (pun intended) on this one.

This is one of those things that requires long-term deep focus to get correct, especially since it's a matter of philosophy (e.g. "what does it really mean to do partial or weighted progress and what needs does it cover and what are we missing?") and design ("should it be absolute or relative step sizes, weights, or what?"). As soon as that is decided on/settled, then I think it'll be fairly quick to implement it. As soon as it's implemented, I think there's no going back and changing, so it's critical to get the design correct from the beginning.

I'm not sure when I'll have time to deep into this one, but feel free to comment here how you would use it and expand on your needs and use cases. Then more people who do this, the easier it'll be to focus on what the design should be.

[martinju]

Thanks for the quick update. I understand that this is not straightforward. My (current) use case is that the different tasks conducted in the lapply-call are not equally time consuming, first and foremost as involve different number of subtasks. See a constructed example below:

l <- list(1:3,2:9,5:7,2,3:4,5:6)
l

handlers(global = TRUE)
func <- function(x,p){
  Sys.sleep(length(x)/2)
  p(message="Computation ongoing") 
  x^2
}

future::plan(sequential)
progress_func <- function(l){
  p <- progressr::progressor(along = l)
  future.apply::future_lapply(l,func,p = p)
  
}

progress_func(l)

I would like the percentage to move by length(l[[i]])/length(unlist(l)) in iteration i, not 1/length(l). From what I understand, this is currently not possible, or is there are a way around this?
Rewriting the lapply-function is not an option here since in the real application slightly different tasks are conducted for the elements in l.

[HenrikBengtsson]

You can do:

progressr::handlers(global = TRUE)
future::plan("sequential")

l <- list(1:3, 2:9, 5:7, 2, 3:4, 5:6)
str(l)

func <- function(x, p) {
  Sys.sleep(length(x) / 2)
  p(amount = length(x),
    message = sprintf("Computation ongoing [n=%d]", length(x)))
  x^2
}

progress_func <- function(l){
  p <- progressr::progressor(sum(lengths(l)))
  future.apply::future_lapply(l, func, p = p)
}

y <- progress_func(l)

[martinju]

Thanks, this works like a charm!

I obviously have a limited understanding of how progressr actually works as I didn't figure out this myself.
I've browsed the vignette and help functions for pointers and examples of the flexibility of the p <- progressor() and p(), but couldn't really find anything but the basic p(along = x) in the vignette. I am looking in the wrong place, or is this missing? I think a document showing developer use cases would be helpful for many (potential) users.