Does each task need to have its own controller (in shiny or otherwise)?

[teofiln]

Hi @wlandau-lilly and @shikokuchuo Thanks for the great package. I've recently started using it in shiny.

I am trying to start a single controller, but submit various tasks to it. In shiny, for example, I'd have task 1 on button 1 and task 2 on button 2. Then, I'd like to push both tasks (different functions, different data/globals), to the same controller. I thought this would just work, but I see that when I poll task 2 with pop (like in the shiny vignette), I get the result of the concurrently running task 1.

I saw the name argument to push. Is this meant to be used to parse the output of pop to check if task named X is complete?

Or am I better of creating separate controllers for each task (and terminating them).

What would be your advice? Thanks!

[shikokuchuo]

I'll let @wlandau answer with what he definitively recommends. It seems you could condition which reactive output is updated on the 'name' argument as you suggest.

If crew features are not essential, this might be a valid use case to work at the underlying mirai level. You may refer to the equivalent shiny vignette here: https://shikokuchuo.net/mirai/articles/shiny.html crew provides a useful abstraction layer by maintaining its own internal stack of 'mirai', whereas using mirai you'd have to explicitly create your own as per the vignette. This requires a bit more boilerplate, but it is then easy to create separate queues for each task.

But in short: you should not need a separate controller for each task, unless it actually makes sense to separate the resources for each. This would be if you want to ensure some instances of each task get done at the same time, even if for example you have a long backlog of one type of task to process.

[wlandau]

when I poll task 2 with pop (like in the shiny vignette), I get the result of the concurrently running task 1.

The workers argument of crew_controller_local() controls the maximum number of parallel processes to launch. The default value is 1 to avoid overburdening the system by accident. If you set this to a value greater than 1, you should see concurrency for a single controller.

Alternatively, the observe() loop may be taking a long time, especially if post-processing steps after controller$pop() take a long time, or if the polling interval in shiny::invalidateLater() is long. In the latter case, you may have to collect all tasks instead of just one:

observe({
    req(reactive_poll())
    invalidateLater(millis = 100)
    while (!is.null(result <- controller$pop()$result)) {
      reactive_result(result[[1]])
    }
    reactive_status(status_message(n = length(controller$tasks)))
    reactive_poll(controller$nonempty())
})

If that doesn't work, it would help to see a reprex.

If crew features are not essential, this might be a valid use case to work at the underlying mirai level.

An opinion: I find it cumbersome to manually maintain individual mirai task objects when all the tasks are alike: for example, rendering the same kind of generative artwork as in the example. This is why the central controller was a fundamental design/interface goal of crew: it can simplify away a lot of bookkeeping and conditional logic in apps that already have a lot to manage.

That said, the granularity of individual task objects in promises-like scenarios may be more convenient when there is a need for context during post-processing, e.g. if one task was meant to generate artwork, another was meant to download data, both are one-off instead of repeated many times. mirai seamlessly integrates with promises, whereas I have no plans to do so in crew. And regardless, mirai can be used on its own without promises.

@teofiln, I am not sure which pattern would best fit your situation. If there can be only one "button 1" task running at a given time (likewise for button 2) then it seems more promises-like. But if there are only 2 buttons and each can submit multiple concurrent tasks, then crew could help.

By the way, there's also auto-scaling in crew if you need it: looping over pop() automatically launches workers in response to demand. Combine with seconds_idle in the controller (which is forwarded to the mirai daemons), crew and mirai work together to dynamically scale up and down parallel processes in response to the fluctuating task load of the app.

[wlandau]

you should not need a separate controller for each task, unless it actually makes sense to separate the resources for each.

I agree. For multiple resource types, see controller groups: https://wlandau.github.io/crew/articles/groups.html

[teofiln]

Thank you both for the clarification and suggestions. I think my use case is closer to the promise-like scenario. So I am probably going to look at working with mirai directly.

[teofiln]

I think this example shows the kind of pattern I need. I am hoping to run concurrent jobs in the background, but keep the main shiny process active. And make sure that each mirai resolves in the correct place. I am also abstracting the polling logic to make it easier to write. Is this sensible use of mirai?

library(shiny)
library(mirai)

head_six <- function(x, sleep) {
  Sys.sleep(sleep)
  head(x)
}

head_six_async_mirai <- function(x, sleep) {
  args <- list(head_six = head_six, x = x, sleep = sleep)
  bg_process <- mirai::mirai(.expr = head_six(x, sleep), .args = args)
  return(bg_process)
}

# m <- head_six_async_mirai(mtcars, 0)
# m$data

mod_async_srv_mirai <- function(id, fun_async, fun_args, wait_for_event = FALSE, verbose = FALSE) {
  moduleServer( id, function(input, output, session){
    res_rct <- shiny::reactiveVal(NULL)
    poll_rct <- shiny::reactiveVal(TRUE)

    if (isTRUE(wait_for_event)) {
      poll_rct(FALSE)
    }
    
    bg_job <- reactive({
      req(isTRUE(poll_rct()))
      do.call(fun_async, fun_args)
    }) |> bindEvent(poll_rct())

    observe({
      req(isTRUE(poll_rct()))
      invalidateLater(250)
      if (verbose) {
        message(sprintf("checking: %s", id))
      }

      alive <- mirai::unresolved(bg_job())
      if (isFALSE(alive)) {
        res_rct(bg_job()$data)
        if (verbose) {
          message(sprintf("done: %s", id))
        }
        poll_rct(FALSE)
      }
    })

    return(list(
      start_job = function() poll_rct(TRUE),
      get_result = reactive(res_rct())
    ))

  })
}

 ui <- shiny::fluidPage(
    shiny::actionButton("go_iris", "Go Iris"),
    shiny::verbatimTextOutput("iris"),

    # shiny::actionButton("go_mtcars", "Go Mtcars"),
    shiny::verbatimTextOutput("mtcars"),
    
    shiny::sliderInput("slider", "Observations", min= 10, max = 100, value = 50),
    shiny::plotOutput("some_plot")
  )

  server <- function(input, output, session) {

    mirai::daemons(2L)
    onStop(function() mirai::daemons(0L))

    iris_result <- mod_async_srv_mirai(
      id = "iris_srv", 
      fun_async = "head_six_async_mirai",
      fun_args = list(x = iris, sleep = 5),
      wait_for_event = TRUE
    )

    observe({
      iris_result$start_job()
    }) |>  bindEvent(input$go_iris)

    output$iris <- shiny::renderPrint({
      iris_result$get_result()
    })

    mtcars_result <- mod_async_srv_mirai(
      id = "mtcars_srv", 
      fun_async = "head_six_async_mirai",
      fun_args = list(x = mtcars, sleep = 0),
      wait_for_event = FALSE
      )

    output$mtcars <- shiny::renderPrint({
      mtcars_result$get_result()
    }) 

  output$some_plot <- shiny::renderPlot({
    hist(sample(1000, input$slider))
  })
}

shiny::shinyApp(ui, server)

This generally works, although sometimes I get 'miraiInterrupt' chr "" error on one of the processes.

[teofiln]

For more context. In a large app, there might be 10 potential async jobs that get triggered on the same event. So I am hoping to run them on separate processes in parallel, and seamlessly get the results (and shiny objects rendered) as they resolve. Seems like mirai would be more efficient here than starting a process with callr::r_bg for each of the ten tasks. And of course, with callr I would have to ensure somehow that I don't spawn more processes than available cores. mirai should handle this much better such that there can only be as many concurrent jobs as daemons available, while the other wait in the queue.

[shikokuchuo]

Looks good, yes mirai can solve the problem of keeping the UI responsive whilst long-running computations are going on in the background.

mirai is designed for simplicity which allows (i) robustness (few possible points of failure) and (ii) performance (partly due to use of {nanonext} which is a high-performance binding to the NNG C library) and partly through preventing feature bloat. So, yes it should be much more performant that callr in these kind of situations. Important to note that it is also much more flexible as it can also distribute tasks to remote servers.

This generally works, although sometimes I get 'miraiInterrupt' chr "" error on one of the processes.

Are you able to give me more details on this? I cannot reproduce using the code you provided. It is most likely due to: https://blog.r-project.org/2023/05/23/not-interrupting-background-tasks-with-ctrl-c/index.html However with dispatcher, the individual tasks should be protected against that, also I wonder what could be sending such interrupts in the first place. If you are able to isolate this more / send a reprex, that would be much appreciated.

[teofiln]

A quick follow up. What is the appropriate strategy for initializing the daemons. Should this be done in the server, or earlier, in the global (or equvalents)? Does it matter? Or is it on a case by case basis?

Regular shiny scoping rules say that objects defined in global.R are available for all sessions. But those defined in server are session-specific. Are daemons subject to this? i.e., if I want all users of my app to use up to 4 threads, then I initialize the daemons outside the server function. But if I want each session to have up to 2 daemons available, then I can initialize them in the server. Feels like the former is safer, as one can't anticipate how many users would log in concurrently so might spawn more daemons than available threads?

Thanks!!

[teofiln]

For the miraiInterrupt error. Its transient, I can't reliably replicate it. But I wll follow up on it in the mirai repo if I see it the larger application.

[shikokuchuo]

A quick follow up. What is the appropriate strategy for initializing the daemons. Should this be done in the server, or earlier, in the global (or equvalents)? Does it matter? Or is it on a case by case basis?

It shouldn't matter as daemons settings would apply to the entire R session (rather than individual Shiny sessions). It would be fine to define them outside of server.