console being held on call to sge

[yonicd]

is there a way to free the console when sending a job to a grid (like sge)? this would seem like a natural thing to do since the master isnt actually working.

thanks

[DavisVaughan]

So you want to just use furrr as a way to send off jobs to the grid, get the future objects back immediately, and then manually call values() on that yourself when you are ready to try and collect from the grid? I could see that being useful.

I think it would just require an extra wait argument in future_options() and then a change to this line that does or does not try and resolve based on the value of wait. https://github.com/DavisVaughan/furrr/blob/master/R/future_map_template.R#L188

[yonicd]

I think it can be controlled in the tweak you define internally, can that be made into a functional expr for user to pass in?

[DavisVaughan]

What do you mean? What else would it do?

[yonicd]

i use this (from henrik's suggestion), i use sge in this case

my_sge <- future::tweak(future.batchtools::batchtools_sge, template = PATH_TO_TMPL)

future::plan(list(multiprocess, my_sge))

then i run

Y1 %<-% future_lapply(rep(300, 20),
                      FUN = function(nr){
                        solve( matrix(rnorm(nr^2), nrow=nr, ncol=nr))
                        },
                      future.scheduling = 5)

and the grid is governed by another thread and frees the console to continue working

i think you can wrap that around a functional and have furrr eval internally to what the user gives you... having a default in place in case they don't.

[DavisVaughan]

I think I'm going to keep furrr as is for now. I would use future() directly if you need more control over the lazy resolution of values