WISH: Polling frequency

[HenrikBengtsson]

Background

Currently, the package uses batchtools::waitForJobs() every second (getOption("future.wait.delay", 1.0)) to poll the scheduler to see when jobs are finished. Polling the scheduler to check when jobs are ready can but a heavy burden on the scheduler if there are lots of simultaneous requests.

Suggestion

The future.BatchJobs package had a similar approach, but it slowly increased the waiting time between each poll such that for very long running jobs we didn't keep checking every minute. This should be implemented in future.batchjobs too. This requires to implement our own batchtools::waitForJobs() version (or fix it upstream).

[mllg]

Note: There already is a slowly increasing sleep time (up to 2 mins) between requests:

• https://github.com/mllg/batchtools/blob/master/R/sleep.R
• https://github.com/mllg/batchtools/blob/master/R/waitForJobs.R#L24

If you do not like the defaults I picked in default.sleep you can easily provide your own function.

[HenrikBengtsson]

This is awesome; I missed that feature - it looks like exactly what I need.