Working with future and memoise together? #506
DarwinAwardWinner
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am doing some fairly large computations, using both future for parallelization and memoise to cache results of certain long and frequently repeated computations. One issue I'm running into is that for a memoised function, there is no obvious way to express the logic of "if this call is already cached by the memoisation backend, then retrieve the result synchronously, otherwise create a future as normal". For example, suppose I have a memoised function, which I want to pass to
future_lapply
or something:Of course, I'll get the right result here no matter what, since f is a pure function. However, on the second run, and for most of the calls to
f_memo
on the first run, there was no need to fork a new process to handle that call, since the call was already cached. The process was forked, and the forked process just retrieved the cached value and returned it to the main process. This unnecessary overhead gets worse as the values getting passed between processes get larger, and as the cost of creating a future increases (e.g. using future.batchtools on a cluster).So, it would be nice if the future package could optionally detect when the future being created is calling a memoised function, and if so, whether the particular call is already cached. And if it detects this case, it should instead return a local sequential future that just reads the cached value in the current R process when resolved, instead of using whatever future backend is configured.
Is this something you'd be interested in supporting? My understanding is that there's some work that would have to be done on the memoise side to properly expose an API for checking if a call is memoised without running it, but I wanted to check if you thought this might be worthwhile.
Alternatively, is there a relatively simply way to do this already that I just haven't found?
Beta Was this translation helpful? Give feedback.
All reactions