Is prepare_distributed in run_future_lapply necessary?

[kendonB]

As far as I can tell, this is related to attempts? If so, I'd suggest conditioning this running on attempts > 1.

[wlandau-lilly]

Yes, it is necessary. For both forms of distributed parallelism ("Makefile" and "future_lapply"), the side effects of prepare_distributed() are much more important than the "attempts" flag (which currently exists only to tell drake when it can print "All targets are already up to date."). Initially, your environment is cached so that jobs on the cluster can load it. Next, outdated() is run, which both gets information for the attempts flag and processes all the imports. With all the imports processed, we can devote future_lapply() entirely to proper targets. The imports are usually fast, so there is no reason to waste jobs on them for any kind of distributed computing.

For "Makefile" parallelism, we need to return build_these so we know what fake timestamps to write (to trick the Makefile into running the correct jobs), and it does not slow down "future_lapply" parallelism.

I know it's bad form to have a function with both side effects and a return value, but I would rather do that than have duplicated code.