Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BENCHMARKING: Record timing and memory stats for the various steps in futures #59

Open
HenrikBengtsson opened this issue Mar 14, 2016 · 5 comments

Comments

@HenrikBengtsson
Copy link
Owner

@HenrikBengtsson HenrikBengtsson commented Mar 14, 2016

Record timing and memory stats for the various steps in futures, e.g. creation, identification of global variables, exporting globals, launching future, time for future to complete, collection of value/exceptions etc. This should be done (optional?) for all types of futures.

@HenrikBengtsson HenrikBengtsson changed the title BENCHMARKING: Record time stats for the various steps in futures BENCHMARKING: Record timing stats for the various steps in futures Mar 14, 2016
@HenrikBengtsson HenrikBengtsson changed the title BENCHMARKING: Record timing stats for the various steps in futures BENCHMARKING: Record timing and memory stats for the various steps in futures Mar 27, 2016
@HenrikBengtsson
Copy link
Owner Author

@HenrikBengtsson HenrikBengtsson commented Jun 11, 2016

For memory profile, we have utils::Rprofmem() which logs all memory allocations done by allocateVector() (part of the native R API). Note that the information logged by Rprofmem() could be improved, but that would require updates to core R, cf. HenrikBengtsson/Wishlist-for-R#25

Loading

@HenrikBengtsson
Copy link
Owner Author

@HenrikBengtsson HenrikBengtsson commented Feb 26, 2017

See also snow::snow.time().

Loading

@HenrikBengtsson
Copy link
Owner Author

@HenrikBengtsson HenrikBengtsson commented Dec 13, 2017

"Note to self": This will most likely require an update to the Future API. More precisely, a backend needs to return not only the value but also other information. Currently, backends return the value (or errors) "as is". In order to return other information, this needs to be updated, e.g. list(value = value, benchmarks = benchmarks).

UPDATE 2018-12-27: This part was resolved in future 1.8.0 (2018-04-08) where FutureResult was introduced.

Loading

@sethberry
Copy link

@sethberry sethberry commented May 31, 2018

Hi Henrik - Curious if you have done any benchmarking on what the memory usage is when multiple R sessions are generated (ie, multisession) using a future? Do the new R sessions get new RAM allocations in Windows or are they constrained by the initial R session’s RAM allocation? Have used tcltk::tclTaskSchedule in the past to do similar non blocking parallelization and ran into this RAM issue. Just curious if you’ve run across this at all? Any insight you might have would be greatly appreciated. Your package looks a like it might be better than going the task scheduler route. Thx, Seth

Loading

@HenrikBengtsson
Copy link
Owner Author

@HenrikBengtsson HenrikBengtsson commented Jun 14, 2018

Hi. When using multisession, a new R session is launched in the background (basically as if you'd start another R session manually). If you launch a vanilla R session you can use the Windows Task Manager to see how much memory that consumes. That'll be your lower-bound memory consumption per background workers. Then, if you parallel code uses functions in packages, then those packages will be loaded/attached as well. That'll consume additional memory (look at Task Manager) - again, this is per worker. On top of that, you'll find that "input data" (arguments and global variables) that the future expression needs, will be exported to the workers, which adds to the memory usage. Because multisession workers live over multiple futures, that is, they don't shut down immediately after a future is resolved, any packages loaded will stay loaded in those workers. However, input data and other created objects will be erased and garbage collected as soon as each worker is done with a future - that helps to keep the memory down.

As a rule of thumb, there is no magic parallelization method/framework in R that is more memory efficient that others. I often assume they all use roughly the same amount. Making sure to rm() large objects no longer needed and avoid coercions is often the best way to keep the memory usage down - regardless of sequential or parallel processing.

Also, some people argue that forked processes (used by multicore futures, mclapply(), ... - so not Windows) may consume less memory because of the "shared memory" property of process forking. However, it has been shown/mentioned several that R's garbage collector can really mess this up - if the garbage collector starts running in one of the forked child processes, or the master process, then that originally shared memory can no longer be shared and the operating system starts copying memory blocks into each child process. Since the garbage collector runs whenever it wants to, there is no simple way to avoid this.

Also, just in case your tcltk::tclTaskSchedule approach relied on it: In R (< 3.3.2) there was an inefficiency in the parallel package (used my multisession futures) causing the workers to hold on to results from previous calls. For large results, that meant that each worker consumed twice the amount of memory as really need. We have since fixed that (HenrikBengtsson/Wishlist-for-R#27).

Hope this helps.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants