Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up vis_drake_graph() for large numbers of dynamic sub-targets #1207

Closed
3 tasks done
wlandau opened this issue Mar 6, 2020 · 2 comments
Closed
3 tasks done

Speed up vis_drake_graph() for large numbers of dynamic sub-targets #1207

wlandau opened this issue Mar 6, 2020 · 2 comments

Comments

@wlandau
Copy link
Collaborator

wlandau commented Mar 6, 2020

Prework

  • Read and abide by drake's code of conduct.
  • Search for duplicates among the existing issues, both open and closed.
  • Advanced users: verify that the bottleneck still persists in the current development version (i.e. remotes::install_github("ropensci/drake")) and mention the SHA-1 hash of the Git commit you install.

Description

One of my real-life drake workflows has 32000 dynamic sub-targets. vis_drake_graph() is slow for this, and it spends most of its time reading build times from the cache (see below). Rather than read the sub-target build times, we should just read the build time of the whole dynamic target and make a note of how many jobs were used at the time.

Benchmarks

I cannot show the code because it is confidential, but here is a flame graph.

Screen Shot 2020-03-06 at 2 19 01 PM

@wlandau
Copy link
Collaborator Author

wlandau commented Mar 6, 2020

0050de9 should do it, but I need to test on the large project to be sure.

@wlandau
Copy link
Collaborator Author

wlandau commented Mar 7, 2020

Indeed, vis_drake_graph() is several orders of magnitude faster on the full project now.

@wlandau wlandau closed this as completed Mar 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant