Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

brew linkage should be faster (if possible) #3008

Closed
MikeMcQuaid opened this issue Aug 5, 2017 · 6 comments

Comments

@MikeMcQuaid
Copy link
Member

commented Aug 5, 2017

If brew linkage was hilariously fast then we could run it globally more often and automatically detect when e.g. a brew reinstall is needed because the user was using an optional dependency. Ideally on an SSD and modernish hardware it should be able to scan ~50 packages (with some large ones like boost) in under 5 seconds. This may require heavy caching or multiprocess parallelisation outside of Ruby (because Ruby Threads aren't really good enough for this). brew prof uses is probably the best way to optimise this.

@armcburney

This comment has been minimized.

Copy link
Contributor

commented Jan 17, 2018

Hi @MikeMcQuaid,

I haven't contributed to Homebrew before, but I'd like to start contributing and figured this would be a good first issue to tackle.

I've been running the brew linkage command on my local machine (2017 MacBook Pro) with over 50 packages (including boost). In the issue description, you said ideally you'd like the command to run in under 5 seconds. However, the command has been finishing close to 5 seconds, making it difficult to compare benchmarking with optimizations I've done.

Does Homebrew have any sort of existing caching which would increase performance the more times a dev-command like brew linkage is run? If there is caching, what's the best way to clear the cache so I may resume comparing benchmarks with optimizations I've made?

Input

 time brew linkage gtk autoconf docker-machine gimme graphite2 libffi little-cms2 pcre readline automake dockutil git harfbuzz libpng nettle pixman redis bash emacs-plus glib hugo librsvg node pkg-config ruby-build boost fontconfig gmp icu4c libtasn1 nvm poppler sqlite cairo freetype gnu-sed imagemagick@6 libtiff openjpeg postgresql texinfo coreutils gdbm gnutls ispell libtool openssl pyenv wxmac docker gdk-pixbuf go jpeg libunistring p11-kit python xz docker-compose gettext gobject-introspection libcroco libyaml pango rbenv yarn

Output

...

brew linkage gtk autoconf docker-machine gimme graphite2 libffi little-cms2    3.12s user 2.49s system 96% cpu 5.843 total

I've been running the brew linkage command quite a bit, so if there's caching it would explain why the command executes faster than I expected.

Thank you!

@alyssais

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2018

Weird. My email reply to this issue didn’t show up. I’ll paste it below, but apologies if it somehow ends up duplicated or something.


I haven't contributed to Homebrew before, but I'd like to start contributing and figured this would be a good first issue to tackle.

👋🙂

Does Homebrew have any sort of existing caching which would increase performance the more times a dev-command like brew linkage is run? If there is caching, what's the best way to clear the cache so I may resume comparing benchmarks with optimizations I've made?

I’m pretty sure it doesn’t (currently). 🤔

I’ll let Mike reply with how long he’d expect it to take, but fwiw running linkage over my 150 packages took 27.5 seconds.

@ilovezfs

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2018

40.5 seconds for 121 packages here. 💤 😴

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

commented Jan 18, 2018

So, another option for this would be to do a half-and-half on this and #3007 and have brew upgrade (and maybe brew install) go through all the reverse runtime_dependencies from the Tab and just check brew linkage on them. If this operation can be done in, say, <5s with 100 packages installed and returning 1-5 results then I think that would be fast enough to be useful.

@armcburney

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2018

Hi all!

I created a WIP PR to improve the performance of the brew linkage command by using SQLite caching. I'd love to get some feedback on it. 🙂

There are still a few outstanding issues outlined in the PR description. Any advice/help would be greatly appreciated.

All the best,
Andrew

@MikeMcQuaid

This comment has been minimized.

Copy link
Member Author

commented Jul 5, 2018

This has now been fixed and is enabled for all users . Great work @AndrewMcBurney!

@MikeMcQuaid MikeMcQuaid closed this Jul 5, 2018

@ghost ghost removed the help wanted label Jul 5, 2018

@lock lock bot added the outdated label Aug 4, 2018

@lock lock bot locked as resolved and limited conversation to collaborators Aug 4, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
4 participants
You can’t perform that action at this time.