Loading…
Talk to Homebrew team about bottles and monkey patching #8
9 participants
Awesome. 
I'll get some notes together and ping you back after a few days of rest. Totally cool to hash it out here instead of email or IM.
Cool beans. Some of this stuff may be merged by then anyway.
@jbarnette No rush but give me a shout at some point; I've got a pull request that's implementing at least some of the features in Homebrew/homebrew#17735 and I'll get it merged when we've had a chat.
@mikemcquaid 
I'm taking a brief break in New Orleans, but I'm back in the saddle on Monday. Looking forward to hacking with you.
Cools. Have fun there and gimme a shout next week 
@jbarnette Gentle ping but no worries if you're busy. I'm just annoying 
@mikemcquaid Soon! On a plane back home now, in fact. I'll ping ya tomorrow.
Sorry to nag. Thanks (and 
for planeternet).
This would help with things like QT needing to compile from source because the Cellar path isn't within /usr/local, correct?
@shanesveller We might be able to make some milage here, yes. In short though: if you install Homebrew outside of /usr/local we will never be able to provide certain packages (like Qt) as binaries. If boxen do then great but after the Homebrew Kickstarter we're going to try and provide binaries for everything so using a non-/usr/local install will be a far worse experience.
@mikemcquaid Okay, I'm back in the saddle and almost awake. Let's do this. First, some historical things:
- Boxen manages Homebrew outside of
/usr/localbecause we're often onboarding a box with an existing install, and we want to control the whole thing. - Boxen runs all services on non-default ports, primarily to ease the path for a box with an existing install.
- ...and custom directories (
$BOXEN_HOME/{config,data,log}/service-name) so it's easy to always know where to look for stuff.
Here's what we found ourselves wanting:
- A way to specify a repository of overlay formulae. We've basically abused taps to do what we need, but it'd be way better if we could say, "hey, see this directory here? Any formulae in this directory take precedence over the regular ones." The difference between
mysqlandgithub/brews/mysqlstill causes frequent problems. We originally ran with a Homebrew fork, but keeping our divergent formulae up to date was a challenge, and certain things only worked when the origin wasmxcl/homebrew. If I remember what was busting, I'll make a note later. - A way to inject code or a script with right of first refusal on install. We have a CI machine that packages every installed package, and we needed to short-circuit the normal install process for anything we'd already dealt with.
Here's the current set of monkeypatches:
-
files/boxen-monkeypatches.rbcompletely (I think) replacesFormulaInstaller's bottle support by checking for a prebuilt file in our S3 bucket and curling it down when available. Because of Homebrew's (very legit) focus on/usr/local, it's an all-or-nothing affair. -
files/boxen-install.rbaddsbrew boxen-install, which is exactly likebrew installbut loads our patches. -
files/boxen-upgrade.rbis the same. Just abrew boxen-upgradeshim to force a load of the patches. -
files/boxen-latest.rbaddsbrew boxen-latest, which simply prints out the latest version of a formula.
boxen-install, boxen-upgrade, and boxen-latest are used by our Puppet provider to keep things relatively sane, but I don't like reaching inside Homebrew to override bottle pouring, and I especially don't like github/brews/service-name and service-name formulas coexisting.
So that's where we are, I think. Hit me.
Ok, here we go:
Boxen manages Homebrew outside of /usr/local because we're often onboarding a box with an existing install, and we want to control the whole thing.
Existing install of Homebrew? I'd strongly advise you to use /usr/local and perhaps move the old install out the way or something. Some Homebrew packages don't work outside /usr/local and after the Kickstarter we're going to be building bottles for literally every package. If you don't use /usr/local you'll need to rebuild at least some if not all of these yourselves. That seems like a bit of a waste of effort but I guess that's your call.
A way to specify a repository of overlay formulae. We've basically abused taps to do what we need
Taps should be exactly what you want to use for this. Am I correct in saying what is missing here is a way of instructing taps to be prioritised over formulae in core?
A way to inject code or a script with right of first refusal on install.
Not sure I understand this bit.
replaces FormulaInstaller's bottle support by checking for a prebuilt file in our S3 bucket and curling it down when available.
This is the main bit I have a problem with. As mentioned above this wastes effort because we aren't sharing infrastructure. Also, as already demonstrated by the changes to that file, this is really brittle and will break whenever I change the bottle logic (which I will definitely be doing in the next few months while we improve stuff around the Kickstarter). These APIs are intentionally internal; we try and keep the formula DSL relatively static but regularly do massive internal refactors which will bite you. Finally, you don't seem to have any checksumming so you're adding a fairly trivial man-in-the-middle attack to install and run arbitrary code on users machines.
My suggestion here is that we try and move any of these features you need into Homebrew properly. Looking at the list above boxen latest could be added to Homebrew fairly easily. I've got an outstanding pull request (Homebrew/homebrew#17735) which adds support for building bottles that can pull from a different URL root (i.e. your S3 bucket), specify if bottles are relocatable (i.e. can be installed outside /usr/local) and I can add something to specify the custom root for bottles (so instead of the /usr/local check it'll check against wherever you've built the bottles for).
With these bottle features I've added hopefully you should be able to get by (eventually) with no monkey-patching of Homebrew and instead just have a tap that specifies your binaries with the generated checksums (which you could have your CI autocommit; brew bottle will shortly update the formula file for you).
Hope I don't sound too grumpy here; what you guys have done is cool but I just want to avoid either of our sets of users getting pissed because of the fairly major changes Homebrew is going to make in these areas over the next few months. I'm more than happy to help with upstreaming some of this code; we're very receptive to both pull requests and actually writing new features when they are required by users of the Homebrew formula DSL.
@jbarnette ^ in case you needed pinged; never quite sure how people's notifications are setup.
I've added the custom bottle root so you can specify that inside formulae now too.
Hey @mikemcquaid,
We'll be maintaining our use of /opt/boxen/homebrew over /usr/local. One of the things people love about Boxen is that we self-contain as many things as possible, so removing Boxen is dead-simple. That said, we're not really worried about duplicating of effort in bottling things so long as we can tell our Homebrew to bias towards our bottling source — our CI box for GitHub's Boxen already automatically bottles up any packages that don't exist in our S3 bucket yet each run.
Am I correct in saying what is missing here is a way of instructing taps to be prioritised over formulae in core?
Yes. This is exactly it. Homebrew has some very idiosyncratic behavior regarding package names and taps. EG brew install boxen/brews/mysql will show up in brew list | grep mysql as mysql and brew uninstall mysql will uninstall it, but the behavior of brew install mysql is explicitly different.
We would like to be able to set priority similar to apt's pinning system so we can say "unless explicitly told otherwise, prefer a boxen tap formula if there is one."
Not sure I understand this bit. (re right of first install)
We want a hook that we can insert code to run prior to Homebrew's normal package install mechanism. A good example of this is how we integrate with rbenv/ruby-build to distribute binary builds of Ruby. We're able to give rbenv a shell script to run be run prior to its normal install action that gives us a chance to run some checks and short-circuit their install if we want to:
#!/bin/bash
VERSION_NAME="${DEFINITION##*/}"
VERSIONS_DIR="${RBENV_ROOT}/versions"
OSX_RELEASE=`sw_vers -productVersion | cut -f 1-2 -d '.'`
PREV_PWD=`pwd`
cd $VERSIONS_DIR
echo "Trying to download precompiled Ruby from Boxen..."
curl -s -f http://s3.amazonaws.com/boxen-downloads/rbenv/$OSX_RELEASE/$VERSION_NAME.tar.bz2 > /tmp/ruby-$VERSION_NAME.tar.bz2 && \
tar xjf /tmp/ruby-$VERSION_NAME.tar.bz2 && rm -rf /tmp/ruby-$VERSION_NAME.tar.bz2
cd $PREV_PWD
if [ -d "$VERSIONS_DIR/$VERSION_NAME" ]; then
# install from download was successful
rbenv rehash
exit 0
fi
I've got some more stuff to respond with later, but there's a cursory dump.
Yes. This is exactly it. Homebrew has some very idiosyncratic behavior regarding package names and taps.
Yeah, the main culprit here is probably the Formula::canonical_name method---a wild and hairy beast that has evolved organically over time and is probably in need of a refactor.
Yep, I agree with you on tap behaviour. I'll have a look at some point on amending this (or seeing if there's a way of setting a tap as primary). Currently the best way of doing this is just to keep a fork of the main Homebrew repo but I think you've said this wasn't really working for you.
I'm still not sure I understand what the right of first install is; you could just do the above outside of Homebrew (extract stuff into the Cellar) and then just run brew link and only run brew install if your own shell scripts fail. That seems a nicer solution than monkey patching Homebrew. Given that bottles now support custom URL roots and cellars that should probably support your use case here without the need for either of the above once the tap issue is fixed and just maintain a single tap for anything you have binary packages for. This will also let you checksum your binary downloads which would be a good thing.
@jbarnette @wfarr I've added (hopefully) decent documentation for the current state of bottles. This should enlighten a bit on what's possible with/without monkey-patching:
https://github.com/mxcl/homebrew/wiki/Bottles
We got a user report today (Homebrew/homebrew#18890) related to the bottle monkey patching. It would be good, I think, for both parties if we could work out a way of resolving this. Due to the Kickstarter the bottle code is going to be touched a lot in the next few months so, in it's current state, sadly Boxen's Homebrew support is going to break lots.
So... is there any chance that there will ever be a public or community source for boxen-compatible bottles? It's super cool that I could host my own (and that Github can host their own), but I simply don't have enough machines for that to be worthwhile. Which means that for small groups of machines, boxen currently removes all the benefits provided by homebrew allowing precompiled bottles for everything. :(
@indirect You could do the same thing with Homebrew bottles now that they support non-/usr/local prefixes and URL prefixes in the formulae themselves. It would make it fairly easy to have a community run bottle server. See here for more details: https://github.com/mxcl/homebrew/wiki/Bottles
I'd love to see this resolved in some way. Even just allowing a setup option for Boxen to use/install Homebrew at /usr/local would be really nice from my perspective. I know others may have existing installs there, but it'd be good to allow this sort of thing if we so desired.
Agreed that it would be a good user-configurable option so people can make use of Homebrew bottles.
Any updates on this?
Nothing on my end I'm afraid.
Sad day, as it looks like the monkey patches are breaking with 10.9. It'd be nice if boxen could simply ensure home brew is installed if it's not, and just always use homebrew as a package manager. Puppet on Debian systems doesn't try to install its own copy of apt-get, so it doesn't make sense to do so here either.
Puppet on Debian systems doesn't try to install its own copy of apt-get, so it doesn't make sense to do so here either.
That's because apt-get comes bundled on every Debian system by default.
@wfarr Very true. In my opinion, I think with boxen, we should set a precedent that home brew be installed by default (as it's our OS X package manager). If it's not installed, boxen should install it and inform the user as such. Maintenance-wise (as we're seeing with 10.9) it seems like it's going to be messier to handle two instances of home brew, or even just one if it's boxen's home brew in a non-default location. Although I do appreciate boxen being self-contained.
Closing this as @dgoodlad and I have had some productive conversations.
Any place we can see a summary or plan that resulted from that?
@indirect There isn't one really. I've mostly closed this because there's nothing particularly happening here. We've agreed that longer-term we want to work together to get rid of the monkey patching and try and collaborate on the binary situation.
On the Homebrew side we're heavily ramping up the number of binary packages we ship and trying to make as many as possible work with cellar :any (i.e. outside /usr/local). All our binary packaging is automated and open-source so it should be fairly straightforward for others to set up e.g. a community tap where they automatically bottle formulae. Feel free to contact me directly if you want to talk more about the Homebrew side of things.
@mikemcquaid awesome! I do actually have Homebrew installed in /usr/local, I just also have it in /opt/boxen/homebrew. I'm completely convinced that the Boxen Way™ of just being able to rm -rf /opt/boxen and then clone the repo and run again is great. It just doesn't work very well with hardcoded compiled paths. :/ Thanks for working on this!
@indirect You can also do cd /usr/local; git clean for a similar result but your call obviously. Homebrew installs outside /usr/local will always be second-class citizens unfortunately due to, as you pointed out, hardcoded compiled paths.
Following on from b34506a:
I'm the Homebrew guy who created bottles and just wanted to talk about this a bit as the bottle code changes often enough that I'm going to end up breaking your monkeypatching regularly. Can we talk a bit here (or by IM or mail) what you want to do with binaries and we'll work out a way to integrate this into Homebrew?
I think ideally we'll eliminate all monkeypatching from Homebrew and move any changes that you need into Homebrew itself.
When the Homebrew Kickstarter completes then we'll have CI hardware and be bottling a lot more stuff so we maybe want to try and collaborate on this (I'm happy for us to share S3 hosting and upload all our created bottles there).
In the shorter term if you're just looking for support for installing bottles from custom URLs I have commits locally which I was planning on merging in the next week that do that.
Feel free to give me a shout by mail or IM if that's easier.