Enable upgrades of keg-only libraries

[adamv]

Keg-only libraries like libpng and readline are referenced by a Cellar path that contains the version number.

This prevents those libraries from being upgraded without breaking existing software compiled against those paths.

For keg-only libraries, we either need to allow multiple versions to be installed (not idea) or create a special "Current" symlink in the keg's prefix that points to the current version. (Idea based on how the internal structure of a Framework looks.)

This way, software could reference kegs via the "Current" path instead of the versioned path, and survive past library upgrades (assuming binary compatibility.)

See also:

• http://github.com/mxcl/homebrew/issues/issue/693
• http://github.com/mxcl/homebrew/issues/issue/948

(But note that libpng 1.4 isn't always backwards-compat with 1.2.)

[adamv]

Perhaps "brew cleanup" should skip keg-only formulae for the time being.

[justinclift]

Having a "current" path like this sounds pretty much guaranteed to (silently) create broken applications. i.e. in the example above, if libpng was upgraded from 1.2 to 1.4, then things using it would just break. Cue sentiment of "Homebrew allows broken packages" and similar. (not a road to a quality reputation)

As a first thought, having some way of accurately tracking library dependencies, so that if (say) libpng was upgraded from 1.2 to 1.4, then the packages that use it would be automatically recompiled too. That might allow for less breakage, though it would need to be done in a way that ensures the correct order of recompilation.

i.e. say we have two packages, libfoo and bar, which both depend on libpng. If bar also depends upon libfoo, but bar gets rebuilt before libfoo, then bar would be getting built against a still-broken libfoo. So, we still don't have a good result. Maybe we could use some magic with ldd/nm/etc to work out dependencies? (this is beyond my skill set)

[adamv]

I would rather never upgrade some things rather than "automatically recompile", as that automatic recompile for gettext would be "half of everything in Homebrew" and take forever.

[justinclift]

Yeah, having some things never upgraded (or generally skipped) sounds like the safer approach in the short term.

In the longer term, it might be worth having something along the lines of a "keg-upgrade" command though, which is known (and advertised) to take a long time, as it specifically upgrades (all of the?) kegs and then recompiles anything depending upon them.

That's all theory though, and would need someone to actually code it and see how well it works. ;)

[mxcl]

So basically, currently everything is fine as long as the user doesn't brew cleanup. And we disabled that for keg-only anyway.

As an aside, I note that often the install_name is wrong for non-keg-only stuff. So cleanup broke anything with old library deps, often. So I fixed that just now.

But this bug is in fact not urgent?

[adamv]

Give that we disabled cleanup of keg-only brews, I no longer feel this issue is urgent or really worth spending time on. The downside is that a user may accumulate some unused keg-only brews over time, but typically these do not take up enough space to matter.

[mxcl]

I worse downside is that potentially security fixes won't get applied. But we are a bunch of cowboys, and aren't hugely fussed.

But I still have to fix it, so I'll reopen it.

[adamv]

Ok, there should probably be a (new command?) thing that walks all bin/dylib/so files in the Brew cellar, does otool on them or equivalent, and looks for linkages to keg-only brews, warns if newer versions are available, etc.

[mxcl]

Easier to just fix the install names when installing keg_onlys for anything that depends on that keg_only formula?

[adamv]

Is this still an open issue / something we want to do?

[mxcl]

Yes, this is something that should be done. Reinstalling all formula because of a security update for readline would suck.

[jmlane]

I am interested in this issue and would like to contribute to the resolution. I may need a bit of clarification on how brews and formula rely on keg_onlys. I am a bit uncertain about how rigid the linking process may be when brewing is dependent on a keg_only brew.

With the current strategy, if a keg_only formula is updated to a newer version, does that necessarily allow the previous version to be safely removed? I suspect the dependent brews would have to be rebrewed to link against the newest keg_only before we can safely remove the older one. I have no sense of how we could trivially verify when a keg_only of a specific version is no longer needed (otool calls mentioned above seem heavy-handed).

If homebrew really cannot safely remove outdated keg_only brews once a newer version is on the system without rebrewing all dependent brews, does that not seem to indicate we should require keg_only dependencies to be specific about which version of the package is needed? Alternatively, what would bring us closer to being able to safely brew cleanup outdated and no longer necessary keg_only brews?

[jacknagel]

I suspect the dependent brews would have to be rebrewed to link against the newest keg_only before we can safely remove the older one.

Correct (under the current system).

I think the most robust approach would probably be as @mxcl stated above (given a keg-only formula foo, fix the install names for things that depend on foo).

[jmlane]

Thanks Jack.

What kind of system would be needed to avoid having to rebrew packages that rely on a keg_only package that is likely to be updated often?

[Sharpie]

Trying to pull a lot of tricky business with otool and install_name_tool seems like a house of cards to me. All it would take is a few keg_only releases that break ABI compatibly and our issue tracker would blow up with all sorts of weird problems.

Seems better, and simpler, to just have brew cleanup ignore kegs and leave it to the user to re-build when they want something updated.

[jacknagel]

All it would take is a few keg_only releases that break ABI compatibly and our issue tracker would blow up with all sorts of weird problems.

Sure, but we already run that risk by allowing library upgrades without a system to force rebuilds or upgrades of things that link to that library.

[Sharpie]

Sure, but we already run that risk by allowing library upgrades without a system to force rebuilds or upgrades of things that link to that library.

How so? Since brew cleanup does not remove old keg_only installs, everything that was linked to the old install remains linked to that install---so there should be no breakage due to changes in the ABI of the new version.

[jacknagel]

What I mean is that Homebrew software that links against non keg-only libraries is already susceptible this kind of breakage when the library is upgraded.

[Sharpie]

Ahh... but given that non-keg-only stuff is purged by brew cleanup while keg-only stuff is spared, that is a separate issue is it not?

[jacknagel]

I guess my point is that:

• when package foo is installed, we fix the install names so that it is possible to upgrade the libraries that foo uses without breaking foo;
• foo is still susceptible to breakage if the updated library is not ABI- or API-compatible;
• currently, keg-only linkages are safe from this, because the libraries are referenced through a specific, versioned keg; and
• enabling upgrades of keg-only libraries would put things that link against them in, at the very worst, the same situation that things that link against non-keg-only libraries already are.

[MikeMcQuaid]

I think we should try and fix this stuff where we can. We won't get it any worse than any Linux distribution does and if stuff breaks then it's a bug that should be fixed. Would be nice to not have to keep old versions around "just in case".

[mxcl]

@Sharpie is right, however:

1. Every Linux distribution on the planet runs the same risks, so we wouldn't be alone if such a thing happened.
2. Library authors are aware of this and don't break ABI compat intentionally without bumping the major soname.
3. Major libraries are very aware of this and the major ones are the only ones with many deps.

Finally, I came here to comment as superenv branch contains the /opt feature which will fix this. I apologize that it wasn't submitted separately for review. Essentially it does what adam suggested above, though at the root prefix level instead of in individual kegs.

This is pretty neat though as it turns out. I like stepping into opt and having a tree of all my software.

[mxcl]

This is now merged, was part of superenv, the opt bit. The cleanup command is smart and doesn't remove kegs that weren't built when the opt feature existed. So should be safe.