Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

A patch for dynamic linking for Mac #1660

Closed
kazu-yamamoto opened this Issue · 7 comments

6 participants

@kazu-yamamoto

As discussed in https://ghc.haskell.org/trac/ghc/ticket/8266, one patch should be applied before GHC 7.8 is released.

diff --git a/Cabal/Distribution/Simple/GHC.hs b/Cabal/Distribution/Simple/GHC.hs
index c7ea633..78cdcbb 100644
--- a/Cabal/Distribution/Simple/GHC.hs
+++ b/Cabal/Distribution/Simple/GHC.hs
@@ -867,11 +867,6 @@ buildOrReplLib forRepl verbosity pkg_descr lbi lib clbi = do
               ghcOptDynLinkMode        = toFlag GhcDynamicOnly,
               ghcOptInputFiles         = dynamicObjectFiles,
               ghcOptOutputFile         = toFlag sharedLibFilePath,
-              -- For dynamic libs, Mac OS/X needs to know the install location
-              -- at build time.
-              ghcOptDylibName          = if buildOS == OSX
-                                          then toFlag sharedLibInstallPath
-                                          else mempty,
               ghcOptPackageName        = toFlag pkgid,
               ghcOptNoAutoLinkPackages = toFlag True,
               ghcOptPackageDBs         = withPackageDB lbi,

Would you merge this patch ASAP?
The credit should go to darchon.

@hvr
Owner

just wondering: does this patch have any negative impact when used in combination with a GHC prior to 7.8? I.e. do we have to introduce a conditional based on the GHC version used?

@christiaanb

It does have a negative impact on GHC prior to 7.8: prior versions would make the install_name of a library it's build location, instead of it's install location.

So you would have to use a conditional based on the GHC version used. Good catch!

See https://ghc.haskell.org/trac/ghc/ticket/8266#comment:43 for the purpose of the patch.
TL;DR The Cabal patch is not needed, it just makes dynamic library paths more uniform in GHC 7.8.

@23Skidoo
Collaborator

I see that @dcoutts has some objections to this. Leaving this for him to review.

@dcoutts
Collaborator

According to https://ghc.haskell.org/trac/ghc/ticket/8266#comment:43, this Cabal patch is not necessary.

Honestly I'm still trying to understand what exactly is going on and how the OSX rpath stuff works. Given that it's apparently not necessary, and it has backwards compatibility issues and that it's not totally clear (at least to me) then I think we should hold off.

@dcoutts
Collaborator

Ok, so having dug into it in more detail, here is the summary:

ELF

First, how ELF does it, so the differences in OSX's MachO are clearer.

In an ELF exe or shared lib, it records the names of the libs it depends on, e.g. "libfoo.so.1". It finds that lib name at link time by looking in the library file (the library file embeds its own name).
Most of the time that's enough because most libs are found in /lib or /usr/lib.

There is a mechanism to find libs not in standard locations, each exe or lib has a list of directories it searches to find the libs it depends on (by their name). This is called the "rpath" (technically DT_RPATH or DT_RUNPATH).

So the point is that when I make an exe or lib, I tell it where to look for its dependencies at runtime.
(Note also that rpaths can be relative to the location of the lib/exe).

OSX / MachO

Historically, OSX finds libs at runtime in a rather backwards way. In ELF, when I make an exe or lib, I tell it the dirs to search at runtime to find its libs (the rpath). In MachO when I make an exe or lib, the runtime location for the libs it depends on are found by looking in those libs (at link time), which embed their own location.

Let's look at it another way: in ELF when I create a lib, I tell the lib what its name is, e.g. "libfoo.so.1" where as with MachO I tell it what it's full install location is, e.g. "/opt/foo/lib/libfoo.dynlib". So ELF libs embed just their name, while MachO libs embed their location (though this is called its "install name"). Historically that location could only be a full absolute path, meaning that the libs cannot be relocated.

But also notice that it's backwards, we specify where to find a lib when we create the lib, rather than when we make something that depends on it and will need to find it.

Anyway, so historically Cabal has followed this scheme and has linked dynlibs on OSX by calling ghc with -dylib-install-name with the full final install location, as required. This has meant that a binary ghc build has to pick where it's going to be installed, e.g. /usr/local and if you instead installed it into /opt/ghc then the dynamic linking will not work.

Now apparently Apple engineers noticed that this was very annoying and added a couple mechanisms to make it a bit more ELF like. Specifically, the install name embedded into a lib no longer has to be the full absolute path, but can be relative to certain variables understood by the runtime linker. The useful one (available since OSX 10.5) is @rpath. So instead of making a lib with an install name of /usr/local/lib/libfoo.dynlib we can instead say @rpath/libfoo.dynlib. This more or less makes it equivalent to the ELF approach of only embedding the name into the lib itself, and not the full location. So of course we want to know what that @rpath means at runtime. It gets more ELF-like. An exe or lib can now embed an rpath, and any @rpath-relative libs we depend on will be looked for in the rpath dirs.

So as of GHC 7.8, when linking a shared lib it will by default give it an @rpath-relative install name (unless we override it on the command line with -dylib-install-name). Also, when linking an exe or shared lib it will embed an rpath into the exe/lib containing a list of all the dirs of all the libs we depend on.

So what this means is that as of GHC 7.8 Cabal can stop specifying the final absolute install location as the install name, and let GHC set the '@rpath'-relative one. The @rpath feature is available on OSX 10.5 and later, and 7.8 only supports later versions anyway.

Summary

The patch is fine, but it must be altered to be conditional for GHC 7.8+.

There is no immediate critical rush to get this included. In almost all cases using absolute paths for cabal-installed libs works fine. However since linker things can have unintended consequences it would be good to get as wide testing as possible, so including it into some 7.8 RC is probably good, but it doesn't need to be the first one.

@23Skidoo 23Skidoo referenced this issue from a commit in 23Skidoo/cabal
@23Skidoo 23Skidoo Don't use -dylib-install-name on OS X with GHC > 7.8.
Fixes #1660.
41fc807
@23Skidoo 23Skidoo referenced this issue from a commit in 23Skidoo/cabal
@23Skidoo 23Skidoo Don't use -dylib-install-name on OS X with GHC > 7.8.
Fixes #1660.
cb96670
@23Skidoo 23Skidoo referenced this issue from a commit in 23Skidoo/cabal
@23Skidoo 23Skidoo Don't use -dylib-install-name on OS X with GHC > 7.8.
Fixes #1660. Based on a patch by Christiaan Baaij.
6efad17
@23Skidoo 23Skidoo referenced this issue from a commit in 23Skidoo/cabal
@23Skidoo 23Skidoo Don't use -dylib-install-name on OS X with GHC > 7.8.
Fixes #1660. Based on a patch by Christiaan Baaij.
94b5159
@christiaanb

Actually, it turns out that dynamic linking was actually sort-of broken prior to GHC 7.8: https://ghc.haskell.org/trac/ghc/ticket/8721

@23Skidoo 23Skidoo closed this in #1661
@23Skidoo 23Skidoo referenced this issue from a commit
@23Skidoo 23Skidoo Don't use -dylib-install-name on OS X with GHC > 7.8.
Fixes #1660. Based on a patch by Christiaan Baaij.
ee6d1cf
@hvr hvr referenced this issue from a commit in ghc/ghc
@hvr hvr Update to latest Cabal 1.18 branch tip
This update pulls in the fix for #8266

(recommended add-on reading for those interested in OSX linker peculiarities:
haskell/cabal#1660 (comment) )

Signed-off-by: Herbert Valerio Riedel <hvr@gnu.org>
5671ad6
@hvr hvr referenced this issue from a commit in ghc/ghc
@hvr hvr Update to latest Cabal 1.18 branch tip
This update pulls in the fix for #8266

(recommended add-on reading for those interested in OSX linker peculiarities:
haskell/cabal#1660 (comment) )

Signed-off-by: Herbert Valerio Riedel <hvr@gnu.org>
(cherry picked from commit 5671ad6)
37d6e2c
@mchakravarty

@dcoutts Just some clarification re OS X programming.

So ELF libs embed just their name, while MachO libs embed their location (though this is called its "install name"). Historically that location could only be a full absolute path, meaning that the libs cannot be relocated.

OS X has a tool called install_name_tool that let's you alter the install name of a library (including rpath), see http://www.unix.com/man-page/OSX/1/install_name_tool/ (note also how there can be more than one rpath). OS X frameworks and applications use that to relocate dynamic libraries.

Secondly, there is a reason for this "backwards" mechanism. OS X libraries and applications are typically distributed in, so called, bundles. These are directories that contain all the various pieces of a library (aka framework) or application. If there is a dynamic library in a bundle, you can have an rpath that leaves the location of the bundle flexible, but fixes the location of the library within the bundle.

@nomeata nomeata referenced this issue from a commit in nomeata/ghc-complete
@nomeata nomeata Changes to ghc(3)
Changes to ghc:
commit f30a01626461c3824aa6bccb1ed633bee26382bc
Author: Austin Seipp <aseipp@pobox.com>
Date:   Tue Feb 4 05:17:15 2014 -0600

    That should have been 7.8.0

    Signed-off-by: Austin Seipp <aseipp@pobox.com>

commit 9976c2ece3aa43dea21879e16dd911bb9c16e6b2
Author: Austin Seipp <aseipp@pobox.com>
Date:   Tue Feb 4 05:13:35 2014 -0600

    Bump version to 7.8.1

    The version for RC2 will appear as '7.8.1.<date>'

    Signed-off-by: Austin Seipp <aseipp@pobox.com>

commit 37d6e2c54f117f0a65f3032b6f30b6878b3f6f58
Author: Herbert Valerio Riedel <hvr@gnu.org>
Date:   Fri Jan 31 22:30:00 2014 +0100

    Update to latest Cabal 1.18 branch tip

    This update pulls in the fix for #8266

    (recommended add-on reading for those interested in OSX linker peculiarities:
    haskell/cabal#1660 (comment) )

    Signed-off-by: Herbert Valerio Riedel <hvr@gnu.org>
    (cherry picked from commit 5671ad66b8c938939a44c883002caa4e13be098c)
b5e12ab
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.