Add `--dynlibdir` #3979

christiaanb · 2016-10-13T12:51:21Z

--dynlibdir indicates the directory in which dynamic libraries
are installed. By default this setting is equal to:

$libdir/$abi

The static libraries will still end up in:

$libdir/$libsubdir

With $libsubdir/$abi as the default directory for dynamic
libraries, dynamic libraries will by default end up in a
single shared directory (per package database). This has the
potential to reduce start-up times for dynamically linked
executable as only one RPATH per package database will be
needed.

mention-bot · 2016-10-13T12:51:24Z

@christiaanb, thanks for your PR! By analyzing the history of the files in this pull request, we identified @ezyang, @23Skidoo and @dcoutts to be potential reviewers.

bgamari · 2016-10-13T15:28:06Z

Thanks @christiaanb!

23Skidoo

Looks good modulo minor comments.

23Skidoo · 2016-10-13T13:18:26Z

Cabal/Distribution/Simple/LocalBuildInfo.hs

 import Distribution.Text
 import qualified Distribution.Compat.Graph as Graph

 import Data.List (stripPrefix)
 import System.FilePath
 import qualified Data.Map as Map

-import System.Directory (doesDirectoryExist, canonicalizePath)
+import System.Directory (doesDirectoryExist, canonicalizePath, getDirectoryContents)


Formatting seems off.

23Skidoo · 2016-10-13T13:22:13Z

Cabal/Distribution/Simple/LocalBuildInfo.hs

@@ -273,6 +274,19 @@ depLibraryPaths inplace relative lbi clbi = do

        allDepLibDirs' = internalLibs ++ allDepLibDirs
    allDepLibDirsC <- traverse canonicalizePathNoFail allDepLibDirs'
+    -- Only get the directories that contain dynamic libraries
+    let soExtension = case buildOS of


You can use dllExtension from D.S.BuildPaths here.

ilovezfs · 2016-10-14T08:53:30Z

@christiaanb Will there be a Sierra solution for the static libraries too?

christiaanb · 2016-10-14T09:12:45Z

@ilovezfs There is a problem with static libraries on Sierra????

ilovezfs · 2016-10-14T09:21:26Z

Probably not ... just trying to get git-annex to build in a cabal sandbox using this PR and still getting stuck at yesod-auth, but I'm probably doing something wrong.

christiaanb · 2016-10-14T12:38:44Z

Closing this PR, as it lead to problems with RPATH calculations in GHC. The definitive solution for https://ghc.haskell.org/trac/ghc/ticket/12479 should be #3982. I'm terribly sorry that it took 3 PRs to fix this.

23Skidoo · 2016-10-14T12:40:28Z

I'm terribly sorry that it took 3 PRs to fix this.

Oh, you shouldn't be. We are all very grateful to you for working so hard on this.

dcoutts · 2016-10-16T17:48:18Z

Cabal/Distribution/Simple/LocalBuildInfo.hs

@@ -266,24 +268,37 @@ depLibraryPaths inplace relative lbi clbi = do
    -}
        getLibDir sub_clbi
          | inplace    = componentBuildDir lbi sub_clbi
-          | otherwise  = libdir (absoluteComponentInstallDirs pkgDescr lbi (componentUnitId sub_clbi) NoCopyDest)
+          | otherwise  = dynlibdir (absoluteComponentInstallDirs pkgDescr lbi (componentUnitId sub_clbi) NoCopyDest)


I don't get this bit.

We'll need to resolve this.

Well, that entire function is there to find the locations of dynamic libraries. Since libdir now only contains the location of static libraries, we should use dynlibdir.

dcoutts · 2016-10-18T18:52:23Z

So we're planning to resurrect this approach and make ghc do the right thing.

See https://phabricator.haskell.org/D2611

dcoutts

Not that much to do to get this working with the new approach based on using dynamic-library-dirs.

dcoutts · 2016-10-18T18:54:54Z

Cabal/Distribution/Simple/Register.hs

-                               then libdir installDirs : extraLibDirs bi
-                               else                      extraLibDirs bi,
+                               then libdir installDirs    :
+                                    dynlibdir installDirs : extraLibDirs bi


This bit will need to be extended to set the libraryDynDirs if that's supported by the target compiler, and otherwise put both libdir and dynlibdir in the libraryDirs as the patch does now.

dcoutts · 2016-10-18T18:58:22Z

Cabal/tests/PackageTests/Tests.hs

@@ -779,18 +779,20 @@ tests config = do
                uid = componentUnitId (targetCLBI target)
                dir = libdir (absoluteComponentInstallDirs pkg_descr lbi uid
                              NoCopyDest)
+                dyndir = dynlibdir (absoluteComponentInstallDirs pkg_descr lbi
+                                    uid NoCopyDest)


we should share this better, e.g.

InstallDirs{libdir,dynlibdir} = absoluteComponentInstallDirs pkg_descr lbi uid NoCopyDest

dcoutts · 2016-10-18T18:59:53Z

Cabal/Distribution/Simple/LocalBuildInfo.hs

+    internalLibsC <- traverse canonicalizePathNoFail internalLibs
+
+    let allDepLibDirs'   = internalLibsC ++ allDepLibDirsC
+        p                = prefix installDirs


So all this stuff can be reverted.

dcoutts · 2016-10-18T19:00:15Z

Cabal/Distribution/Simple/LocalBuildInfo.hs

@@ -266,24 +268,37 @@ depLibraryPaths inplace relative lbi clbi = do
    -}
        getLibDir sub_clbi
          | inplace    = componentBuildDir lbi sub_clbi
-          | otherwise  = libdir (absoluteComponentInstallDirs pkgDescr lbi (componentUnitId sub_clbi) NoCopyDest)
+          | otherwise  = dynlibdir (absoluteComponentInstallDirs pkgDescr lbi (componentUnitId sub_clbi) NoCopyDest)


We'll need to resolve this.

dcoutts · 2016-10-18T19:05:07Z

Other dynamic-library-dirs changes in https://github.com/dcoutts/cabal/tree/osx-dyn-linking-hackery branch

dcoutts

This is looking really good.

dcoutts · 2016-10-19T11:26:49Z

Cabal/Distribution/Simple/LocalBuildInfo.hs

@@ -266,24 +268,37 @@ depLibraryPaths inplace relative lbi clbi = do
    -}
        getLibDir sub_clbi
          | inplace    = componentBuildDir lbi sub_clbi
-          | otherwise  = libdir (absoluteComponentInstallDirs pkgDescr lbi (componentUnitId sub_clbi) NoCopyDest)
+          | otherwise  = dynlibdir (absoluteComponentInstallDirs pkgDescr lbi (componentUnitId sub_clbi) NoCopyDest)


dcoutts · 2016-10-19T11:46:40Z

Cabal/Distribution/Simple/Register.hs

+    comp = compiler lbi
+    supportsLibraryDynDir = case compilerFlavor comp of
+      GHC -> compilerVersion comp > mkVersion [8,0,1]
+      _   -> False


We now try to avoid sprinkling ghc version tests throughout the code. They now all live in the GhcImplInfo record (from Distribution.Simple.GHC.ImplInfo) or for (g)hc-pkg they live in the HcPkgInfo record from Distribution.Simple.Program.HcPkg.

So initially I was going to say that this is a hc-pkg feature, and so it should be defined as a new field in HcPkgInfo, e.g. supportsLibDynDirs. However, I've changed my mind because it will be quite awkward since we cannot trivially get the HcPkgInfo value for the compiler, since not all of them support a hc-pkg.

@23Skidoo what do you reckon? In principle, perhaps this belongs in the Compiler value (from D.Simple.Compiler) since this question is valid whether or not the compiler uses a hc-pkg program. Indeed perhaps the Compiler type should contain a Maybe HcPkgInfo.

@christiaanb so if @23Skidoo agrees with me then I think you'd add this as a Bool feature of the Compiler type from Distribution.Simple.Compiler, and set it appropriately in the per-compiler configure functions.

Alternatively, could follow the approach of your TODO above and add per-compiler InstalledPackageInfo -> InstalledPackageInfo munging functions. Indeed we might be able to move the ABI stuff into those.

I'd make libDynDirsSupported a function of type Compiler -> Bool instead of a field, which seems to be more in line with existing precedent (see profilingSupported and coverageSupported in D.S.Compiler). The alternative is probably also fine.

I created a function D.S.Compiler.supportsLibraryDynDir in the latest version of the PR.

dcoutts · 2016-10-19T11:51:38Z

cabal-install/Distribution/Client/Setup.hs

@@ -390,11 +391,15 @@ filterConfigureFlags flags cabalLibVersion
      configAllowNewer  = Just (Cabal.AllowNewer Cabal.RelaxDepsNone)
      }

+    -- Cabal < 1.25.0 doesn't know about --dynlibdir.
+    flags_1_25_0 = flags_latest { configInstallDirs = configInstallDirs_1_25_0}
+    configInstallDirs_1_25_0 = (configInstallDirs flags) { dynlibdir = NoFlag }


@23Skidoo I'm never too confident about the logic here. Can you double check.

filterConfigureFlags changes are solid IMO.

dcoutts · 2016-10-19T11:52:15Z

cabal-install/Distribution/Client/Setup.hs

@@ -423,7 +428,7 @@ filterConfigureFlags flags cabalLibVersion
    -- Cabal < 1.18.0 doesn't know about --extra-prog-path and --sysconfdir.
    flags_1_18_0 = flags_1_19_1 { configProgramPathExtra = toNubList []
                                , configInstallDirs = configInstallDirs_1_18_0}
-    configInstallDirs_1_18_0 = (configInstallDirs flags) { sysconfdir = NoFlag }
+    configInstallDirs_1_18_0 = (configInstallDirs flags_1_19_1) { sysconfdir = NoFlag }


Is this an independent bug fix?

It wasn't a "bug" until I added another configInstallDirs restriction. I mean, flags is the completely unrestricted flags, we should be using the restricted version: flags_1_19_1.

dcoutts · 2016-10-19T12:13:39Z

Cabal/Distribution/Simple/Register.hs

+                                      then dynlibdir installDirs :
+                                           extraLibDirs bi
+                                      else extraLibDirs bi
+                               else [],


I think it might be easier to pull these out to the where clause (like the absinc, relink) and share the logic, e.g.

(libdir, dynlibdir) | not hasLibrary = (extraLibDirs bi, []) -- the dynamic-library-dirs defaults to the library-dirs if not specified, -- so this works if the dynamic-library-dirs field is supported or not | supportsLibraryDynDir comp = (libdir installDirs : extraLibDirs bi, dynlibdir installDirs : extraLibDirs bi) | otherwise = (libdir installDirs : dynlibdir installDirs : extraLibDirs bi, []) -- the compiler doesn't understand the dynamic-library-dirs field so we -- add the dyn directory to the "normal" list in the library-dirs field

but do double check I've transcribed the logic correctly! :-)

dcoutts · 2016-10-19T12:24:43Z

Cabal/Distribution/Simple/Register.hs

+    -- more compilers start to support this, we should drop the
+    -- `supportsLibraryDynDir` check here, and just handle `IPI.libraryDynDirs`
+    -- in the actual package registration function belonging to those particular
+    -- compilers.


Unfortunately we cannot do this yet given the structure of the code. We need to be able to generate the InstalledPackageInfo so we can write it to a file or script, without ending up calling the compiler specific registerPackage. What would be nice is if we had a compiler specific bit of generating the registration info, then we could do this compat bit there.

FYI, doing that might be an alternative to the thing I suggest below about compiler version tests.

23Skidoo · 2016-10-19T13:50:15Z

AppVeyor failure seems to be real.

ezyang · 2016-10-19T20:29:42Z

It looks like this regressed tests on GHC HEAD https://travis-ci.org/haskell/cabal/jobs/168959154

christiaanb · 2016-10-20T08:28:17Z

@ezyang Yes... this patch expects https://phabricator.haskell.org/D2611 to be merged into HEAD. What is the proper way to handle this?

23Skidoo · 2016-10-20T08:39:13Z

@christiaanb I think we can use the same approach as in a088119 - change the version check from > 8.0.1 to > 8.0.1.201610XX.

23Skidoo · 2016-10-20T08:41:47Z

Cabal/Distribution/Simple/Compiler.hs

@@ -321,6 +322,13 @@ unitIdSupported = ghcSupported "Uses unit IDs"
 backpackSupported :: Compiler -> Bool
 backpackSupported = ghcSupported "Support Backpack"

+-- | Does this compiler support a package database entry with:
+-- "dynamic-library-dirs"?
+supportsLibraryDynDir :: Compiler -> Bool


Bikeshedding: libraryDynDirSupported would be more in line with existing precedent (see above).

23Skidoo · 2016-10-20T08:53:20Z

Though I don't get why it's failing with 8.0.1 if the dynlibdir code path is > 8.0.1 only.

christiaanb · 2016-10-20T09:03:31Z

It fails on the 8.0.1 because that travis build has TEST_OTHER_VERSION=YES, so it also tests the testsuite using ghc-head. Anyhow, D2611 is not even merged in HEAD yet... so I don't know which exact GHC version it will be. I guess we'll have to wait until D2611 is merged.

`--dynlibdir` indicates the directory in which dynamic libraries are installed. By default this setting is equal to: `$libdir/$abi` The static libraries will still end up in: `$libdir/$libsubdir` With `$libsubdir/$abi` as the default directory for dynamic libraries, dynamic libraries will by default end up in a single shared directory (per package database). This has the potential to reduce start-up times for dynamically linked executable as only one RPATH per package database will be needed. This commit uses the functionality defined in https://phabricator.haskell.org/D2611 to tell GHC's > 8.0.1 package database that dynamic libraries are copied to the directories mentioned in the `dynamic-library-dirs` field.

christiaanb · 2016-10-20T13:54:05Z

Ok, just talked with @bgamari and @dcoutts, D2611 will be merged today, so I've updated this patch to expect the changes in D2611 to live in GHC 8.0.1.20161021 or later.

23Skidoo · 2016-10-20T20:51:50Z

Merged, thanks!

23Skidoo · 2016-10-21T00:40:26Z

OK, why is CI still broken if the version bound is now correct?

ezyang · 2016-10-21T03:23:56Z

It's because it was testing >= 8.0.1.20161021, but 8.1 nightlies are always greater than that, even without the fix. I pushed a fix.

23Skidoo · 2016-10-21T08:57:47Z

Gah, I could've guessed.

christiaanb mentioned this pull request Oct 13, 2016

Add hidir setting - master branch #3968

Closed

christiaanb force-pushed the dynlibdir branch from dd9e6c2 to 2076850 Compare October 13, 2016 15:26

23Skidoo approved these changes Oct 13, 2016

View reviewed changes

christiaanb force-pushed the dynlibdir branch from 2076850 to 389039a Compare October 13, 2016 17:36

ilovezfs mentioned this pull request Oct 14, 2016

GHC panic when building Stack on macOS Sierra commercialhaskell/stack#2577

Closed

christiaanb mentioned this pull request Oct 14, 2016

Store interfaces files separately from library object files #3982

Closed

christiaanb closed this Oct 14, 2016

dcoutts reviewed Oct 16, 2016

View reviewed changes

dcoutts reopened this Oct 18, 2016

dcoutts requested changes Oct 18, 2016

View reviewed changes

23Skidoo mentioned this pull request Oct 18, 2016

Backport "Store interfaces files separately from library object files #3982" to 1.24 #3983

Closed

ilovezfs mentioned this pull request Oct 19, 2016

git-annex 6.20161012 Homebrew/homebrew-core#5895

Closed

christiaanb force-pushed the dynlibdir branch from 389039a to 6be6365 Compare October 19, 2016 09:54

dcoutts requested changes Oct 19, 2016

View reviewed changes

christiaanb force-pushed the dynlibdir branch from 6be6365 to 123018a Compare October 19, 2016 13:35

christiaanb force-pushed the dynlibdir branch from 123018a to 7bddd44 Compare October 19, 2016 15:23

23Skidoo reviewed Oct 20, 2016

View reviewed changes

christiaanb force-pushed the dynlibdir branch from 7bddd44 to d2da655 Compare October 20, 2016 13:51

23Skidoo merged commit 8cac8be into haskell:master Oct 20, 2016

ezyang mentioned this pull request Oct 22, 2016

Cabal-HEAD/ghc-HEAD injects insecure RUNPATHs to a temp build directory in dynamic binaries using library of the same package #4025

Closed

ilovezfs mentioned this pull request Oct 30, 2016

cabal-install: improve Sierra compatibility Homebrew/homebrew-core#6412

Closed

ilovezfs mentioned this pull request Nov 15, 2016

Dependency resolution problem when compiling on OSX with GHC 8 elm/compiler#1519

Closed

domenkozar mentioned this pull request Jul 21, 2017

Linking hack for macOS Sierra NixOS/nixpkgs#27536

Merged

8 tasks

Add --dynlibdir #3979

Add --dynlibdir #3979

Conversation

christiaanb commented Oct 13, 2016

mention-bot commented Oct 13, 2016

bgamari commented Oct 13, 2016

23Skidoo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilovezfs commented Oct 14, 2016

christiaanb commented Oct 14, 2016

ilovezfs commented Oct 14, 2016

christiaanb commented Oct 14, 2016

23Skidoo commented Oct 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcoutts commented Oct 18, 2016

dcoutts left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcoutts commented Oct 18, 2016

dcoutts left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

23Skidoo Oct 19, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

23Skidoo commented Oct 19, 2016

ezyang commented Oct 19, 2016

christiaanb commented Oct 20, 2016

23Skidoo commented Oct 20, 2016 • edited Loading

Choose a reason for hiding this comment

23Skidoo commented Oct 20, 2016

christiaanb commented Oct 20, 2016

christiaanb commented Oct 20, 2016

23Skidoo commented Oct 20, 2016

23Skidoo commented Oct 21, 2016

ezyang commented Oct 21, 2016

23Skidoo commented Oct 21, 2016

Add `--dynlibdir` #3979

Add `--dynlibdir` #3979

23Skidoo Oct 19, 2016 •

edited

Loading

23Skidoo commented Oct 20, 2016 •

edited

Loading