Store interfaces files separately from library object files #3982
Conversation
These two flags indicate where the two distinct aspects of a
Haskell library end up.
--commonlibdir: The subdirectory of --libdir where the object
library files (.so/.a/.dll/.dylib) get installed.
--hidir: The directory where the interface files (.hi)
get installed.
The reason we want to do this is because we want dynamic libraries
to all end up in a shared directory to reduce start-up times of
the run-time linker. However, we still want the .hi to end up
in one directory per package.
We cannot repurpose --libsubdir to take over the role of what
--commonlibdir is doing because then we would run into trouble
with Setup.hs files build against an older Cabal. If we were
to repurpose --libsubdir, then all the object and interface
files would end up in a single shared directory for all
packages when using an older Cabal.
|
@christiaanb, thanks for your PR! By analyzing the history of the files in this pull request, we identified @dcoutts, @hvr and @sopvop to be potential reviewers. |
|
/cc @dcoutts |
|
@christiaanb seems to work and |
|
This appears to fail with new-build https://gist.github.com/ilovezfs/ff5539ab4b6b16ab9a832ff7c9476284 |
|
@ilovezfs what you mean by "it fails on new-build" is "this patch doesn't solve the problems reported in GHC issue 12497 when using new-build"? Or that things are actually broken by this patch that used to work before? |
|
I mean that this PR is an incomplete solution to GHC 12479 https://ghc.haskell.org/trac/ghc/ticket/12479 not that it causes a regression. Is there no hope of a common solution that will address non-new-build and new-build alike? If there is a prospect for a common solution, then it should be understood that the change in this PR may need to be backed out in favor of that common solution at some point in the near future. If there is no such prospect, then no harm, no foul in treating them as separate problems even though GHC issue 12479 is the underlying cause for both. |
If it's true, I'd prefer to get it right the first time, since backing this change out will result in more backwards compatibility cruft. |
|
IMHO I'd like to hear @dcoutts or someone who knows on how new-build store works, to comment on this. EDIT it seems that those flags will affect where stuff goes also in |
|
@phadej: yes these flags allow you to place library objectfiles (.so/.a/.dll/.dylib) in another directory than the interface files (.hi). That's all this PR basically does:
So yes, to fix new-build and stack with regards to https://ghc.haskell.org/trac/ghc/ticket/12479 should just be to teach them these new flags, and make sure that library object files end up in a shared directory. |
|
Yeah, the fix for new-build should not be too difficult. Just passing the new flag. |
On Saturday, October 15, 2016, Edward Z. Yang notifications@github.com
|
Not exactly, but yes. See here and here. HFS+ and FAT32 have the lowest limits on the number of files per directory (32K and 65K, respectively), for other commonly used filesystems it is practically unlimited. |
There was a problem hiding this comment.
This is a lovely patchset for a complicated problem. I think the only major question is whether or not we want to bikeshed the name commonlibdir any further (sorry, probably a bit my fault). The reason why commonlibdir may not be a good name is because it doesn't describe what gets installed to the directory; it describes a "mode of use" of the directory (which may or may not be true.) Perhaps binlibdir? Another possibility is to add ANOTHER parameter so we have both dynlibdir and staticlibdir (and then staticlibdir could be set equal to hidir, since rpath considerations don't apply to static libraries.)
|
|
||
| , option "" ["libsubdir"] | ||
| ("subdirectory of libdir in which libs are installed." ++ | ||
| "Only has an effect on Setup files build against Cabal < 1.25" |
| "subdirectory of libdir in which libs are installed" | ||
| libsubdir (\v flags -> flags { libsubdir = v }) | ||
| , option "" ["commonlibdir"] | ||
| "subdirectory of libdir in which the object files of libs are installed" |
There was a problem hiding this comment.
The object files aren't installed here, right? Only the .so/.a/.dll/.dylib libraries.
| @@ -690,6 +698,22 @@ path options: | |||
| ``$pkg``, ``$version``, ``$compiler``, ``$os``, ``$arch``, ``$abi``, | |||
| ``$abitag`` | |||
|
|
|||
| .. option:: --libsubdir=dir | |||
|
|
|||
| For use with Setup.hs files build against a version of Cabal prior to 1.25. | |||
|
|
||
| For use with Setup.hs files build against a version of Cabal prior to 1.25. | ||
| For later versions of Cabal, this flag is basically deprecated, and you | ||
| should use ``--commonlibdir=dir``. |
There was a problem hiding this comment.
With later versions of Cabal, you should prefer :option:--commonlibdir and :option:--hidir, which let you separately specify where binary libraries and interface files get installed, so that binary libraries can be installed to a shared directory.
| ``/usr/local/lib/mypkg-0.2/ghc-6.4``. | ||
| default *libdir* is ``/usr/local/lib``, and *commonlibdir* contains the | ||
| ABI, e.g. ``x86_64-linux-8.0.1``, so libraries would be installed in | ||
| ``/usr/local/lib/x86_64-linux-8.0.1``. |
There was a problem hiding this comment.
It is recommended that a single, common directory to be used to store all installed libraries (as opposed to using $pkgid or similar variables to create a directory per installed library), as this helps reduce the size of rpath in executables built against dynamic libraries. See #3982 for more details.
I don't understand this bit. Surely the situation is that in older Cabal lib versions the Isn't building Setup.hs against older Cabal lib versions a red herring? Since that doesn't let us control them independently anyway. We know what version of the Cabal Setup.hs CLI we're talking to, so we can set the flags accordingly. What am I missing? |
|
So as I see it, there's a few parts to this change:
One or both of:
The advantage of changing the default in Cabal is that it affects Linux distros that install libraries globally, for those Linux distros that don't override things already and just go with the defaults. The disadvantage of changing the default in Cabal is that it adds extra version-dependent behaviour changes, that downstream tools need to be aware of and allow for or override (e.g. cabal-install already overrides all the flags in new-build at least, though it obviously does not yet override new flags like So an advantage of only changing the defaults in cabal-install/stack is that it may be simpler to do there, since they have to do it anyway. |
|
This is to help me understand... So the original layout scheme was: The .hi files and the .a .so .dll .dynlib files all get installed under The Note that the libname is the package id plus a hash that is unique locally to a linked app/lib. The hash includes the names and versions of all deps, but no more. It is not a nix hash including sources and configuration. The layout proposed in this patch (and please correct me if I'm wrong) is: The .hi files get installed under So this introduces both So a few things I'm concerned about:
|
|
So let's assume we only add This all works fine for And so what happens is that the And that is the reason I had to add another flag, |
|
The proposed layout is: |
|
@dcoutts Currently in this PR, But I'm not really sure of your comments: is this something you find desirable? or do you want |
|
Could we not use symlinks to collect all the libs Into a flattened On Sunday, October 16, 2016, Christiaan Baaij notifications@github.com
|
|
@cartazio If we did, where would be "store" this location so that GHC and Cabal know where to find this symlinked directory? Would we update the package database with an extra field? We need to know the location of the symlinked libraries so that GHC and Cabal can tell the linker which RPATHs to add. |
`--binlibsubdir` more properly describes what this flag does: it indicateds the directory in which the binary libraries go, and it is a subdir of `--libdir`.
I was wrong actually. The |
|
Reopened #3979 since we're pursuing that approach. So I'll take the liberty of closing this PR. |
These two flags indicate where the two distinct aspects of a
Haskell library end up.
--binlibsubdir: The subdirectory of--libdirwhere the binary library files (.so/.a/.dll/.dylib) get installed.--hidir: The directory where the interface files (.hi) get installed.The reason we want to do this is because we want dynamic libraries
to all end up in a shared directory to reduce start-up times of
the run-time linker. However, we still want the .hi to end up
in one directory per package.
We cannot repurpose --libsubdir to take over the role of what
--commonlibdir is doing because then we would run into trouble
with Setup.hs files build against an older Cabal. If we were
to repurpose --libsubdir, then all the object and interface
files would end up in a single shared directory for all
packages when using an older Cabal.
After several attempts (#3955,#3968,#3979), this should be the definitive solution for: https://ghc.haskell.org/trac/ghc/ticket/12479
EDIT:
--binlibsubdirwas originally called--commonsubdirwhen this PR was first created.