Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add `stack build --prefix` option #848

Open
borsboom opened this issue Aug 25, 2015 · 37 comments
Open

Add `stack build --prefix` option #848

borsboom opened this issue Aug 25, 2015 · 37 comments
Milestone

Comments

@borsboom
Copy link
Contributor

@borsboom borsboom commented Aug 25, 2015

This new stack build option will add a post-build step that rebuilds the targets with a cabal configure --prefix= option in a separate dist directory. This will allow installing packages that use data-files in a fixed location on the file system outside of any stack-controlled sandbox.

@borsboom
Copy link
Contributor Author

@borsboom borsboom commented Aug 25, 2015

In order to support installation into directories where the user doesn't have write permission (e.g., /usr/local), should support a flag to specify a command to prefix the Setup.hs copy step with (e.g., sudo). Before starting, check whether writing to the destination works, and if it does not, display a friendly error with a suggestion to use the prefixing option.

@borsboom
Copy link
Contributor Author

@borsboom borsboom commented Nov 25, 2015

Make sure this covers the case outlined in #1262 (comment)

@mgsloan
Copy link
Contributor

@mgsloan mgsloan commented May 2, 2016

Seems like this issue is cropping up for multiple people. Bumping to P2

@borsboom
Copy link
Contributor Author

@borsboom borsboom commented May 29, 2016

Important use case for supporting a --prefix option: with it, it would be feasible for Homebrew to build their packages using Stack (for those packages that have a stack.yaml), which would aid reproducibility. See Homebrew/homebrew-core#1480 and Homebrew/legacy-homebrew#49158 for related discussion.

@borsboom
Copy link
Contributor Author

@borsboom borsboom commented Jun 1, 2016

Here's a more concrete example for Homebrew: Homebrew/homebrew-core#1630

@Blaisorblade
Copy link
Collaborator

@Blaisorblade Blaisorblade commented Jul 13, 2016

Further rationale—sorry if already mentioned in some links: clearing snapshots (to recover space) leaves compiled binaries around, but not their data-files.

@borsboom
Copy link
Contributor Author

@borsboom borsboom commented Jul 17, 2016

Many packages have data-files that aren't actually required for the package to work (like READMEs and Changelogs), and since this feature would require rebuilding those packages and all packages that depend on them (which could be a lot of packages), it should have a way to skip some or all dependencies.

@mgsloan
Copy link
Contributor

@mgsloan mgsloan commented Feb 27, 2017

More demand for this feature, bumping priority.

@essandess
Copy link

@essandess essandess commented Jul 7, 2019

@borsboom

executables will have the staged location of the data files hardcoded

This is the issue. This is what GNU DESTDIR is for. Package managers need to copy from DESTDIR builds into PREFIX with all dependencies hardcoded for PREFIX, not the temporary build location.

You’re correct that the entire Haskell stack—ghc, cabal, etc. —is missing this basic GNU DESTROOT capability for package managers, which is a shame because there are some great Haskell packages.

Another example for the backflips required to destroot ghc: macports/macports-ports#4699

essandess added a commit to essandess/macports-ports that referenced this issue Sep 1, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
essandess added a commit to essandess/macports-ports that referenced this issue Sep 2, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
essandess added a commit to essandess/macports-ports that referenced this issue Sep 2, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
essandess added a commit to essandess/macports-ports that referenced this issue Sep 2, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
essandess added a commit to essandess/macports-ports that referenced this issue Sep 2, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
essandess added a commit to essandess/macports-ports that referenced this issue Sep 3, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
essandess added a commit to essandess/macports-ports that referenced this issue Sep 3, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
essandess added a commit to essandess/macports-ports that referenced this issue Sep 3, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
@essandess
Copy link

@essandess essandess commented Sep 3, 2019

@borsboom @snoyberg @mgsloan @Blaisorblade
At MacPorts we’ve automated stack builds and this has been successful for packages that don’t use datadir.

However, stack’s and cabal’s lack of a GNU-standard capability for DESTDIR and PREFIX to produce relocatable binaries and installs is causing major problems for package managers that need to build with Haskell development tools. E.g. see:

There’s obviously a lot of interest in the capability and the lack of it is holding back deployment of Haskell-based tools.

Are there any plans to tackle this soon?

Related:

@essandess
Copy link

@essandess essandess commented Sep 4, 2019

@borsboom

@essandess Oh one other thing: since you're talking about an intermediate staging area, note that executables will have the staged location of the data files hardcoded, which may not be what you want (it looks like you have some symlink workarounds the ihaskell example to compensate for this). Unfortunately this is a limitation to how data files work, but is out of scope for Stack to fix. In fact, the Stack team recommends against using Cabal data files at all, since it makes location portability of executables so difficult (instead, read-only data can be embedded directly into the executable using file-embed).

What about packages that explicitly specify cabal’s data-files option? Take hlint as an example. Heres’s hlint.cabal:
https://github.com/ndmitchell/hlint/blob/ef26ffcbd0425b98bcc5b330b310df9264a31add/hlint.cabal#L17-L26

When stack is used to build this package, what step causes the breakage of the temporary build directory being hardcoded into the binary?

How can this be fixed when stack is used to build such a package?

@essandess
Copy link

@essandess essandess commented Sep 4, 2019

@jgm
I see that pandoc.cabal has a ton of data-files, https://github.com/jgm/pandoc/blob/0e31483d4358a6d2b4ba96c71237e3f7b32979a1/pandoc.cabal#L42-L179, yet the pandoc binary that stack builds is free of this issue.

Would you please provide any tips or recipes for converting a package to use embedded files and avoid this issue? Is it relatively easy? Would it be reasonable to ask any packages like hlint that have this problem to refactor their code? What’s the recipe to do that? I see https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Data.hs, but I don’t understand how you’ve turned off cabal’s data-files In pandoc.

Hearing your experience about how to fix this issue with code would be much appreciated.

@essandess
Copy link

@essandess essandess commented Sep 5, 2019

This older post from @ndmitchell appears to provide at least a couple straightforward solutions to cabal‘s hardcoded path problem:

  1. Create a Paths_mypackagename.hs file that monkeypatches the default hardcoded cabal path. If this path is relative to the binary, it would be easy to use ../share/mypackage_datadir or such like.
  2. One comment mentions that cabal will use the environment variable mypackage_datadir to set the datadir paths encoded in the executable.

Does anyone have any experience or pointers with this cabal behavior? I’d like to start barking up the right tree.

cc: @borsboom @snoyberg @mgsloan @Blaisorblade @jgm @acfoltzer @bubba @phadej @typedrat @23Skidoo @bos @simonmar @christiaanb

@christiaanb
Copy link

@christiaanb christiaanb commented Sep 5, 2019

I did once implement partial support of "relocatable" packages in Cabal: haskell/cabal#2255; it was sufficient to get relocatable Cabal sandboxes: http://qbaylogic.com/blog/2016/05/08/relocatable-sandboxes.html

@essandess
Copy link

@essandess essandess commented Sep 5, 2019

@christiaanb

I did once implement partial support of "relocatable" packages in Cabal: haskell/cabal#2255; it was sufficient to get relocatable Cabal sandboxes: http://qbaylogic.com/blog/2016/05/08/relocatable-sandboxes.html

Thank you. Does the --enable-relocatable address the problem with hardcoded paths to cabal data-files? If it does, it’s not clear to me.

@christiaanb’s post brings us to three possible solutions:

  1. Create a Paths_mypackagename.hs file that monkeypatches the default hardcoded cabal path. If this path is relative to the binary, it would be easy to use ../share/mypackage_datadir or such like.
  2. One comment mentions that cabal will use the environment variable mypackage_datadir to set the datadir paths encoded in the executable.
  3. cabal --enable-relocatable

@borsboom @snoyberg Would any of these solutions work within stack?

If any of these do work, I could personally modify specific ports of stack-based packages, or even MacPorts automated process for stack builds, but this issue goes above specific packages built on macOS.

If cabal provides this capability, then stack should support it to provide relocatable binaries.

@jgm
Copy link

@jgm jgm commented Sep 5, 2019

@essandess - pandoc has an embed_data_files cabal flag, which is enabled by default in the stack build. That avoids the issue. This flag causes all the data files to be embedded as bytestring blobs in the binary, making the executable portable. The file-embed package is used for this. See Text.Pandoc.Data.

Of course, this is not the ideal solution when pandoc is installed by a package manager. In that case it's usually good practice to have the data files live separately in the file hierarchy, where they can be inspected and replaced. (This is how debian linux installs of pandoc work, for example.) Unfortunately, stack doesn't currently support this because of the lack of --prefix.

essandess added a commit to essandess/macports-ports that referenced this issue Sep 6, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Fix stack/cabal data-files hardcoded path issue with Paths_${name}.hs file
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
@essandess
Copy link

@essandess essandess commented Sep 6, 2019

The approach described by hlint's author @ndmitchell solves the issue of hardcoded cabal data-files in the binary.

All that's required is to specify the path within a file called Paths_packagename.hs, which is normally automatically generated by cabal with its own paths. Here is an automated stack build that solves the issue along with related files:

Also, setting the environment variable packagename_datadir at runtime overrides the binary's hardcoded packagename_datadir path.

In contrast, setting datadir in stack.yaml causes stack to actually try to write to that directory during compilation, which breaks GNU DESDIR capability for package managers. I believe that this inconsistent and breaking behavior is a bug; see #5026.

@borsboom @snoyberg @mgsloan @Blaisorblade @jgm @acfoltzer @bubba @phadej @typedrat @23Skidoo @bos @simonmar @christiaanb
Thank you all for all your longstanding help, comments, and pointers about this issue. I believe that the approach identified by @ndmitchell is sufficient for at least BSD package managers to include stack builds of packages that use cabal's data-files.

essandess added a commit to essandess/macports-ports that referenced this issue Sep 10, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Fix stack/cabal data-files hardcoded path issue with Paths_${name}.hs file
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
essandess added a commit to essandess/macports-ports that referenced this issue Sep 11, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Fix stack/cabal data-files hardcoded path issue with Paths_${name}.hs file
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
neverpanic added a commit to macports/macports-ports that referenced this issue Sep 14, 2019
* Fix destroot issue in binary caused by datadir in stack_root
* Fix stack/cabal data-files hardcoded path issue with Paths_${name}.hs file
* Allow initialization of proxy configuration files
* See commercialhaskell/stack#848
* See commercialhaskell/stack#4857
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.