GHC panic when building Stack on macOS Sierra #2577

Open
bitemyapp opened this Issue Sep 11, 2016 · 89 comments

Projects

None yet
@bitemyapp
Contributor
bitemyapp commented Sep 11, 2016 edited

GHC 7.10.3

stack upgrade --git
$ uname -a
Darwin cumae.local 16.0.0 Darwin Kernel Version 16.0.0: Tue Aug 23 17:02:44 PDT 2016; root:xnu-3789.1.31~2/RELEASE_X86_64 x86_64

Xcode 8.0 GM, macOS Sierra

Preprocessing library stack-1.2.1...
[ 1 of 96] Compiling Text.PrettyPrint.Leijen.Extended ( src/Text/PrettyPrint/Leijen/Extended.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Text/PrettyPrint/Leijen/Extended.o )
[ 2 of 96] Compiling Stack.Ghci.Script ( src/Stack/Ghci/Script.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Stack/Ghci/Script.o )
[ 3 of 96] Compiling Stack.FileWatch  ( src/Stack/FileWatch.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Stack/FileWatch.o )
[ 4 of 96] Compiling System.Process.PagerEditor ( src/System/Process/PagerEditor.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/System/Process/PagerEditor.o )
[ 5 of 96] Compiling System.Process.Log ( src/System/Process/Log.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/System/Process/Log.o )
[ 6 of 96] Compiling Paths_stack      ( .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/autogen/Paths_stack.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Paths_stack.o )
[ 7 of 96] Compiling Path.Find        ( src/Path/Find.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Path/Find.o )
[ 8 of 96] Compiling Path.Extra       ( src/Path/Extra.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Path/Extra.o )
[ 9 of 96] Compiling System.Process.Read ( src/System/Process/Read.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/System/Process/Read.o )
ghc: panic! (the 'impossible' happened)
  (GHC version 7.10.3 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/79/6lft40kj3cqb9f08_8yfg3vr0000gn/T/ghc29687_0/libghc_55.dylib, 5): no suitable image found.  Did find:
        /var/folders/79/6lft40kj3cqb9f08_8yfg3vr0000gn/T/ghc29687_0/libghc_55.dylib: malformed mach-o: load commands size (46680) > 32768

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

Completed 178 action(s).

--  While building package stack-1.2.1 using:
      /private/var/folders/79/6lft40kj3cqb9f08_8yfg3vr0000gn/T/stack-upgrade18342/stack/.stack-work/dist/x86_64-osx/Cabal-1.22.5.0/setup/setup --builddir=.stack-work/dist/x86_64-osx/Cabal-1.22.5.0 build lib:stack exe:stack --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
@zmwangx
zmwangx commented Sep 13, 2016 edited

I was able to reproduce both locally and on Homebrew's CI. (We use GHC 8.0.1.)

@zmwangx zmwangx referenced this issue in Homebrew/homebrew-core Sep 13, 2016
Closed

macOS 10.12 and Xcode 8.0 broken formulae #1957

@zmwangx
zmwangx commented Sep 13, 2016

Maybe the title could be a little bit more informative. This is only a problem on macOS 10.12 Sierra as far as I can tell.

@ilovezfs
ilovezfs commented Sep 13, 2016 edited

@cartazio any ideas how to fix the GHC problem with Sierra? macOS 10.12 has gone GM and Stack is dead in the water.

@cartazio

Is it stack specific or ghc 8.0 generally? Is there a minimal repro?

@zmwangx
zmwangx commented Sep 13, 2016

Since @bitemyapp's top post reports this for GHC 7.10.3, I suppose it's not specific to GHC 8.0.

@cartazio

It looks like it's related to split objects or sections?

It would help if there was a simpler application that reported this issue

@ilovezfs

@cartazio there's doubt that it's about split-sections: https://ghc.haskell.org/trac/ghc/ticket/12479#comment:4

@ilovezfs

@cartazio I think the minimal reproducer is brew instal -s stack at this point, but it looks like it really doesn't get very far

Building stack-1.1.2...
Preprocessing library stack-1.1.2...
[ 1 of 87] Compiling Stack.FileWatch  ( src/Stack/FileWatch.hs, dist/dist-sandbox-68bf8d9c/build/Stack/FileWatch.o )
[ 2 of 87] Compiling System.Process.PagerEditor ( src/System/Process/PagerEditor.hs, dist/dist-sandbox-68bf8d9c/build/System/Process/PagerEditor.o )
[ 3 of 87] Compiling System.Process.Log ( src/System/Process/Log.hs, dist/dist-sandbox-68bf8d9c/build/System/Process/Log.o )
[ 4 of 87] Compiling System.Process.Read ( src/System/Process/Read.hs, dist/dist-sandbox-68bf8d9c/build/System/Process/Read.o )
ghc: panic! (the 'impossible' happened)
  (GHC version 8.0.1 for x86_64-apple-darwin):
    Loading temp shared object failed: dlopen(/var/folders/4d/rttdp_9d2s7f36zplgwnqrgr0000gn/T/ghc94633_0/libghc_44.dylib, 5): no suitable image found.  Did find:
    /var/folders/4d/rttdp_9d2s7f36zplgwnqrgr0000gn/T/ghc94633_0/libghc_44.dylib: malformed mach-o: load commands size (40192) > 32768
@ilovezfs

@borsboom any idea what the problem here is? The Sierra GM has escaped into the wild!

@cartazio

minimal code reproducer, stack is too much code

@borsboom
Contributor

@ilovezfs No idea. It has to be a GHC bug (panics always are) but would be nice to find a workaround. I don't have an Apple developer account, so I don't have any access to Sierra betas. Once it's released I can upgrade my spare mac Mini and try to reproduce.

@zmwangx
zmwangx commented Sep 15, 2016

@borsboom The public beta is already on GM so you don't need to have a developer subscription to have access to pretty much the final version of 10.12.

@borsboom
Contributor

If anyone else has some time, I'd probably start trying to make a smaller reproduction by extracting the failing System.Process.Read module to a separate project and minimizing the dependencies. It's an isolated module so that should be pretty easy. If it still fails, just start taking out pieces of code until you find what triggers the panic (if it doesn't fail... well then it's more complicated I guess).

@borsboom borsboom changed the title from GHC bug when upgrading to Stack master to GHC bug when upgrading to Stack master on macOS Sierra Sep 15, 2016
@borsboom
Contributor

@bitemyapp: where did you get your GHC? Was it a sandboxed one installed by Stack?

@borsboom
Contributor

Also, from one of the GHC reports, looks like this isn't specific to Stack. yesod-auth also has a similar panic:

   [ 4 of 11] Compiling Yesod.Auth       ( Yesod/Auth.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.4.0/build/Yesod/Auth.o )
    ghc: panic! (the 'impossible' happened)
      (GHC version 7.10.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ghc64990_0/libghc_21.dylib, 5): no suitable image found.  Did find:
        /var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ghc64990_0/libghc_21.dylib: malformed mach-o: load commands size (34176) > 32768
@bitemyapp
Contributor
bitemyapp commented Sep 15, 2016 edited
[ callen@cumae ~/work/codex travisci-build-matrix-stack ✔ ]
$ stack exec -- ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3
[ callen@cumae ~/work/codex travisci-build-matrix-stack ✔ ]
$ stack exec -- which ghc
/Users/callen/.stack/programs/x86_64-osx/ghc-7.10.3/bin/ghc

I did say it was a GHC bug in the title, just trying to keep y'all apprised.

@cartazio

If you can do a small single Haskell module repro then I and or other ghc
dev team folk can dig into resolving it. But without a clear minimal repro
that 8.0 can trigger with a small module and clear deps, hard to do.

The moment it's an issue that can be demonstrated In a self contained way,
it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom notifications@github.com
wrote:

If anyone else has some time, I'd probably start trying to make a smaller
reproduction by extracting the failing System.Process.Read module to a
separate project and minimizing the dependencies. It's an isolated module
so that should be pretty easy. If it still fails, just start taking out
pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

@cartazio

Also if the bug is in 8.0 and someone can cook up a repro I can see about
sorting out a fix and making sure it's in an 8.0 bug fix. But I can't do
that until there's a small self contained small code base that has the
problem

On Friday, September 16, 2016, Carter Schonwald carter.schonwald@gmail.com
wrote:

If you can do a small single Haskell module repro then I and or other ghc
dev team folk can dig into resolving it. But without a clear minimal repro
that 8.0 can trigger with a small module and clear deps, hard to do.

The moment it's an issue that can be demonstrated In a self contained way,
it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom <
notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

If anyone else has some time, I'd probably start trying to make a smaller
reproduction by extracting the failing System.Process.Read module to a
separate project and minimizing the dependencies. It's an isolated module
so that should be pretty easy. If it still fails, just start taking out
pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

@cartazio

Looks like misty has provided some debugging data upstream to ghc. I'll
talk with bengamari. His current theory is that somehow the offending
builds have split sections enabled. And thus are drowning the linker

On Friday, September 16, 2016, Carter Schonwald carter.schonwald@gmail.com
wrote:

Also if the bug is in 8.0 and someone can cook up a repro I can see about
sorting out a fix and making sure it's in an 8.0 bug fix. But I can't do
that until there's a small self contained small code base that has the
problem

On Friday, September 16, 2016, Carter Schonwald <
carter.schonwald@gmail.com
javascript:_e(%7B%7D,'cvml','carter.schonwald@gmail.com');> wrote:

If you can do a small single Haskell module repro then I and or
other ghc dev team folk can dig into resolving it. But without a clear
minimal repro that 8.0 can trigger with a small module and clear deps, hard
to do.

The moment it's an issue that can be demonstrated In a self contained
way, it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom <
notifications@github.com> wrote:

If anyone else has some time, I'd probably start trying to make a
smaller reproduction by extracting the failing System.Process.Read module
to a separate project and minimizing the dependencies. It's an isolated
module so that should be pretty easy. If it still fails, just start taking
out pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

@ilovezfs

His current theory is that somehow the offending builds have split sections enabled.

which would be quite weird since as misty also pointed out it seems to be a non-default option that we didn't pass in, but that doesn't mean it's impossible it built that way anyway for some reason ...

@cartazio

I'm pretty sure split objects is the default for the libraries. Could you
try disabling that? I could be misremembering. Also I'm totally weak at
linker stuff

On Friday, September 16, 2016, ilovezfs notifications@github.com wrote:

His current theory is that somehow the offending builds have split
sections enabled.

which would be quite weird since as misty also pointed out it seems to be
a non-default option that we didn't pass in, but that doesn't mean it's
impossible it built that way anyway for some reason ...


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwhcph23yUUDuzVrrMcW5uam3W4M0ks5qqhzmgaJpZM4J546w
.

@ilovezfs

--disable-split-objs and --disable-split-sections?

@zmwangx zmwangx referenced this issue in Homebrew/homebrew-core Sep 16, 2016
Closed

Popular formulae with bottling trouble on 10.12 #4841

21 of 27 tasks complete
@cartazio

Could be / should be those

On Friday, September 16, 2016, ilovezfs notifications@github.com wrote:

--disable-split-objs and --disable-split-sections?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwjyBGkFtBJ8zvJI5KsIsa7GtGthcks5qqia7gaJpZM4J546w
.

@cartazio

It's important to note that some builds of ghc base and boot libs may also
have those flags enabled.

In which case this could be a regression in linker on the OS X release. Eg
is it a format limit or a hard coded "safety limit" in the Sierra linker?
Are they still using gnu linker or move to llvm? In latter case is there
some hard coded sanity limits?

Probably easy to test by unpacking the Sierra cli tools and using them on a
previous os release.

On Friday, September 16, 2016, Carter Schonwald carter.schonwald@gmail.com
wrote:

Could be / should be those

On Friday, September 16, 2016, ilovezfs <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

--disable-split-objs and --disable-split-sections?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwjyBGkFtBJ8zvJI5KsIsa7GtGthcks5qqia7gaJpZM4J546w
.

@cartazio

Looks like it's because they moved to llvms tool lld, and it hard coded a
limit on number of sections it'll accept. So the near term mitigation
would be to disable split objects on a ghc Mac build , but the Better
medium / long term is
A) we all fire radars on this
B) we make sure that upstream lld is fixed.

Me and other ghc devs are collaborating on IRC right now to confirm that
we've isolated the problem. Affected users could also unpack a pre Sierra
cli tools and put dl in their path as an alternative mitigation (this is me
speculating on an approach I have not tested but would probably work fine )

On Friday, September 16, 2016, Carter Schonwald carter.schonwald@gmail.com
wrote:

Looks like misty has provided some debugging data upstream to ghc. I'll
talk with bengamari. His current theory is that somehow the offending
builds have split sections enabled. And thus are drowning the linker

On Friday, September 16, 2016, Carter Schonwald <
carter.schonwald@gmail.com
javascript:_e(%7B%7D,'cvml','carter.schonwald@gmail.com');> wrote:

Also if the bug is in 8.0 and someone can cook up a repro I can see about
sorting out a fix and making sure it's in an 8.0 bug fix. But I can't do
that until there's a small self contained small code base that has the
problem

On Friday, September 16, 2016, Carter Schonwald <
carter.schonwald@gmail.com> wrote:

If you can do a small single Haskell module repro then I and or
other ghc dev team folk can dig into resolving it. But without a clear
minimal repro that 8.0 can trigger with a small module and clear deps, hard
to do.

The moment it's an issue that can be demonstrated In a self contained
way, it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom <
notifications@github.com> wrote:

If anyone else has some time, I'd probably start trying to make a
smaller reproduction by extracting the failing System.Process.Read module
to a separate project and minimizing the dependencies. It's an isolated
module so that should be pretty easy. If it still fails, just start taking
out pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

@cartazio

Ld. not dl

On Friday, September 16, 2016, Carter Schonwald carter.schonwald@gmail.com
wrote:

Looks like it's because they moved to llvms tool lld, and it hard coded a
limit on number of sections it'll accept. So the near term mitigation
would be to disable split objects on a ghc Mac build , but the Better
medium / long term is
A) we all fire radars on this
B) we make sure that upstream lld is fixed.

Me and other ghc devs are collaborating on IRC right now to confirm that
we've isolated the problem. Affected users could also unpack a pre Sierra
cli tools and put dl in their path as an alternative mitigation (this is me
speculating on an approach I have not tested but would probably work fine
)

On Friday, September 16, 2016, Carter Schonwald <
carter.schonwald@gmail.com
javascript:_e(%7B%7D,'cvml','carter.schonwald@gmail.com');> wrote:

Looks like misty has provided some debugging data upstream to ghc. I'll
talk with bengamari. His current theory is that somehow the offending
builds have split sections enabled. And thus are drowning the linker

On Friday, September 16, 2016, Carter Schonwald <
carter.schonwald@gmail.com> wrote:

Also if the bug is in 8.0 and someone can cook up a repro I can see
about sorting out a fix and making sure it's in an 8.0 bug fix. But I can't
do that until there's a small self contained small code base that has the
problem

On Friday, September 16, 2016, Carter Schonwald <
carter.schonwald@gmail.com> wrote:

If you can do a small single Haskell module repro then I and or
other ghc dev team folk can dig into resolving it. But without a clear
minimal repro that 8.0 can trigger with a small module and clear deps, hard
to do.

The moment it's an issue that can be demonstrated In a self contained
way, it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom <
notifications@github.com> wrote:

If anyone else has some time, I'd probably start trying to make a
smaller reproduction by extracting the failing System.Process.Read module
to a separate project and minimizing the dependencies. It's an isolated
module so that should be pretty easy. If it still fails, just start taking
out pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

@borsboom borsboom changed the title from GHC bug when upgrading to Stack master on macOS Sierra to GHC panic when building Stack on macOS Sierra Sep 16, 2016
@borsboom
Contributor
borsboom commented Sep 16, 2016 edited

Is this a setting that is "baked into" the GHC binary when the bindist is built, or can it be controlled by an argument to the bindist's configure when installing the bindist (or, alternatively, by adjusting the lib/ghc-8.0.1/settings or other files after installation)?

@cartazio

The build of GHC and boot lib time thing. Default on most tier want
platforms is to enable split objects for the base and boot libs so that end
executable size is smaller

It's also a flag for userland module builds

On Sep 16, 2016 11:23 AM, "Emanuel Borsboom" notifications@github.com
wrote:

Is this a setting that is "baked into" the GHC binary when the bindist is
built, or can it be controlled by an argument to the bindist's configure
when installing the bindist (or, alternatively, by adjusting the
lib/ghc-8.0.1/settings or similar file after installation)?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwhj1w5pujW__B5N57yGG7ceUfbHvks5qqrSJgaJpZM4J546w
.

@borsboom
Contributor

Ah, unfortunate. Still, we can have Stack detect that it's running on macOS >=Sierra and install GHC using alternative bindists.

@cartazio

I'm not sure if that will suffice. We are still investigating

On Friday, September 16, 2016, Emanuel Borsboom notifications@github.com
wrote:

Ah, unfortunate. Still, we can have Stack detect that it's running on
macOS >=Sierra and install GHC using alternative bindists.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwrDpt8h0YR8WHS25Ej-fnCVCvsQdks5qqvGVgaJpZM4J546w
.

@rwbarton

It's actually not a GHC bug and doesn't have to do with split objects at all. Sierra has a new limit on the "load commands size" of a shared library (that you can see mentioned in the error message) which includes the paths to its dependencies, and Stack's many dependencies and long directory paths exceed this limit.

There are possible workarounds, but I can't think of anything simple.

@ilovezfs
ilovezfs commented Sep 18, 2016 edited

@rwbarton @cartazio From one of our subject matter experts:

I guess this was the hidden message behind this year's WWDC session 406 where they talked about dynamic linker internals and optimizing app startup times.

The apparent new limit of 32K enforced by 10.12 feels very conservative, but then again GHC easily blowing that limit is also a bit scary.

I think their best bet is indeed to implement the idea from the brain-storming and make sure they have only one (or very few) ⁠⁠⁠⁠LC_RPATH⁠⁠⁠⁠ load commands that cover the majority of the path prefix(es) and the remaining ⁠⁠⁠⁠LC_LOAD_DYLIB⁠⁠⁠⁠ commands build on top of that to keep things small. Having a separate ⁠⁠⁠⁠LC_RPATH⁠⁠⁠⁠ command for every single library is a bit insane (but I can totally sympathize with the simplicity of such a solution).

I know basically nothing about GHC, but if the directory layout is somewhat predictable and fixed (aside from a changing prefix), i.e. it's easy to infer where a library should be located relative to the executable/library loading it, then another solution could be to rely on ⁠⁠⁠⁠@loader_path/⁠⁠⁠⁠ and/or ⁠⁠⁠⁠@executable_path/⁠⁠⁠⁠ to avoid the lengthy and repeated prefixes.

I think they will really need to optimize the load commands they are placing in every single binary to stay below the limit. The discussed common prefix idea should be fairly simple to implement. If not directly in the compiler, then at least as a post-processing step for the binaries.

The dyld man page in its entirety, but especially the bottom was highly recommended reading.

For post-processing macho programmatically this has been suggested as a useful resource https://github.com/Homebrew/ruby-macho for various techniques.

@UniqMartin
UniqMartin commented Sep 18, 2016 edited

Apparently I'm a “subject matter expert” and I'm the one being quoted above. I'm am contributor to the above mentioned Ruby library for dissecting and modifying Mach-O files and I'm decently knowledgeable about loaders and linkers in general and on macOS in particular.

I'm also completely clueless about GHC or Haskell in general, but if that's not a problem, I'm happy to help with some of my “wisdom”. (Disclaimer: I've also got a lot of other stuff on my plate at the moment, so I can't really promise super timely responses, but I'll try.)

@borsboom
Contributor

So it sounds like this is probably something that should be fixed in properly Cabal and GHC, but may be possible to somewhat work around in Stack by shortening the paths it uses? We already do some tricks on Windows like that because of the path length limits there by putting the STACK_ROOT in C:\sr and using a hash instead of some of the longer subdirectory chains.

@ilovezfs

@borsboom definitely.

@davorb
davorb commented Sep 22, 2016

The same issue is being discussed over at the GHC trac.

In the mean time, what do you guys suggest that those of us who need stack on a Mac do? Is there a way to get homebrew to install prebuilt binaries of stack?

@ilovezfs

I'd install El Capitan in a virtual machine until this is resolved.

@borsboom
Contributor

I'd suggest following the manual download instructions for Mac OS X. Those will get you a binary that reportedly runs on Sierra and works for smaller projects. For projects with a large number of dependencies, stick with El Capitan (e.g. in a VM like @ilovezfs suggested).

@borsboom borsboom added a commit that referenced this issue Sep 22, 2016
@borsboom borsboom Install instruction warnings for macOS Sierra (#2577)
Also, get.haskellstack.org no longer installs using Homebrew until this
is fixed.
e0952f0
@borsboom borsboom added a commit that referenced this issue Sep 22, 2016
@borsboom borsboom Install instruction warnings for macOS Sierra (#2577)
Also, get.haskellstack.org no longer installs using Homebrew until this
is fixed.
b5e8d1b
@simon-nicholls

Thanks @borsboom. I just updated to macOS Sierra, and wanted to confirm that the manual download binary works for me.

My existing brew install of stack failed to work, which I had expected from reading this issue, and replacing the stack binary with the manual download version got me up and running.

To test, I just built a small lts-7.0 Spock application (103 dependencies), having removed my global .stack & project work directories first, and there are seemingly no problems with my existing brew installed ghc-8.0.1.

@bitemyapp
Contributor

@borsboom Manual download binary worked for me as well, thank you for that.

@borsboom
Contributor

I've update get.haskellstack.org and the manual installation instructions to prefer the binary download over Homebrew, and added some Sierra warnings, for now.

@bitemyapp
Contributor

@borsboom tyvm 👍

@ilovezfs

I think we'll probably just boneyard it for now.

@ilovezfs ilovezfs referenced this issue in Homebrew/homebrew-core Sep 23, 2016
Closed

haskell-stack: migrate to boneyard #5152

@krambs
krambs commented Sep 23, 2016

But I NEEEED Siri! LOL FML

@zmwangx zmwangx referenced this issue in Homebrew/homebrew-core Sep 24, 2016
Closed

haskell-stack: set maximum macOS to El Capitan #5184

4 of 4 tasks complete
@borsboom
Contributor

I did some experimentation with shortening the paths like we do on Windows, but no luck: still getting the GHC panic. I can't think of any other workarounds, so I guess we'll just have to wait until it's fixed upstream in GHC/Cabal.

@ilovezfs

OK, then I shall reveal my workaround :)

We can simply strip the useless rpaths from the El Capitan bottle using install_name_tool and ship a Sierra bottle for this.

@EugeneN
EugeneN commented Sep 25, 2016

I'm not using brew-installed stack, but binary manually-installed one, and I'm still having the problem (on a rather big project).

@ilovezfs

Correct, a working stack binary will only work on projects that don't themselves hit this same bug that occurs when building stack.

@ilovezfs

@borsboom I've now shipped bottles for Sierra: Homebrew/homebrew-core@c1dce90

@simon-nicholls

Thanks. The Sierra bottle works in my (study) environment.

@bitemyapp
Contributor

I'm not using brew-installed stack, but binary manually-installed one, and I'm still having the problem (on a rather big project).

I was fine compiling a Servant project, but the yesod-simple scaffold still trips this for me.

@ilovezfs

Yes, yesod* is known to be affected. GHC/Cabal needs to fix this. Listing affected projects here won't improve the situation.

@bitemyapp
Contributor

Sure, I'm reading along with https://ghc.haskell.org/trac/ghc/ticket/12479 to see where they've gotten so far.

@ilovezfs

It looks like a fix is now on the way haskell/cabal#3979.

@shanemikel

Now that we have haskell/cabal#3982 and haskell/cabal#3983, has there been any thought about "teaching [stack] the new flags?"

@23Skidoo

We probably won't be adding new flags to Cabal after all and go with an updated version of haskell/cabal#3979. Instead GHC 8.0.2 will gain some new flags.

@nader77
nader77 commented Oct 23, 2016

What's the status on this issue ? If it's resolved, what we should do to fix it on MacOS Sierra ?

@Tehnix
Tehnix commented Oct 25, 2016

It wasn't entirely clear to me (or maybe I just hope I'm wrong) from haskell/cabal#3979, and the other tickets etc, but does that mean that only GHC 8.0.2 and forward will build on macOS Sierra, and older GHC versions will still have the same problem?

@borsboom
Contributor

@Tehnix that is usually how it's gone in the past, unfortunately (e.g GHC 7.8.4 is broken on Yosemite and up). These kinds of fixes haven't been back-ported. However, if a volunteer wants to take on backporting, we could have Stack use a backported fixed version when it needs to.

@cartazio

It's fixed in 8.0.2 and associated cabal

On Sunday, October 23, 2016, Nader Safadi notifications@github.com wrote:

What's the status on this issue ? If it's resolved, what we should do to
fix it on MacOS Sierra ?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2577 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwhd2VLlnY-YbVpUoKrGVPMnOkxJ1ks5q23vDgaJpZM4J546w
.

@steakknife
steakknife commented Oct 25, 2016 edited

For maintainers, something to consider Late to the party: On macOS, Homebrew casks are typically used for complex/isolated/versioned (i.e., ChefDK, TeX Live and maybe stack) binary (i.e., .dmg, .pkgs, etc.) distributions. Whereas brew formulas are typically preferred for hackable/component software built from source (although bottles are pre-built binaries of the formula). Some people may want brew formulas, but a complete, vendored-dependencies, binary distribution is usually easier, cheaper and less error-prone to both deploy and support. (Casks are awesome because it's like a centralized CLI Sparkle for full lifecycle mgmt, i.e., update everything: brew cask update.)

@shanemikel

@steakknife are you sure you're in the right thread?

@steakknife

@shanemikel I encountered this issue some time ago. Most recently addressing commercial haskell people and others whom may need to think about process improvement in release engineering to reduce community/ development support burden. Don't have time to find the appropriate feedback channel. Good luck, hope it works out.

@cartazio

Older ghc will work fine as long as there's no use of th in packages that have as many transitive dynamic library deps as the union of snap and yesod. Also can mitigate by making sure the paths are shorter to the dynamic libs.

Alternatively a build of ghc that uses the ghci static linker for ghci on OS X would also dodge this issue.

@Tehnix
Tehnix commented Oct 26, 2016

@borsboom makes sense if that is what's usually done. I think I'll rather hope for the packages/libraries I use to be updated and perhaps help with that :)

@cartazio Unfortunately my use case is actually specifically for Yesod, so that leaves me on either 8.0.2+ for macOS Sierra, or using docker (which is quite easy with stack luckily, but sometimes hangs on the linking phase).

@katrielalex katrielalex referenced this issue in tamarin-prover/tamarin-prover Nov 10, 2016
Closed

macos Sierra messes with ghc in unfortunate ways :( #187

@chadbrewbaker
Contributor

The yesod-sqlite template is still busted on Sierra. If GHC < 8.0.2 is busted on OSX Sierra, shouldn't there be a LTS that supports this? LTS Haskell 7.10 is ghc-8.0.1.

@shanemikel

@chadbrewbaker as of the time of this comment, the progress is at 99%. https://ghc.haskell.org/trac/ghc/roadmap

There's an open issue on this repo for backporting the relevant fixes (which I believe are available now), but... Well you can see for yourself, nobody's done it yet.

@ilovezfs

Sources for Sierra's dyld have now been posted:

https://opensource.apple.com/source/dyld/dyld-421.1/

https://opensource.apple.com/tarballs/dyld/dyld-421.1.tar.gz

https://opensource.apple.com/source/dyld/dyld-421.1/src/ImageLoader.h.auto.html

#define MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE (32*1024)

https://opensource.apple.com/source/dyld/dyld-421.1/src/ImageLoaderMachO.cpp.auto.html

if ( sizeofcmds > (MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE-sizeof(macho_header)) )
		dyld::throwf("malformed mach-o: load commands size (%u) > %u", sizeofcmds, MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE);

Note the changes from El Capitan https://opensource.apple.com/source/dyld/dyld-360.22/src/ImageLoaderMachO.cpp.auto.html

Other Sierra sources: https://opensource.apple.com/release/os-x-1012.html

@alexanderkjeldaas
Contributor

GHC-8.0.2 RC1 is out http://downloads.haskell.org/~ghc/8.0.2-rc1/

@ilovezfs

@alexanderkjeldaas and that is what the Homebrew Sierra bottle is currently, btw.

@ilovezfs

@borsboom given that GHC and Cabal have both been fixed, should the "resolution: upstream issue" tag be removed from this issue, since the only remaining problems are actually internal to stack itself?

@Tehnix
Tehnix commented Nov 26, 2016

@ilovezfs are you doing anything special to get brew to download 8.0.2? I'm on macOS Sierra and running brew update && brew install ghc will fetch me 8.0.1_3 even though I can see the formula should have 8.0.2 for macOS Sierra :/ ...

@ilovezfs

The name is lying. 8.0.1_3 on Sierra is 8.0.2-rc1. ghc --version will show 8.0.1.20161117

@Tehnix
Tehnix commented Nov 26, 2016

@ilovezfs ah yeah, just figured when I was trying to manually download it. Thanks for the super quick response though! :)

@alexanderkjeldaas
Contributor

how does brew solve this issue? does stack use the homebrew distributions on os x?

@ilovezfs

brew builds the Sierra bottle from the 8.0.2-rc1 source tarball. Before that was available, we used the head of the ghc-8.0 branch. That, combined with the most recent cabal, is sufficient to resolve the problem for GHC + cabal build. The bug is still unfixed for cabal new-build, though.

Note that a more conservative approach would be to backport the fix to 8.0.1:
Homebrew/homebrew-core#6416
(https://gist.githubusercontent.com/ilovezfs/c01cedf6dcf03d35d528e3aab244f42e/raw/1be2b95f935429553a132f0dbb7c2c25aea6c871/gistfile1.txt)

To my knowledge, stack hasn't backported the GHC fixes to their GHC binaries or made the changes to stack itself necessary to take advantage of the GHC and Cabal fixes.

stack will automatically get the GHC side of the equation once there's a stack 8.0.2 but additional changes will be needed to stack itself to actually fix the bug when using stack.

@fizruk
Contributor
fizruk commented Nov 26, 2016 edited

@alexanderkjeldaas Note that Stack can use system GHC if that's available on PATH.
See system-ghc option for stack.yaml.

That's what I use to fix my builds on Sierra now.

@alexanderkjeldaas
Contributor

@fizruk all well and good, but stack --system-ghc upgrade --git doesn't seem to work for example. so when stack is fixed, how do we upgrade?

@fizruk
Contributor
fizruk commented Nov 26, 2016

@alexanderkjeldaas what does stack upgrade --git do? If it simply clones stack repo, builds and then installs then I suspect you can just do that manually?

Note that if you don't need to upgrade to the Git version, you can install Stack with brew install haskell-stack. At least until the dust settles with GHC 8.0.2.

@ilovezfs

By the way, brew can now build stack from source on Sierra, though stack itself cannot build stack yet. If nothing else, it's a neat parlor trick:

brew install --without-bootstrap haskell-stack
@alexanderkjeldaas
Contributor

Another issue: The new homebrew version will not compile the ghcjs releases from http://tolysz.org/ghcjs/untested/ .

Upgrading stack to a ghc version that doesn't build ghcjs is even worse for me than not being able to build large programs with the compiler shipping with stack.

@ilovezfs

will not compile

meaning?

Are you sure you don't need to bump the ghc version constraint or something?

@alexanderkjeldaas
Contributor
@alexanderkjeldaas
Contributor
@noraesae
noraesae commented Dec 2, 2016

For someone looking for TL;DR:

  1. This is because macOS Sierra has changed something internal.
  2. This will be fixed in GHC 8.0.2, which is in RC. Which means it will be fixed in Stack when a next resolver using GHC 8.0.2 is released. Current Stack resolvers use GHC 8.0.1.
  3. brew install ghc will install GHC 8.0.2 RC. (it shows 8.0.1 when ghc --version though)
  4. We can force Stack to use the ghc installed by brew with stack --system-ghc.
    For example:
    $ brew install ghc
    $ stack --system-ghc build
    

I have just tried and saw the problem solved. Please correct me if anything is wrong.

@fizruk
Contributor
fizruk commented Dec 2, 2016

It appears that you can also use GHC 8.0.2-rc1 directly with Stack (no brew needed) with a stack.yaml like this one. Basically you just tell Stack where to get GHC 8.0.2-rc1 for every OS you need. I'll copy the contents here (note that ghc-8.0.1.20161117 is what GHC 8.0.2-rc1 is called):

compiler-check: match-exact
resolver: lts-7.10
compiler: ghc-8.0.1.20161117
setup-info:
  ghc:
    linux64:
      8.0.1.20161117:
        url: http://downloads.haskell.org/~ghc/8.0.2-rc1/ghc-8.0.1.20161117-x86_64-deb8-linux.tar.xz
        content-length: 112047972
        sha1: 6a6e4c9c53c71cc84b6966a9f61948542fd2f15a
    macosx:
      8.0.1.20161117:
        url: https://downloads.haskell.org/~ghc/8.0.2-rc1/ghc-8.0.1.20161117-x86_64-apple-darwin.tar.xz
        content-length: 113379688
        sha1: 53ed03d986a49ea680c291540ce44ce469514d7c
    windows64:
      8.0.1.20161117:
        url: https://downloads.haskell.org/~ghc/8.0.2-rc1/ghc-8.0.1.20161117-x86_64-unknown-mingw32.tar.xz
        content-length: 155652048
        sha1: 74118dd8fd8b5e4c69b25df1644273fbe13177c7
@tolysz
Collaborator
tolysz commented Dec 2, 2016

@alexanderkjeldaas
It is an interesting find. But once the lts moves to ghc 8.0.2 all packages will be compiling together.
Thus this issue will go away. For now, you would need to find versions which work and you could create a new ghcjs bundle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment