Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flycheck-haskell breaks with library-vanilla:false and shared:true, executable-dynamic:true under cabal sandbox #23

Closed
da-x opened this issue Dec 11, 2014 · 8 comments

Comments

@da-x
Copy link

da-x commented Dec 11, 2014

Consider the following under ~/.cabal/config:

library-vanilla: False
shared: True
executable-dynamic: True

With cabal sandbox for building some package (ghc 7.8.3) and the settings above, imports of the package fail during during flycheck's run.

I switched off library-vanilla back to the default of True, rebuilt the cabal sandbox and it fixed it. So I am guessing that the ghc invocation did not receive the right parameters for loading the shared library builds.

(EDITS: fixes)

@swsnr
Copy link
Contributor

swsnr commented Dec 11, 2014

@da-x I'm sorry, but your description is a little imprecise. Which imports fail? Imports of dependencies, or of modules from your package? Also, what does “fail” mean?

Also, I'm not sure about the last paragraph. I might be mistaken, but as far as I know the default of library-profiling is False, not True, so I'm not sure what you actually changed.

Please try to be more precise. At best, give me a recipe to reproduce the issue from scratch, ideally with a vanilla .cabal/config. Also, please show me the exact invocation of the GHC syntax checker, by typing C-c ! C-c in an affected buffer, selecting haskell-ghc, and copy and paste the output in the compilation buffer that pops up.

@da-x
Copy link
Author

da-x commented Dec 11, 2014

Oops, sorry - I meant to write library-vanilla, and instead copy-pasted the wrong variable. Edited. I'm happy to provide you with more details.

The imports that fail (ghc's "Could not find module") are dependencies of the local package, i.e. stuff that comes from the sandbox.

I managed to reproduce it with yaml package. But I guess it would break on any package that has dependencies that come from the cabal sandbox.

  1. Modify ~/.cabal/config like above
cabal unpack yaml-0.8.10
cd yaml-0.8.10
cabal sandbox init
cabal install

So far so good. But then try to open Text/Libyaml.hs, followed your suggestion with C-c ! C-c. It would first break on CPP (an unrelated issue) but if you edit the #ifdefs away it would break on import, whereas it would have not if the same procedure has been executed with library-vanilla:true.

ghc -Wall -fno-code -no-user-package-db -package-db /home/dan/test/yaml-0.8.10/.cabal-sandbox/x86_64-linux-ghc-7.8.3-packages.conf.d -i/home/dan/test/yaml-0.8.10/dist/build -i/home/dan/test/yaml-0.8.10/dist/build/autogen -i/home/dan/test/yaml-0.8.10/dist/build/json2yaml/json2yaml-tmp -i/home/dan/test/yaml-0.8.10/dist/build/yaml2json/yaml2json-tmp -i/home/dan/test/yaml-0.8.10/ -i/home/dan/test/yaml-0.8.10/ -x hs /home/dan/test/yaml-0.8.10/Text/Libyaml.hs

/home/dan/test/yaml-0.8.10/Text/Libyaml.hs:50:8:
    Could not find module ‘Control.Monad.Trans.Resource’
    There are files missing in the ‘resourcet-1.1.3’ package,
    try running 'ghc-pkg check'.
    Use -v to see a list of the files searched for.

/home/dan/test/yaml-0.8.10/Text/Libyaml.hs:51:8:
    Could not find module ‘Data.Conduit’
    There are files missing in the ‘conduit-1.2.3’ package,
    try running 'ghc-pkg check'.
    Use -v to see a list of the files searched for.

Pasted 'cabal sandbox hc-pkg list' below just to see the dependencies are really there as far as the sandbox is concerned:

...
...
/home/dan/test/yaml-0.8.10/.cabal-sandbox/x86_64-linux-ghc-7.8.3-packages.conf.d
   aeson-0.6.2.1
   blaze-builder-0.3.3.4
   conduit-1.2.3
   dlist-0.7.1
   exceptions-0.6.1
   lifted-base-0.2.3.0
   mmorph-1.0.4
   monad-control-0.3.3.0
   nats-0.2
   resourcet-1.1.3
   scientific-0.3.3.2
   semigroups-0.15.4
   transformers-base-0.4.3
   void-0.6.1
   yaml-0.8.10

@da-x da-x changed the title flycheck-haskell's breaks with library-vanilla:false and shared:true, executable-dynamic:true under cabal sandbox flycheck-haskell breaks with library-vanilla:false and shared:true, executable-dynamic:true under cabal sandbox Dec 11, 2014
@swsnr
Copy link
Contributor

swsnr commented Dec 11, 2014

@da-x Well, there are no dependencies anymore. By disabling vanilla libraries, you've just opted out from building the dependencies. You only have the profiling libraries left, but GHC doesn't look for these, because Flycheck doesn't pass -prof to GHC.

I'd argue that Flycheck is right here: It's a configuration issue on your side.

Is there a reason why you disable vanilla libraries? As far as I know, vanilla and profiling libraries can co-exist side by side, so there's no need to disable the former in order to get the latter. You just need to enable profiling for your own package with cabal configure -p --enable-executable-profiling.

@swsnr swsnr assigned swsnr and unassigned swsnr Dec 11, 2014
@da-x
Copy link
Author

da-x commented Dec 11, 2014

(edited for typos/formatting)

With "library-vanilla: True" and "shared:True", ghc links both the static and the dynamic binaries of the libraries, e.g:

-rw-r--r-- 1 dan dan 777792 Dec 11 10:33 ./lib/x86_64-linux-ghc-7.8.3/scientific-0.3.3.2/libHSscientific-0.3.3.2.a
-rwxr-xr-x 1 dan dan 582122 Dec 11 10:33 ./lib/x86_64-linux-ghc-7.8.3/scientific-0.3.3.2/libHSscientific-0.3.3.2-ghc7.8.3.so

This increases the storage footprint of the cabal sandbox by a factor 2. However with executable-dynamic:true the static libraries (*.so files) are not used, and therefore we don't need to link the static archive (*.a files), and therefore we can use library-vanilla:false. From aspects unrelated to flycheck, this seems to be a valid ghc/cabal state (with 7.8, I am not sure about earlier versions).

I also have profiling disabled. The purpose of this configuration is to avoid linking static libraries completely, so that there are no static archive files under the cabal sandbox, and in addition, the executable under the bin directory are dynamically linked to their dependencies in the cabal sandbox which are all arriving as shared libraries. Dynamically linked binaries are much smaller. For example when building the hakyll package the result binary is a few KBs instead of 80 MB.

I am not sure that flycheck is right here, as 'cabal repl' is able to find the dependencies inside the sandbox even though they are no static libraries (for the yaml package, I needed to add -fPIC to cc-options, will push this fix upstream). I think this information can be transferred to flycheck via the get-cabal-configuration.hs method.

$ cabal repl
Preprocessing library yaml-0.8.10...
GHCi, version 7.8.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package array-0.5.0.0 ... linking ... done.
Loading package deepseq-1.3.0.2 ... linking ... done.
Loading package bytestring-0.10.4.0 ... linking ... done.
Loading package text-1.1.0.0 ... linking ... done.
Loading package hashable-1.2.2.0 ... linking ... done.
Loading package scientific-0.3.3.2 ... linking ... done.
Loading package filepath-1.3.0.2 ... linking ... done.
Loading package old-locale-1.0.0.6 ... linking ... done.
Loading package time-1.4.2 ... linking ... done.
Loading package unix-2.7.0.1 ... linking ... done.
Loading package directory-1.2.1.0 ... linking ... done.
Loading package containers-0.5.5.1 ... linking ... done.
Loading package transformers-0.3.0.0 ... linking ... done.
Loading package mtl-2.1.3.1 ... linking ... done.
Loading package exceptions-0.6.1 ... linking ... done.
Loading package stm-2.4.2 ... linking ... done.
Loading package transformers-base-0.4.3 ... linking ... done.
Loading package monad-control-0.3.3.0 ... linking ... done.
Loading package lifted-base-0.2.3.0 ... linking ... done.
Loading package mmorph-1.0.4 ... linking ... done.
Loading package resourcet-1.1.3 ... linking ... done.
Loading package nats-0.2 ... linking ... done.
Loading package unordered-containers-0.2.4.0 ... linking ... done.
Loading package semigroups-0.15.4 ... linking ... done.
Loading package void-0.6.1 ... linking ... done.
Loading package conduit-1.2.3 ... linking ... done.
Loading package attoparsec-0.10.4.0 ... linking ... done.
Loading package blaze-builder-0.3.3.4 ... linking ... done.
Loading package dlist-0.7.1 ... linking ... done.
Loading package syb-0.4.1 ... linking ... done.
Loading package pretty-1.1.1.1 ... linking ... done.
Loading package template-haskell ... linking ... done.
Loading package primitive-0.5.2.1 ... linking ... done.
Loading package vector-0.10.9.1 ... linking ... done.
Loading package aeson-0.6.2.1 ... linking ... done.
Loading object (static) dist/build/c/helper.o ... done
Loading object (static) dist/build/libyaml/api.o ... done
Loading object (static) dist/build/libyaml/dumper.o ... done
Loading object (static) dist/build/libyaml/emitter.o ... done
Loading object (static) dist/build/libyaml/loader.o ... done
Loading object (static) dist/build/libyaml/parser.o ... done
Loading object (static) dist/build/libyaml/reader.o ... done
Loading object (static) dist/build/libyaml/scanner.o ... done
Loading object (static) dist/build/libyaml/writer.o ... done
final link ... done
[1 of 6] Compiling Text.Libyaml     ( Text/Libyaml.hs, interpreted )
[2 of 6] Compiling Data.Yaml        ( Data/Yaml.hs, interpreted )
[3 of 6] Compiling Data.Yaml.Aeson  ( Data/Yaml/Aeson.hs, interpreted )
[4 of 6] Compiling Data.Yaml.Builder ( Data/Yaml/Builder.hs, interpreted )
[5 of 6] Compiling Data.Yaml.Parser ( Data/Yaml/Parser.hs, interpreted )
[6 of 6] Compiling Data.Yaml.Include ( Data/Yaml/Include.hs, interpreted )
Ok, modules loaded: Text.Libyaml, Data.Yaml, Data.Yaml.Aeson, Data.Yaml.Builder, Data.Yaml.Parser, Data.Yaml.Include.
*Text.Libyaml> 

/proc/<pid>/maps shows the loaded DSOs from the cabal sandbox:

...
7f34c08fb000-7f34c0905000 r-xp 00000000 08:04 12587335                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/void-0.6.1/libHSvoid-0.6.1-ghc7.8.3.so
7f34c0905000-7f34c0b04000 ---p 0000a000 08:04 12587335                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/void-0.6.1/libHSvoid-0.6.1-ghc7.8.3.so
7f34c0b04000-7f34c0b05000 r--p 00009000 08:04 12587335                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/void-0.6.1/libHSvoid-0.6.1-ghc7.8.3.so
7f34c0b05000-7f34c0b06000 rw-p 0000a000 08:04 12587335                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/void-0.6.1/libHSvoid-0.6.1-ghc7.8.3.so
7f34c0b06000-7f34c0ba2000 r-xp 00000000 08:04 12587325                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/semigroups-0.15.4/libHSsemigroups-0.15.4-ghc7.8.3.so
7f34c0ba2000-7f34c0da2000 ---p 0009c000 08:04 12587325                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/semigroups-0.15.4/libHSsemigroups-0.15.4-ghc7.8.3.so
7f34c0da2000-7f34c0da3000 r--p 0009c000 08:04 12587325                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/semigroups-0.15.4/libHSsemigroups-0.15.4-ghc7.8.3.so
7f34c0da3000-7f34c0da9000 rw-p 0009d000 08:04 12587325                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/semigroups-0.15.4/libHSsemigroups-0.15.4-ghc7.8.3.so
...

This is how the ghci is executed via 'cabal repr'. Not sure how the information about the DSOs gets there.

/opt/ghc-7.8.3/lib/ghc-7.8.3/bin/ghc -B/opt/ghc-7.8.3/lib/ghc-7.8.3 --interactive -fbuilding-cabal-package -O0 -outputdir dist/build -odir dist/build -hidir dist/build -stubdir dist/build -i -idist/build -i. -idist/build/autogen -Idist/build/autogen -Idist/build -Ic -Ilibyaml -optP-include -optPdist/build/autogen/cabal_macros.h -package-name yaml-0.8.10 -hide-all-packages -no-user-package-db -package-db /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/x86_64-linux-ghc-7.8.3-packages.conf.d -package-db dist/package.conf.inplace -package-id aeson-0.6.2.1-51da61d52752149eabb23fda6e899080 -package-id attoparsec-0.10.4.0-7ee6fb69f8d373108d8583068f8120f7 -package-id base-4.7.0.1-e4b74d27ad8c8987c63abc42a80e7335 -package-id bytestring-0.10.4.0-73edd0ec5827e07aeacc42c99897b171 -package-id conduit-1.2.3-03a25b103caf9f8dfb81d0b8b7701c4e -package-id containers-0.5.5.1-23e2a2b94d6e452c773209f31d8672c5 -package-id directory-1.2.1.0-d2e4ec65a1afb7a1a6cc06930dc01ea8 -package-id filepath-1.3.0.2-1580a61d3226e4be45fe2130dc2881e3 -package-id resourcet-1.1.3-d91528b040b56ff8f0d7fc56ddb346d9 -package-id scientific-0.3.3.2-884380915060af766df7df17ddeaca91 -package-id text-1.1.0.0-67a836d0620144282e62b9f73f92f738 -package-id transformers-0.3.0.0-16a97696ae672940f1523b262e566ba5 -package-id unordered-containers-0.2.4.0-f6bbc0197292a7807d3274e6b47bc6af -package-id vector-0.10.9.1-b7220db8da564c8c2b77e5d55b53a7d4 -XHaskell98 Text.Libyaml Data.Yaml Data.Yaml.Aeson Data.Yaml.Builder Data.Yaml.Parser Data.Yaml.Include dist/build/c/helper.o dist/build/libyaml/api.o dist/build/libyaml/dumper.o dist/build/libyaml/emitter.o dist/build/libyaml/loader.o dist/build/libyaml/parser.o dist/build/libyaml/reader.o dist/build/libyaml/scanner.o dist/build/libyaml/writer.o -Wall

@swsnr
Copy link
Contributor

swsnr commented Dec 11, 2014

@da-x I wasn't aware that library-vanilla only refers to static archives, but other than that you are not telling me anything new, nor does what you have said contradict my previous comment: Flycheck doesn't pass -shared either, so for Flycheck there are still no dependencies available, even if the shared libraries are present.

cabal repl finds your dependencies because GHCi 7.8 uses dynamic linking by default—which is precisely the reason why you end up with two libraries by default, and why you can't opt out from dynamic libraries in cabal install. Internally, Cabal passes -dynamic-too to GHC while building in order to build static and dynamic libraries for all packages. In GHC 7.6, cabal repl would fail with static libraries absent, because it used static linking.

Besides, for this reason Shared: True doesn't affect your dependencies at all. It only affects your own package, and only for cabal build.

Flycheck however has no chance to figure out whether your sandbox is build with or without static libraries. All it could do is to pass -shared by default, and hope that dynamic libraries are present. That would work well for 7.8, where these are always there anyway, but not so well for 7.6, which doesn't build dynamic libraries by default.

OTOH, all that you gain by enable-vanilla: False is storage space. Sure, sandboxes can get (very) large, but does it really matter if it's 1GB, or just 500MB?

@da-x
Copy link
Author

da-x commented Dec 11, 2014

Beside space, it also affects module build compilation time, though I don't have actual measurements.

test.sh:

#!/bin/bash

cabal unpack yaml-0.8.10
cd yaml-0.8.10
cabal sandbox init
cabal install

shared-only, library-vanilla: False

time ../test.sh  93.37s user 7.63s system 224% cpu 44.960 total

shared+static, library-vanilla: True

time ../test.sh  108.30s user 8.15s system 225% cpu 51.753 total

Space:

du -s *
13528   shared
26504   static

How about detecting ghc's version, then passing -shared if it's 7.8 and above and not pass it if otherwise? If you post a branch I'll test it.

@swsnr
Copy link
Contributor

swsnr commented Dec 11, 2014

@da-x I appreciate your offer, but you'll understand that I have a hard time to see why I should write the code that saves you some 15 seconds of build time and a dozen of megabytes disk space. I find that the balance between effort and gain isn't quite right here.

If that's so important to you, copy the definition of haskell-ghc to your init.el, add -shared to it, and call it a day.

@swsnr swsnr added wontfix and removed needs-input labels Dec 11, 2014
@swsnr
Copy link
Contributor

swsnr commented Dec 11, 2014

@da-x I'll add a catch-all option for arbitrary arguments to haskell-ghc, so as soon as https://github.com/flycheck/flycheck/issues/542 is implemented, you'll be able to use (setq flycheck-ghc-args '("-shared")).

Closing as wontfix, since I don't think that this case warrants some special work around in Flycheck.

@swsnr swsnr closed this as completed Dec 11, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants