Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault when running hls with TH and libpq #1982

Closed
aschmois opened this issue Jun 25, 2021 · 22 comments
Closed

Segfault when running hls with TH and libpq #1982

aschmois opened this issue Jun 25, 2021 · 22 comments
Labels
type: bug Something isn't right: doesn't work as intended, documentation is missing/outdated, etc..

Comments

@aschmois
Copy link

Your environment

Output of haskell-language-server --probe-tools or haskell-language-server-wrapper --probe-tools:

haskell-language-server-wrapper --probe-tools
haskell-language-server version: 1.1.0.0 (GHC: 8.10.4) (PATH: /home/andres/.ghcup/bin/haskell-language-server-wrapper-1.1.0) (GIT hash: f1c096927186a93d8e3ccd4fe8385cc1b070350b)
Tool versions found on the $PATH
cabal:          3.4.0.0
stack:          2.7.1
ghc:            8.10.4

Which OS do you use:
Ubuntu 20.04

Which lsp-client do you use:
VS Code (but irrelevant to the problem)

Describe your project (alternative: link to the project):
https://github.com/aschmois/th-pq-segfault

Steps to reproduce

Run haskell-language-server-wrapper

Expected behaviour

Completed hls command

Actual behaviour

Segfault exit

Include debug information

Execute in the root of your project the command haskell-language-server --debug . and paste the logs here:

Debug output:
haskell-language-server-wrapper --debug
Found "/home/foo/projects/example/hie.yaml" for "/home/foo/projects/example/a"
Module "/home/foo/projects/example/a" is loaded by Cradle: Cradle {cradleRootDir = "/home/foo/projects/example", cradleOptsProg = CradleAction: Stack}
Run entered for haskell-language-server-wrapper(haskell-language-server-wrapper) Version 1.1.0.0, Git revision f1c096927186a93d8e3ccd4fe8385cc1b070350b (dirty) x86_64 ghc-8.10.4
Current directory: /home/foo/projects/example
Operating system: linux
Arguments: ["--debug"]
Cradle directory: /home/foo/projects/example
Cradle type: Stack

Tool versions found on the $PATH
cabal:          3.4.0.0
stack:          2.7.1
ghc:            8.10.4


Consulting the cradle to get project GHC version...
Project GHC version: 8.10.4
haskell-language-server exe candidates: ["haskell-language-server-8.10.4","haskell-language-server-8.10","haskell-language-server"]
Launching haskell-language-server exe at:/home/foo/.ghcup/bin/haskell-language-server-8.10.4
haskell-language-server version: 1.1.0.0 (GHC: 8.10.4) (PATH: /home/foo/.ghcup/bin/haskell-language-server-8.10.4~1.1.0) (GIT hash: f1c096927186a93d8e3ccd4fe8385cc1b070350b)
 ghcide setup tester in /home/foo/projects/example.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Step 1/4: Finding files to test in /home/foo/projects/example
Found 3 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle
  (/home/foo/projects/example/hie.yaml)

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
2021-06-25 18:44:24.988266282 [ThreadId 4] DEBUG hls:   Set files of interest to: [(NormalizedFilePath "/home/foo/projects/example/pqexample/src/PQ.hs",OnDisk),(NormalizedFilePath "/home/foo/projects/example/src/B.hs",OnDisk),(NormalizedFilePath "/home/foo/projects/example/src/A.hs",OnDisk)]
2021-06-25 18:44:24.989627599 [ThreadId 79] DEBUG hls:  hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/src/B.hs"
2021-06-25 18:44:24.989635104 [ThreadId 78] DEBUG hls:  hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/pqexample/src/PQ.hs"
2021-06-25 18:44:24.989662114 [ThreadId 77] DEBUG hls:  hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/src/A.hs"
2021-06-25 18:44:24.990188754 [ThreadId 93] INFO hls:   Consulting the cradle for "src/B.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/home/foo/projects/example", cradleOptsProg = CradleAction: Stack}
> Configuring GHCi with the following packages: example
> /home/foo/projects/example/.stack-work/install/x86_64-linux-tinfo6/e857431271b4cd29c78b6f2c970b9abcee6d8c8df77a6c0b3dbb05e050835ce0/8.10.4/pkgdb:/home/foo/.stack/snapshots/x86_64-linux-tinfo6/e857431271b4cd29c78b6f2c970b9abcee6d8c8df77a6c0b3dbb05e050835ce0/8.10.4/pkgdb:/home/foo/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.4/lib/ghc-8.10.4/package.conf.d
2021-06-25 18:44:26.009985491 [ThreadId 93] DEBUG hls:  Session loading result: Right (ComponentOptions {componentOptions = ["-i","-odir=/home/foo/projects/example/.stack-work/odir","-hidir=/home/foo/projects/example/.stack-work/odir","-hide-all-packages","-i/home/foo/projects/example/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0/build","-i/home/foo/projects/example/src","-i/home/foo/projects/example/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0/build/autogen","-i/home/foo/projects/example/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0/build/global-autogen","-stubdir=/home/foo/projects/example/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0/build","-package-id=base-4.14.1.0","-package-id=template-haskell-2.16.0.0","-package-id=pqexample-0.0.0-AYIBm2U3Aq37TXbYsGw9pl","-optP-include","-optP/home/foo/projects/example/.stack-work/ghci/78a2b7ee/cabal_macros.h","-ghci-script=/tmp/haskell-stack-ghci/f9e95283/ghci-script","-package-db","/home/foo/projects/example/.stack-work/install/x86_64-linux-tinfo6/e857431271b4cd29c78b6f2c970b9abcee6d8c8df77a6c0b3dbb05e050835ce0/8.10.4/pkgdb","-package-db","/home/foo/.stack/snapshots/x86_64-linux-tinfo6/e857431271b4cd29c78b6f2c970b9abcee6d8c8df77a6c0b3dbb05e050835ce0/8.10.4/pkgdb","-package-db","/home/foo/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.4/lib/ghc-8.10.4/package.conf.d"], componentRoot = "/home/foo/projects/example", componentDependencies = ["example.cabal","package.yaml","stack.yaml"]},"/home/foo/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.4/lib/ghc-8.10.4")
2021-06-25 18:44:26.0532338 [ThreadId 93] INFO hls:     Using interface files cache dir: /home/foo/.cache/ghcide/main-a70ef1e6d9d19e2d3c084465e0fae5c1e4416e60
2021-06-25 18:44:26.053352853 [ThreadId 93] INFO hls:   Making new HscEnv[main]
2021-06-25 18:44:26.057649144 [ThreadId 93] DEBUG hls:  New Component Cache HscEnvEq: (([],Just HscEnvEq 5),fromList [("/home/foo/projects/example/hie.yaml",Just 2021-06-25 22:41:56.808948705 UTC),("example.cabal",Just 2021-06-25 22:30:37.774550911 UTC),("package.yaml",Nothing),("stack.yaml",Just 2021-06-25 22:30:57.242622243 UTC)])
2021-06-25 18:44:26.060152966 [ThreadId 93] DEBUG hls:  Known files updated: fromList [(TargetModule A,fromList ["/home/foo/projects/example/src/A.hs"]),(TargetModule B,fromList ["/home/foo/projects/example/src/B.hs"]),(TargetFile NormalizedFilePath "/home/foo/projects/example/src/B.hs",fromList ["/home/foo/projects/example/src/B.hs"])]
2021-06-25 18:44:26.060493996 [ThreadId 34] DEBUG hls:  Finishing build session(exception: AsyncCancelled)
2021-06-25 18:44:26.06059176 [ThreadId 93] DEBUG hls:   Restarting build session (aborting the previous one took 0.00s)
2021-06-25 18:44:26.061951083 [ThreadId 178] DEBUG hls: hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/src/B.hs"
2021-06-25 18:44:26.061986299 [ThreadId 182] DEBUG hls: hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/src/A.hs"
2021-06-25 18:44:26.061957265 [ThreadId 179] DEBUG hls: hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/pqexample/src/PQ.hs"
2021-06-25 18:44:26.063105552 [ThreadId 198] INFO hls:  Consulting the cradle for "pqexample/src/PQ.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/home/foo/projects/example", cradleOptsProg = CradleAction: Stack}
> Configuring GHCi with the following packages: pqexample
> /home/foo/projects/example/.stack-work/install/x86_64-linux-tinfo6/e857431271b4cd29c78b6f2c970b9abcee6d8c8df77a6c0b3dbb05e050835ce0/8.10.4/pkgdb:/home/foo/.stack/snapshots/x86_64-linux-tinfo6/e857431271b4cd29c78b6f2c970b9abcee6d8c8df77a6c0b3dbb05e050835ce0/8.10.4/pkgdb:/home/foo/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.4/lib/ghc-8.10.4/package.conf.d
2021-06-25 18:44:27.106834505 [ThreadId 198] DEBUG hls: Session loading result: Right (ComponentOptions {componentOptions = ["-i","-odir=/home/foo/projects/example/.stack-work/odir","-hidir=/home/foo/projects/example/.stack-work/odir","-hide-all-packages","-i/home/foo/projects/example/pqexample/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0/build","-i/home/foo/projects/example/pqexample/src","-i/home/foo/projects/example/pqexample/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0/build/autogen","-i/home/foo/projects/example/pqexample/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0/build/global-autogen","-stubdir=/home/foo/projects/example/pqexample/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0/build","-lpq","-package-id=base-4.14.1.0","-optP-include","-optP/home/foo/projects/example/.stack-work/ghci/78665a2f/cabal_macros.h","-ghci-script=/tmp/haskell-stack-ghci/6f47add2/ghci-script","-package-db","/home/foo/projects/example/.stack-work/install/x86_64-linux-tinfo6/e857431271b4cd29c78b6f2c970b9abcee6d8c8df77a6c0b3dbb05e050835ce0/8.10.4/pkgdb","-package-db","/home/foo/.stack/snapshots/x86_64-linux-tinfo6/e857431271b4cd29c78b6f2c970b9abcee6d8c8df77a6c0b3dbb05e050835ce0/8.10.4/pkgdb","-package-db","/home/foo/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.4/lib/ghc-8.10.4/package.conf.d"], componentRoot = "/home/foo/projects/example/pqexample", componentDependencies = ["pqexample/pqexample.cabal","pqexample/package.yaml","stack.yaml"]},"/home/foo/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.4/lib/ghc-8.10.4")
2021-06-25 18:44:27.148598826 [ThreadId 198] INFO hls:  Using interface files cache dir: /home/foo/.cache/ghcide/main-d720665dfd93b30fab40a64d4b09404d904bc716
2021-06-25 18:44:27.148714323 [ThreadId 198] INFO hls:  Using interface files cache dir: /home/foo/.cache/ghcide/main-d720665dfd93b30fab40a64d4b09404d904bc716
2021-06-25 18:44:27.148766752 [ThreadId 198] INFO hls:  Making new HscEnv[main,main]
2021-06-25 18:44:27.152914914 [ThreadId 198] DEBUG hls: New Component Cache HscEnvEq: (([],Just HscEnvEq 7),fromList [("/home/foo/projects/example/hie.yaml",Just 2021-06-25 22:41:56.808948705 UTC),("pqexample/package.yaml",Nothing),("pqexample/pqexample.cabal",Just 2021-06-25 22:32:17.438916399 UTC),("stack.yaml",Just 2021-06-25 22:30:57.242622243 UTC)])
2021-06-25 18:44:27.15370085 [ThreadId 198] DEBUG hls:  New Component Cache HscEnvEq: (([],Just HscEnvEq 8),fromList [("/home/foo/projects/example/hie.yaml",Just 2021-06-25 22:41:56.808948705 UTC),("example.cabal",Just 2021-06-25 22:30:37.774550911 UTC),("package.yaml",Nothing),("stack.yaml",Just 2021-06-25 22:30:57.242622243 UTC)])
2021-06-25 18:44:27.156896523 [ThreadId 198] DEBUG hls: Known files updated: fromList [(TargetModule PQ,fromList ["/home/foo/projects/example/pqexample/src/PQ.hs"]),(TargetModule A,fromList ["/home/foo/projects/example/src/A.hs"]),(TargetFile NormalizedFilePath "/home/foo/projects/example/pqexample/src/PQ.hs",fromList ["/home/foo/projects/example/pqexample/src/PQ.hs"]),(TargetModule B,fromList ["/home/foo/projects/example/src/B.hs"]),(TargetFile NormalizedFilePath "/home/foo/projects/example/src/B.hs",fromList ["/home/foo/projects/example/src/B.hs"])]
2021-06-25 18:44:27.157257301 [ThreadId 122] DEBUG hls: Finishing build session(exception: AsyncCancelled)
2021-06-25 18:44:27.157369682 [ThreadId 198] DEBUG hls: Restarting build session (aborting the previous one took 0.00s)
2021-06-25 18:44:27.158906439 [ThreadId 342] DEBUG hls: hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/pqexample/src/PQ.hs"
2021-06-25 18:44:27.158877244 [ThreadId 338] DEBUG hls: hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/src/A.hs"
2021-06-25 18:44:27.161917314 [ThreadId 444] DEBUG hls: hlint:getIdeas:file:NormalizedFilePath "/home/foo/projects/example/src/B.hs"
haskell-language-server-wrapper: callProcess: /home/foo/.ghcup/bin/haskell-language-server-8.10.4 "--debug" (exit -11): failed

Me and a collegue went down the route to figure out why we were getting segfaults when compiling our project. We drilled down until we could get the smallest reproducible project. The issue ends up being that if you import a project that loads pq (or libpq) regardless of its contents you get a segfault with no information as to why. If your own project imports pq it does not segfault. If you depend on a project that imports pq but don't import that project then it won't segfault. It was very interesting to reach this conclusion. You can play around with the example project to see what segfaults and what doesn't. I would be interested to see if different OS's react differently though. Also it's possible that my libpq is at fault as well.

@codygman
Copy link

codygman commented Jun 26, 2021

I can't reproduce from nixos using stacks nix integration to provide libpq using 9cbac50d530397ed519a6c76cfdeed645d9f8956, I might be able to try from an ubuntu vm later:

~/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault $ haskell-language-server-wrapper 
Found "/home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault/hie.yaml" for "/home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault/a"
Module "/home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault/a" is loaded by Cradle: Cradle {cradleRootDir = "/home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault", cradleOptsProg = CradleAction: Stack}
Run entered for haskell-language-server-wrapper(haskell-language-server-wrapper) Version 1.1.0.0 x86_64 ghc-8.10.4
Current directory: /home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault
Operating system: linux
Arguments: []
Cradle directory: /home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault
Cradle type: Stack

Tool versions found on the $PATH
cabal:		3.4.0.0
stack:		2.7.1
ghc:		8.10.4


Consulting the cradle to get project GHC version...
Project GHC version: 8.10.4
haskell-language-server exe candidates: ["haskell-language-server-8.10.4","haskell-language-server-8.10","haskell-language-server"]
Launching haskell-language-server exe at:/run/current-system/sw/bin/haskell-language-server-8.10.4
haskell-language-server version: 1.1.0.0 (GHC: 8.10.4) (PATH: /nix/store/nr5rrd8slk8limz0yk2yrzcqv4vf1k6y-haskell-language-server-1.1.0.0/bin/haskell-language-server)
 ghcide setup tester in /home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Step 1/4: Finding files to test in /home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault
Found 3 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle
  (/home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault/hie.yaml)

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
2021-06-25 21:40:11.816452004 [ThreadId 82] INFO hls:	Consulting the cradle for "pqexample/src/PQ.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault", cradleOptsProg = CradleAction: Stack}
> Configuring GHCi with the following packages: pqexample
> /home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault/.stack-work/install/x86_64-linux-nix/bf9fd1ff1ae588cb8e642dd7bbc8c5b1299cd10a51c5a3ada44fa584d7a449e0/8.10.4/pkgdb:/home/cody/.stack/snapshots/x86_64-linux-nix/bf9fd1ff1ae588cb8e642dd7bbc8c5b1299cd10a51c5a3ada44fa584d7a449e0/8.10.4/pkgdb:/nix/store/d8mbb3cjp24qi0mcwb0rkmw3jhl91rjc-ghc-8.10.4/lib/ghc-8.10.4/package.conf.d
2021-06-25 21:40:18.482948714 [ThreadId 82] INFO hls:	Using interface files cache dir: /home/cody/.cache/ghcide/main-f15759b5ccd5ca24b93acfe0be86fc6e64062e4d
2021-06-25 21:40:18.483191682 [ThreadId 82] INFO hls:	Making new HscEnv[main]
2021-06-25 21:40:18.494930499 [ThreadId 174] INFO hls:	Consulting the cradle for "src/A.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault", cradleOptsProg = CradleAction: Stack}
> Configuring GHCi with the following packages: example
> /home/cody/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault/.stack-work/install/x86_64-linux-nix/bf9fd1ff1ae588cb8e642dd7bbc8c5b1299cd10a51c5a3ada44fa584d7a449e0/8.10.4/pkgdb:/home/cody/.stack/snapshots/x86_64-linux-nix/bf9fd1ff1ae588cb8e642dd7bbc8c5b1299cd10a51c5a3ada44fa584d7a449e0/8.10.4/pkgdb:/nix/store/d8mbb3cjp24qi0mcwb0rkmw3jhl91rjc-ghc-8.10.4/lib/ghc-8.10.4/package.conf.d
2021-06-25 21:40:25.033914653 [ThreadId 174] INFO hls:	Using interface files cache dir: /home/cody/.cache/ghcide/main-b2b08d5ef87e8604285cfae8a7b78379a57abbe4
2021-06-25 21:40:25.034141431 [ThreadId 174] INFO hls:	Using interface files cache dir: /home/cody/.cache/ghcide/main-b2b08d5ef87e8604285cfae8a7b78379a57abbe4
2021-06-25 21:40:25.034243741 [ThreadId 174] INFO hls:	Making new HscEnv[main,main]
2021-06-25 21:40:25.297784631 [ThreadId 468] INFO hls:	finish: User TypeCheck (took 0.25s)
2021-06-25 21:40:25.298777344 [ThreadId 479] INFO hls:	finish: GetHie (took 0.00s)

Completed (3 files worked, 0 files failed)
2021-06-25 21:40:25.299049512 [ThreadId 480] INFO hls:	finish: GenerateCore (took 0.00s)

@aschmois Can you see if you get a segfault after checking my branch out at that commit? Nothing more than a stack build should be needed on your part after switching branches.

EDIT: For completeness here's this output:

~/org-roam/data/42/66517f-9ddf-40fd-b8c1-7b444364dc1b/th-pq-segfault $ haskell-language-server-wrapper --probe-tools
haskell-language-server version: 1.1.0.0 (GHC: 8.10.4) (PATH: /nix/store/nr5rrd8slk8limz0yk2yrzcqv4vf1k6y-haskell-language-server-1.1.0.0/bin/haskell-language-server-wrapper)
Tool versions found on the $PATH
cabal:		3.4.0.0
stack:		2.7.1
ghc:		8.10.4

@codygman
Copy link

codygman commented Jun 26, 2021

I would be interested to see if different OS's react differently though. Also it's possible that my libpq is at fault as well.

Same! Let's see, if we assume that whatever is on the search pages of ubuntu and nix package search respectively then:

nix stable: postgres Version: 11.12
nix unstable: Version: 11.11 (what? interesting it's older)
ubuntu focal: 12+214ubuntu0.1
ubuntu impish: 13+226

So we are almost certainly using different major versions, potentially a difference of two major versions.

@codygman
Copy link

codygman commented Jun 26, 2021

I added a recent commit to my fork which tries different postgres versions.

Upon re-reading your description though, It looks like I needed to clone that external library to reproduce?

I assumed that cloning the repo was enough to reproduce.

@codygman
Copy link

Ok I have reproduced with all of the above versions.

@codygman
Copy link

codygman commented Jun 26, 2021

The segfault only happens if you run with haskell-language-server-wrapper, but without it passes just fine.

I bet if you use the binary directly rather than the wrapper you won't get this segfault.

@jneira jneira added type: bug Something isn't right: doesn't work as intended, documentation is missing/outdated, etc.. type: template haskell related labels Jun 26, 2021
@mpickering
Copy link
Contributor

So how did you reproduce this @codygman? This sounds similar to all the other issues with the in-built linker. Can you reproduce with a dynamically linked ghcide?

@codygman
Copy link

codygman commented Jun 28, 2021

@mpickering I did reproduce with a nix install of haskell-language-server 1.1.0.0 when using haskell-language-server-wrapper at this commit.

I'm not sure if the nix version is dynamically linked or not.

I don't have time now, but I'll add it to my to do list.

@codygman
Copy link

codygman commented Jul 1, 2021

This one was bizarre and triggering it requires:

  • the library interaction above with a native dep/TH
  • stack
  • postgres 12 compiled with libp11-kit
  • haskell language server not built from source downloaded from the release page

Note this isn't reproducible with cabal. It also doesn't happen using haskell-language-server from nix on Ubuntu using stack --nix.

Open questions:

  • what does stack do differently than cabal if anything here that could cause this interaction?
  • is this because of a different code path taken for stack in Haskell language server?
  • Is this new code using initDynLinker somehow related?

This investigation was done during working hours earlier with @aschmois, so I hope that I'm not sure I have all the details about correct.

If I have the time tomorrow or this weekend I plan to try and make a minimal reproduction and dig into this a little more.

If any of this strangeness is familiar to anyone though, any advice or help is appreciated.

@codygman
Copy link

codygman commented Jul 1, 2021

I did some more research and it looks like "p11-kit cannot be used as a static library":

I tried to download p11-kit-0.23.22 and compile it using:
./configure --enable-static
and I run into:
configure: error: p11-kit cannot be used as a static library
Is that intentional behavior or bug?

The author says:

I'm afraid to say it's intentional. p11-kit uses ELF constructor for library initialization and that wouldn't work otherwise. Although it is possible to expose an explicit initialization function as part of the API (currently used in tests but hidden), it would require the calling sites (e.g., gnutls) to call that function explicitly.

It really feels like this could be related, but this is not my area of expertise.

@codygman
Copy link

codygman commented Jul 1, 2021

And another related issue here: nh2/static-haskell-nix#86

Building stack project fails with p11-kit

@codygman
Copy link

codygman commented Jul 1, 2021

Another thing potentially interesting is that when testing this using stack --nix we got a test failure building hls-splice-plugin (where I think) this test timed out and had to rerun stack --nix and it passed the second time.

It's possible that that test timing out isn't too surprising, but since it was a golden test I assumed it might be fast. It's also possible that even if this isn't directly related to this, it deserves its own issue.

@codygman
Copy link

codygman commented Jul 1, 2021

Finally I read something in a related issue that makes me wonder if GHC's dynamic linker is getting used in this particular case:

Yes, GHC has its own dynamic linker, in the rts, and it gets used in certain cases:

when GHC the binary is linked statically, by GHCI and TH
when ghc the library is linked statically, by TH and the ghc interactive api

Implementing a dynamic linker with support for >=3 platforms is a risky and complex exercise. Fortunately there is an easy opt-out (-dynamic) but unfortunately that opt-out makes distribution much harder, and the precompiled HLS binaries are statically linked.

As I mentioned it would be great if you can minify this bug and report it upstream in GHC.

From: #469 (comment)

@aschmois
Copy link
Author

aschmois commented Jul 2, 2021

To summarize:

  • The problematic library is p11-kit, none of its dependencies cause a segfault. This library is a dependency of ldap of which can be a dependency of pq. We are working around the issue by compiling pq without ldap. (ubuntu's libpq-dev contains ldap)
  • The exact same setup will not segfault with a cabal hie
  • When pq is built with musl instead of glibc hls does not seg fault
  • A dynamically linked hls will not segfault with the same p11-kit

@pepeiborra
Copy link
Collaborator

Thank you for your investigation!

The ghc built-in linker is used only when HLS is statically linked, which is the case for the binaries distributed from the GitHub project page, ghcup, and Nix.

You can ask your binary whether it's statically or dynamically linked as follows:

  • ghcup binary:
$ /Users/pepeiborra/.ghcup/bin/haskell-language-server-8.10.2  +RTS --info | grep "RTS way" | grep dyn && echo "Dynamic" || echo "Static"
Static
  • built from source with cabal and executable-dynamic: True:
$ /Users/pepeiborra/scratch/ide/dist-newstyle/build/x86_64-osx/ghc-8.10.5/haskell-language-server-1.2.0.0/x/haskell-language-server/build/haskell-language-server/haskell-language-server  +RTS --info | grep "RTS way" | grep dyn --quiet  && echo "Dynamic"
Dynamic
  • nix-shell -p haskellPackages.haskell-language-server
[nix-shell:~/scratch/ide]$ haskell-language-server  +RTS --info | grep "RTS way" | grep dyn | true && echo "Dynamic" || echo "Static"
Static

Can you confirm that the segfault with p11-kit reproduces consistently with a statically linked binary?

@codygman
Copy link

codygman commented Jul 3, 2021

Thank you for your investigation!

No problem! We really want to help Haskell language server be an option in Industry for everyone including ourselves and our fairly large and complicated work codebase.

Can you confirm that the segfault with p11-kit reproduces consistently with a statically linked binary?

Yes, I can confirm that. We tested 5-10 times say in an empty ubuntu Docker container with:

  • binaries distributed from the GitHub project page
  • stack
  • the linked project, replacing the postgres dependency with p11-kit

I can't make a copy/paste repro yet, but it's on my to-do list.

@pepeiborra
Copy link
Collaborator

pepeiborra commented Jul 4, 2021

The next step in this process is to reproduce with plain GHC using a static build and GHCi:

  1. Build GHC with DYNAMIC_GHC_PROGRAMS=NO. If you are fluent in Nix, just use https://github.com/NixOS/nixpkgs/blob/40bb731ec71bdd7e85ae8c00208d90042e98ce21/pkgs/development/compilers/ghc/8.10.4.nix#L31
  2. Load your repro in GHCi - a simple program that exercises pq/ldap
  3. Run the program in GHCi. It should crash now
  4. Open a bug report in the GHC issue tracker with the appropriate tag for linker bugs

@mpickering
Copy link
Contributor

An alternative way to build GHC from source is:

./hadrian/build.sh --flavour=default+no_dynamic_ghc

@mpickering
Copy link
Contributor

mpickering commented Nov 8, 2021

@codygman Do you know how to reproduce this on NixOS? I have built a static version of GHC and loading the example project using p11-kit works fine in GHCi.

@tek
Copy link
Contributor

tek commented Nov 8, 2021

I think this is fixed in 8.10.7

@GavinRay97
Copy link

GavinRay97 commented Nov 9, 2021

I think this is also relevant to:
#2314

We use libpq and Template Haskell. Get identical output to OP's post.
Also Ubuntu, when I got this issue.

Seems all related to be related to Static vs Dynamic linking issues, and pending Template Haskell fix in GHC here:
https://gitlab.haskell.org/ghc/ghc/-/issues/20628

(Just leaving this here as a note for others/organizational purposes)

I think this is fixed in 8.10.7

FWIW, I tried with 8.10.2 then 8.10.7 and got the same issues, but it could also be a complication of other things in our codebase. We have quite some dependencies + codebase size.

@codygman
Copy link

@codygman Do you know how to reproduce this on NixOS? I have built a static version of GHC and loading the example project using p11-kit works fine in GHCi.

@mpickering I only remember reproducing this with stack --nix, but I recall building a static ghc to follow the above instructions but will have to check my notes later to see if I followed through with it or got pulled to something else.

@jneira
Copy link
Member

jneira commented Jan 31, 2022

I am gonna close this issue as all compiler crashes seems to have the same root cause:

If any of you think the issue should not be included generically feel free to reopen it (with a brief explanation if possible)
Thanks all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't right: doesn't work as intended, documentation is missing/outdated, etc..
Projects
None yet
Development

No branches or pull requests

7 participants