Deadlock during "cabal install" using parallel compilation

[kolmodin]

This is using the newly released cabal-1.18.0. I built it using a clean user package-db, using GHC-7.6.2.

Using parallel compilation and sandbox, cabal fails in building the packages and instead hogs 100% CPU.

$ git clone https://github.com/kolmodin/spdy
$ cd spdy

$ grep jobs ~/.cabal/config 
jobs: $ncpus

$ nproc # number of cpus
32

$ cabal --version
cabal-install version 1.18.0
using version 1.18.0 of the Cabal library 


$ cabal update
$ cabal sandbox init
$ cabal install --only-dependencies 
Resolving dependencies...
Notice: installing into a sandbox located at
/usr/local/google/home/kolmodin/code/spdy/.cabal-sandbox
Configuring ansi-terminal-0.6...
Configuring asn1-types-0.2.0...
Configuring base-unicode-symbols-0.2.2.4...
Configuring binary-0.7.1.0...
Configuring base64-bytestring-1.0.0.1...
Configuring byteable-0.1.1...
Downloading byteorder-1.0.4...
Configuring cereal-0.3.5.2...
Configuring cipher-rc4-0.1.2...
Configuring data-default-class-0.0.1...
Configuring date-cache-0.3.0...
Configuring dlist-0.5...
Downloading entropy-0.2.2.2...
Configuring byteorder-1.0.4...
Downloading file-embed-0.0.4.9...
Configuring nats-0.1...
Configuring primitive-0.5.0.1...
Configuring entropy-0.2.2.2...
Configuring random-1.0.1.1...
Downloading largeword-1.0.5...
Configuring stm-2.4.2...
Configuring stringsearch-0.3.6.4...
Configuring file-embed-0.0.4.9...
Downloading syb-0.4.1...
Configuring tagged-0.4.5...
Configuring transformers-0.3.0.0...
Configuring largeword-1.0.5...
Configuring unix-compat-0.4.1.1...
Downloading text-0.11.3.1...
Configuring word8-0.0.3...

After this, nothing happens. The cabal process is stuck in some loop, using all cpu it can get.

If I explicitly specify fewer CPUs, it still fails.

$ cabal install --only-dependencies --jobs=2
Resolving dependencies...
Notice: installing into a sandbox located at
/usr/local/google/home/kolmodin/tmp/spdy/.cabal-sandbox
Configuring ansi-terminal-0.6...
Configuring base-unicode-symbols-0.2.2.4...

A package with fewer dependencies also fails, even with few CPUs.

$ cabal install cabal-install --jobs=2 --package-db=clear --package-db=global

[23Skidoo]

Thanks! This is very interesting. Which OS/architecture are you on?

[kolmodin]

OS is Goobuntu, essentially Ubuntu 12.04. The machine is x86_64.

It did manage a few times to install cabal-install, so it seems it's not 100% reproducable.

I tried with ghc 7.6.3, same thing.
I'd like to try to reproduce with ghc 7.7 as well, but some dependencies said they don't support base-4.7 and I didn't have time to setup local versions to try with.

[23Skidoo]

Thanks, will try to reproduce.

[23Skidoo]

So far failed to reproduce on an 8-core EC2 instance. Will try on a 16-core one.

[23Skidoo]

Failed to reproduce on a 16-core instance running Ubuntu 12.04/x86_64.

[23Skidoo]

@kolmodin

I have a feeling that this is a GHC RTS bug that only happens on systems with a large number of cores. Unfortunately, the largest EC2 instance I can rent has only 16 cores, which apparently isn't enough to trigger this behaviour.

If you have other ideas on how to reproduce this, I'll be happy to follow your advice.

I'd like to try to reproduce with ghc 7.7 as well, but some dependencies said they don't support base-4.7 and I didn't have time to setup local versions to try with.

IIRC it's only HTTP. All you need to do is relax the version bound in the .cabal file.