hspec-core-2.4.4 test suite fails "Test.Hspec.Core.Runner.hspecResult runs specs in parallel"

[peti]

Citing from https://nix-cache.s3.amazonaws.com/log/d7wb0k5k2vdnm92lrfri6clk32amqy8l-hspec-core-2.4.4.drv:

Failures:

  test/Test/Hspec/Core/RunnerSpec.hs:468: 
  1) Test.Hspec.Core.Runner.hspecResult runs specs in parallel
       expected: Just (Summary {summaryExamples = 10, summaryFailures = 0})
        but got: Nothing

Randomized with seed 990025833

[sol]

@peti Is this a spontaneous failure, or reproducible?

[peti]

It has happened in our builds more than once already, but it does not happen every time the build is run, i.e. I've seen that build succeed, too.

[sol]

I'm not able to reproduce this issue locally. The script I use is:

#!/bin/bash

set -o errexit

while :
do
  ./dist/build/spec/spec -m 'Test.Hspec.Core.Runner.hspecResult runs specs in parallel'
done

The corresponding test case:

it "runs specs in parallel" $ do
  let n = 10
      t = 0.01
      dt = t * (fromIntegral n / 2)
  r <- timeout dt . silence . withArgs ["-j", show n] . H.hspecResult . H.parallel $ do
    replicateM_ n (H.it "foo" $ sleep t)
  r `shouldBe` Just (H.Summary n 0)

So what we are testing here is that 10 concurrent jobs that do nothing for 10ms each won't take any longer than 50ms. This includes Hspec startup time, which could be the culprint, as it's doing some IO by now. Is this on NFS or something?

Any suggestions on how to test this in a more robust way are welcome.

[peti]

Is this on NFS or something?

Yes. The build server uses both network file systems and virtual machines, which might cause some delays, and it also tends to be under an extremely high load.

[sol]

One way to deal with it could be to skip that particular test by passing --skip 'Test.Hspec.Core.Runner.hspecResult runs specs in parallel' to the test driver.

[peti]

OK, I used that work-around in NixOS/cabal2nix@74d58ee.

[sol]

@peti I'm not aware of a way to make sure that this spontaneous failure can never ever happen. What I did is make it less likely.

We spawn 100 jobs. Each job waits 0.01s. Doing this sequentially would take >= 1s. To assert that it's done in parallel, we timeout after 0.5s. If the test machine cannot finis the 100 jobs within 0.5s you will see a spontaneous failure.

(before dc074ae this was 10 jobs and the timeout was 0.05)

[peti]

OK, thank you very much for the effort you've put into this issue. I realize that testing concurrency is inherently difficult, especially if you don't control the test environment.