The test suite is not deterministic

[mrkkrp]

The test suite is not deterministic and contributes to flakiness of Stackage HEAD project. For example, with the same build plan and the some docker image, sometimes we get this:

> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb Setup
> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ ./Setup configure --enable-tests --package-db=clear --package-db=global --package-db=/home/circleci/project/builds/nightly/pkgdb --libdir=/home/circleci/project/builds/nightly/lib --bindir=/home/circleci/project/builds/nightly/bin --datadir=/home/circleci/project/builds/nightly/share --libexecdir=/home/circleci/project/builds/nightly/libexec --sysconfdir=/home/circleci/project/builds/nightly/etc --docdir=/home/circleci/project/builds/nightly/doc/fsnotify-conduit-0.1.0.0 --htmldir=/home/circleci/project/builds/nightly/doc/fsnotify-conduit-0.1.0.0 --haddockdir=/home/circleci/project/builds/nightly/doc/fsnotify-conduit-0.1.0.0
Configuring fsnotify-conduit-0.1.0.0...
> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb Setup
> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ ./Setup build
Preprocessing library for fsnotify-conduit-0.1.0.0..
Building library for fsnotify-conduit-0.1.0.0..
[1 of 1] Compiling Data.Conduit.FSNotify ( src/Data/Conduit/FSNotify.hs, dist/build/Data/Conduit/FSNotify.o )

src/Data/Conduit/FSNotify.hs:104:22: warning: [-Wdeprecations]
    In the use of type constructor or class ‘Producer’
    (imported from Data.Conduit, but defined in conduit-1.3.0.2:Data.Conduit.Internal.Conduit):
    Deprecated: "Use ConduitT directly"
    |
104 |                   -> Producer m FS.Event
    |                      ^^^^^^^^
Preprocessing test suite 'fsnotify-conduit-test' for fsnotify-conduit-0.1.0.0..
Building test suite 'fsnotify-conduit-test' for fsnotify-conduit-0.1.0.0..
[1 of 2] Compiling Data.Conduit.FSNotifySpec ( test/Data/Conduit/FSNotifySpec.hs, dist/build/fsnotify-conduit-test/fsnotify-conduit-test-tmp/Data/Conduit/FSNotifySpec.o )

test/Data/Conduit/FSNotifySpec.hs:31:39: warning: [-Wdeprecations]
    In the use of ‘$$’
    (imported from Data.Conduit, but defined in conduit-1.3.0.2:Data.Conduit.Internal.Conduit):
    Deprecated: "Use runConduit and .|"
   |
31 |                                       $$ CL.mapM_ (liftIO . writeChan chan)) <|>
   |                                       ^^
[2 of 2] Compiling Main             ( test/Spec.hs, dist/build/fsnotify-conduit-test/fsnotify-conduit-test-tmp/Main.o )
Linking dist/build/fsnotify-conduit-test/fsnotify-conduit-test ...
> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ dist/build/fsnotify-conduit-test/fsnotify-conduit-test

Data.Conduit.FSNotify
  sourceFileChanges

Finished in 0.5296 seconds
1 example, 0 failures

Other time we're less lucky:

> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb Setup
> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ ./Setup configure --enable-tests --package-db=clear --package-db=global --package-db=/home/circleci/project/builds/nightly/pkgdb --libdir=/home/circleci/project/builds/nightly/lib --bindir=/home/circleci/project/builds/nightly/bin --datadir=/home/circleci/project/builds/nightly/share --libexecdir=/home/circleci/project/builds/nightly/libexec --sysconfdir=/home/circleci/project/builds/nightly/etc --docdir=/home/circleci/project/builds/nightly/doc/fsnotify-conduit-0.1.0.0 --htmldir=/home/circleci/project/builds/nightly/doc/fsnotify-conduit-0.1.0.0 --haddockdir=/home/circleci/project/builds/nightly/doc/fsnotify-conduit-0.1.0.0
Configuring fsnotify-conduit-0.1.0.0...
> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb Setup
> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ ./Setup build
Preprocessing library for fsnotify-conduit-0.1.0.0..
Building library for fsnotify-conduit-0.1.0.0..
[1 of 1] Compiling Data.Conduit.FSNotify ( src/Data/Conduit/FSNotify.hs, dist/build/Data/Conduit/FSNotify.o )

src/Data/Conduit/FSNotify.hs:104:22: warning: [-Wdeprecations]
    In the use of type constructor or class ‘Producer’
    (imported from Data.Conduit, but defined in conduit-1.3.0.2:Data.Conduit.Internal.Conduit):
    Deprecated: "Use ConduitT directly"
    |
104 |                   -> Producer m FS.Event
    |                      ^^^^^^^^
Preprocessing test suite 'fsnotify-conduit-test' for fsnotify-conduit-0.1.0.0..
Building test suite 'fsnotify-conduit-test' for fsnotify-conduit-0.1.0.0..
[1 of 2] Compiling Data.Conduit.FSNotifySpec ( test/Data/Conduit/FSNotifySpec.hs, dist/build/fsnotify-conduit-test/fsnotify-conduit-test-tmp/Data/Conduit/FSNotifySpec.o )

test/Data/Conduit/FSNotifySpec.hs:31:39: warning: [-Wdeprecations]
    In the use of ‘$$’
    (imported from Data.Conduit, but defined in conduit-1.3.0.2:Data.Conduit.Internal.Conduit):
    Deprecated: "Use runConduit and .|"
   |
31 |                                       $$ CL.mapM_ (liftIO . writeChan chan)) <|>
   |                                       ^^
[2 of 2] Compiling Main             ( test/Spec.hs, dist/build/fsnotify-conduit-test/fsnotify-conduit-test-tmp/Main.o )
Linking dist/build/fsnotify-conduit-test/fsnotify-conduit-test ...
> /tmp/stackage-build2089/fsnotify-conduit-0.1.0.0$ dist/build/fsnotify-conduit-test/fsnotify-conduit-test

Data.Conduit.FSNotify
  sourceFileChanges FAILED [1]

Failures:

  test/Data/Conduit/FSNotifySpec.hs:39:17: 
  1) Data.Conduit.FSNotify sourceFileChanges
       expected: "bar"
        but got: "foo"

Randomized with seed 1900622263

Finished in 0.1669 seconds
1 example, 1 failure

I'll probably need to open a PR to disable this test suite somehow on Stackage till it's fixed.

[snoyberg]

I've just uploaded version 0.1.1.0, which should provide a more reliable test suite.