Number of QuickCheck test cases is inconsistent

[jtdaugherty]

I'm using tasty version 0.11.0.3 with tasty-quickcheck version 0.8.4. I'm using defaultMain with no command-line arguments. I run my test suite consisting of four quickcheck properties and I see that only one of my properties is tested with more than one test case rather than using the same number of cases for each property:

Markup
  no marking:         OK (0.01s)
    +++ OK, passed 100 tests.
  text preservation:  OK
    +++ OK, passed 1 tests.
  no-op marking:      OK
    +++ OK, passed 1 tests.
  merged marking:     OK
    +++ OK, passed 1 tests.
  containing marking: OK
    +++ OK, passed 1 tests.

If I instead run the test suite binary with --quickcheck-tests 1000 I see this:

Markup
  no marking:         OK (0.08s)
    +++ OK, passed 1000 tests.
  text preservation:  OK
    +++ OK, passed 1 tests.
  no-op marking:      OK
    +++ OK, passed 1 tests.
  merged marking:     OK
    +++ OK, passed 1 tests.
  containing marking: OK
    +++ OK, passed 1 tests.

[jtdaugherty]

I also noticed that this behavior manifests even if I specifically select one of the single-case tests with -p:

$ markup-tests --quickcheck-tests 1000 -p containing
Markup
  containing marking: OK
    +++ OK, passed 1 tests.

[UnkindPartition]

Could you construct a minimal reproducible example?

[UnkindPartition]

(Actually, it doesn't have to be minimal -- if this is an open source project buildable with stack, just point me to it.)

[jtdaugherty]

Yes, the project is public. If you install https://github.com/jtdaugherty/text-markup and then run cabal test, that should be enough. Thanks!

[jtdaugherty]

(And for what it's worth, the install.sh included in that repo will do a sandbox build. It doesn't use Stack.)

[UnkindPartition]

The reason why I mentioned stack is that stack provides a little bit more reproducibility than the alternatives. As I was afraid from the beginning, I am not able to reproduce your results.

% git show -s HEAD 
commit 2684b94671d823f301e57071fdafc3a9ec1322fd
Author: Jonathan Daugherty <cygnus@foobox.com>
Date:   Wed Aug 31 13:54:36 2016 -0700

    Relax base bound to support GHC 8

% grep -v '^#' stack.yaml | grep .
resolver: lts-6.0
packages:
- '.'
extra-deps: []
flags: {}
extra-package-dbs: []

% stack test --test-arguments "--quickcheck-tests 1000"              
text-markup-0.1: test (suite: test-markup-tests, args: --quickcheck-tests 1000)

Markup
  no marking:         OK (0.07s)
    +++ OK, passed 1000 tests.
  text preservation:  OK (0.08s)
    +++ OK, passed 1000 tests.
  no-op marking:      OK (0.08s)
    +++ OK, passed 1000 tests.
  merged marking:     OK (0.07s)
    +++ OK, passed 1000 tests.
  containing marking: FAIL
    *** Failed! Falsifiable (after 12 tests): 
    ("\1964\142\1040\1866\1207\1907\360\412\2036\1011",3,11,10,[("\1964\142\1040",1),("\1866",3),("\1207\1907\360\412\2036\1011",2)],[("\1964\142\1040",1),("\1866\1207\1907\360\412\2036\1011",3)])
    Use --quickcheck-replay '11 TFGenR 000000054F52FFDD00000000000F4240000000000000E121000000006E634700 0 65520 16 0' to reproduce.

1 out of 5 tests failed (0.30s)

Test suite failure for package text-markup-0.1
    test-markup-tests:  exited with: ExitFailure 1
Logs printed to console


% stack test --test-arguments "--quickcheck-tests 1000 -p containing"
text-markup-0.1: test (suite: test-markup-tests, args: --quickcheck-tests 1000 -p containing)

Markup
  containing marking: FAIL
    *** Failed! Falsifiable (after 7 tests): 
    ("\7974\11947\31681\63316",2,4,3,[("\7974\11947",1),("\31681",3),("\63316",2)],[("\7974\11947",1),("\31681\63316",3)])
    Use --quickcheck-replay '6 TFGenR 000000002A8DB9B50000000000007A12000000000000E12100000000A52A1AC0 0 127 7 0' to reproduce.

1 out of 1 tests failed (0.00s)

Test suite failure for package text-markup-0.1
    test-markup-tests:  exited with: ExitFailure 1
Logs printed to console

[jtdaugherty]

I've reproduced the problem with the attached cabal.config.broken.txt, obtained from cabal freeze. I've also attached cabal.config.working.txt for comparison. If you rename either to cabal.config and install, cabal will honor the constraints in the file. It looks like a QuickCheck upgrade might be to blame.

cabal.config.broken.txt
cabal.config.working.txt

[UnkindPartition]

Confirmed. I'm going to report this upstream to QuickCheck devs.

[jtdaugherty]

Wonderful. Thank you! Can you CC me on that report? I'm curious to know what's going on.

[UnkindPartition]

It looks like the bug is fixed in QuickCheck (7b06fa0); with that version, I get:

% stack test --test-arguments='--quickcheck-tests 1000 -p containing'
text-markup-0.1: test (suite: test-markup-tests, args: --quickcheck-tests 1000 -p containing)

Markup
  containing marking: FAIL
    *** Failed! Falsifiable (after 3 tests): 
    ("\53607\25627",0,3,2,[("\53607",3),("\25627",2)],[("\53607\25627",3)])
    Use --quickcheck-replay '2 TFGenR 00000008E6E61B8500000000000F4240000000000000E12B000000009272D840 0 7 3 0' to reproduce.

1 out of 1 tests failed (0.00s)