Replaying FsCheck test cases may be broken

[jackfoxy]

See the "testexpectofscheck" branch of https://github.com/jackfoxy/personalServer for demonstration code.

The problems have only been tested/verified in debug compile, not release. See the script file \tests\PersonalServer.Tests\Scripts\ArbRegister.fsx to demo the problems in native FsCheck. Note that all reported problems in Expecto/FsCheck can be made to work in ArbRegister.fsx.

tests/PersonalServer.Tests contains the Expecto project.

Demo test list is in DomainTypes.fs.

Problem 1) custom label does not print to console on test failure

Problem 2) failed test succeeds when replay attempted
take reported failure seed and re-run test with configReplay

Problem 3) Arb.register does not work in Expecto
test case returns message, even though Arb.register present:

The type Jackfoxy.PersonalServer.FullName is not handled automatically by FsCheck. Consider using another type or writing and registering a generator for it.

[haf]

You need to feed arbitraries here https://github.com/jackfoxy/PersonalServer/blob/master/tests/PersonalServer.Tests/DomainTypes.fs#L9

We have docs here – on the config type.

[jackfoxy]

Thanks! arbitrary: Type list works perfectly. It was easy to overlook. Perhaps adding a use case example in the documentation would be helpful. Many FsCheck beginners have difficulty with arbitrary registration. (There is a recent Stack Overflow item about this, and even the FsCheck documentation could use some better examples.)

As for custom label displaying, may Expecto.Logging is the solution. I will have to look into that.

Finally, I think Problem 2 above is still an open issue. Failed test succeeds when replay attempted. You can reproduce it in my test branch as documented.

[haf]

@jackfoxy Could you write a few lines in the README about it, please?

[jackfoxy]

Added steps to the README of the "testexpectofscheck" branch, not in "master" branch. More info follows in the readme.

[AnthonyLloyd]

Hi, I thought I had a similar issue so I had a look at your tests (mine turned out to be a real threading bug).

testPropertyWithConfig config10k "this will fail with config10k, then succeed with configReplay" <|
//testPropertyWithConfig configReplay "this will fail with config10k, then succeed with configReplay" <|
    fun  () ->
        Prop.forAll (DomainGenerators.FullName()) 
            (fun (fullName : FullName) -> 
                    
                    let first = fullName.First |> Option.map (fun x -> x.Value)
                    let middle = fullName.Middle |> List.map (fun x -> x.Value)
                    let family = fullName.Family |> Option.map (fun x -> x.Value)

                    let t =  
                        FullName.TryParse (first, middle, family, fullName.NameOrder, Set.empty<Tag>)
                        
                    t.Value = fullName
                )

Here the stdgen is only being used to generate the function param () so isn't really working. The only difference I can see is the config10k is 10000 tests and configReplay is the default 100 tests.

I think you need to do something more like the following:

let configReplay = {FsCheckConfig.defaultConfig with arbitrary = [typeof<DomainGenerators>]; replay = Some <| (1919174669,296290663)}

testPropertyWithConfig configReplay "configReplay" <|
    (fun (fullName:FullName) -> 
            
            let first = fullName.First |> Option.map (fun x -> x.Value)
            let middle = fullName.Middle |> List.map (fun x -> x.Value)
            let family = fullName.Family |> Option.map (fun x -> x.Value)

            let t =  
                FullName.TryParse (first, middle, family, fullName.NameOrder, Set.empty<Tag>)
                
            t.Value = fullName
        )

Anyway in my testing the replay functionality does look to be working correctly and now I just need to fix this threading bug,

[jackfoxy]

@AnthonyLloyd Thanks. I made an incorrect assumption about how FsCheck seeding works. Since my 10000 tests config was failing after fewer than 100 tests, I assumed I could use the same seed in a config that would only run a max of 100 tests. I learned something.
Closing this issue.