Issue with generated functions that return boxed exceptions

[paulyoung]

I've reduced my issue down to the following:

let exceptionGenerator : Gen<exn> =
  gen { return! Gen.elements [ Exception(); ] }

let resultGenerator<'t, 'e when 'e :> exn> : Gen<Result<'t, 'e>> =
  gen { let! t = Arb.generate<'t>
        let! e = Arb.generate<'e>
//      return! Gen.elements [ Success t; Failure e ] }
//      return! Gen.elements [ Success t; ] }
        return! Gen.elements [ Failure e ] }

type ResultGenerators =
  static member Result() = Arb.fromGen resultGenerator
  static member exn() = Arb.fromGen exceptionGenerator


[<Arbitrary[|typeof<ResultGenerators>|]>]
module ``flatMap`` =

  [<Property>]
  // let ``obeys the left identity law`` (x: int list, f: int list -> Result<int list, exn>) =
  let ``obeys the left identity law`` (x: int list, F (_: (int list * Result<int list, exn>) list ref, f)) =
    unit x >>= f = f x

with this, I get the following output:

FsCheck.Xunit.PropertyFailedException : Falsifiable, after 1 test (0 shrinks) (StdGen (779977231,296038826)):
Original:
([],
 { []->Failure System.Exception: Exception of type 'System.Exception' was thrown. })

I only have problems with tests that generate functions with the signature int list -> Result<int list, exn> (other tests compare generated Result types containing exceptions without issue).

I'm not quite sure what's happening here but I suspect that the exception is somehow being unintentionally raised.

[kurtschelfthout]

Not exactly, it's just that you're generating two different Exception instances to the left and the right of your property, and they compare false.

Try instead

let exceptionGenerator : Gen<exn> = Gen.constant <| Exception() 
    // or: Gen.elements [ Exception() ]

Ultimately this is due to a very subtle issue in the definition of Gen.delay, in the computation expression for gen. That is, if I redefine it as:

let private delay (f : unit -> Gen<_>) : Gen<_> = f()  

Your original definition works fine (i.e. generates the same instance of Exception every time.

I'm going to sleep on whether the existing behavior is a bug or a feature, currently leaning towards the former, but am a bit cautious since the existing definition of delay is pretty ancient.

[paulyoung]

Awesome! I've been trying to figure this one out for a while.

For reference, I went with this approach since I have a long list of exceptions:

let exceptionGenerator : Gen<exn> =
  Gen.elements [
    ...
    Exception()
    ...
  ]

Thanks for the help. Interested to see what you decide.

[kurtschelfthout]

Sure, you can workaround by binding the different exception instances outside of the computation builder, i.e.

let exceptionGenerator = let es = [ ... Exception() ...] in Gen.elements es

But I'll remove the delay anyway. I can't think of any practical benefits and it seems counter-intuitive (and inefficient).

[kurtschelfthout]

Heads up: I've had to revert my change to delay in 2.0.6 because it was causing problems. It has to be so that each execution of a computation expression generates a new object; for example, think about what happens if you generate a mutable object in the computation expression, then set its value....also, this broke while/for in the computation expression (which is much the same problem, they used delay under the hood), see e.g. #131 . So keep using Gen.elements and keep an eye out for this gotcha.

[paulyoung]

I last visited this before delay was removed, so still using Gen.elements in my codebase.

Thanks for the heads up.