Improve message of failed properties

[sgronblo]

Some context: I have created a small SQL query parser/serializer and created a property that a randomly generated query should be the same as serializing it and then re-parsing it.

However the only information that is shown when a counter example is found is the randomly generated query that was created. I would prefer to be able to provide a more informative such as "The query Q was reparsed as Q'" to make it easier to see exactly why the property failed. This is possible when writing specific test cases using OUnit in the form of a named msg parameter. However currently qcheck properties seem to be simply 'a -> bool.

[Gbury]

I think that's the goal of the QCheck.fail_report and QCheck.fail_reportf functions (cf https://github.com/c-cube/qcheck/blob/master/src/QCheck.mli#L606)

[sgronblo]

I looked these functions up and noticed they were not in the latest opam version of qcheck.

I ran opam pin add qcheck --dev-repo and got access to the functions. However, even if I call fail_report error_message the message is not displayed for a failing property.

[sgronblo]

I think I found what would seem an appropriate place to print the msg_l in qcheck.

https://github.com/c-cube/qcheck/blob/master/src/QCheck.ml#L1370

Here we could concatenate the msg_l strings together and put them inside the strings generated for the counter examples. However I'm not sure if there is a great way to handle longer messages with newlines. Since qcheck is using Format with boxed strings to output the final message here any newlines are going to push the indentation to the first column.

I created a test example file:

open OUnit2

let simple_qcheck =
    QCheck.Test.make
        ~count: 100
        QCheck.small_int
        (fun x -> QCheck.Test.fail_reportf "@[<v>this@ will@ always@ fail@]")

let suite = "all tests" >::: [ QCheck_runner.to_ounit2_test simple_qcheck ]

let () =
    run_test_tt_main suite

Running this example gives the following output:

F
==============================================================================
Error: all tests:0:anon_test_1.

File "/Users/sam/projects/private/qcheck-test/_build/oUnit-all tests-Sams-MacBook.local#01.log", line 2, characters 1-1:
Error: all tests:0:anon_test_1 (in the log).

File "src/QCheck.ml", line 1359, characters 1-1:
Error: all tests:0:anon_test_1 (in the code).

  test `anon_test_1`
  failed on ≥ 1 cases:
  0 this
will
always
fail (after 5 shrink steps)


------------------------------------------------------------------------------
Ran: 1 tests in: 0.14 seconds.
FAILED: Cases: 1 Tried: 1 Errors: 0 Failures: 1 Skip:  0 Todo: 0 Timeouts: 0.

You can see that the error message I provided to fail_reportf was already converted to a string with new lines in it and clashes with the boxed printing in pp_print_test_fail.

I'm too inexperienced with OCaml to figure out if there is a way to improve this. I would like to have a way to print a long message that might include new lines.

[c-cube]

Should be better now. Please re-open if it's not! :-)