Feature wish: statistics over classifiers #30

jmid · 2017-05-09T10:19:17Z

To observe the distribution of generators (a probability distribution) it would be great to have a bit of statistics. For example, for a term generator I'm currently hacking I can map each term to its size, which gives rise to the following output:

random seed: 252524381
law anon_test_1: 20 relevant cases (20 total)
  10: 1 cases
  7: 2 cases
  94: 1 cases
  117: 1 cases
  4: 1 cases
  3: 3 cases
  114: 1 cases
  42: 1 cases
  208: 1 cases
  1: 3 cases
  24: 1 cases
  37: 1 cases
  14: 3 cases
success (ran 1 tests)

(where I limit the tests to 20 since much more overflows me with information).
With a bit of shell script hacking I can strip the output and pass it, e.g., to ministat:

x stat_numbers.txt
+------------------------------------------------------------------------------------------------------------+
|             x                                                                                              |
|             x                                                                                              |
|             x                                                                                              |
|             x                                                                                              |
|          x xx                                                                                              |
|          x xx                                                                                              |
|          xxxx                                                                                              |
|          xxxx  x                                                                                           |
|          xxxx  x                                                                 x                         |
|          xxxxxxx              x                                                  x                         |
|          xxxxxxx     x x    x x   x  x                   x                       x                        x|
||____________M________A_____________________|                                                               |
+------------------------------------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
 x  50             1           335           9.5         40.76     75.589266

(where I bumped up the count to 50 to get a prettier histogram).

This gives a better idea of the distribution (min,max,avg,...) and even a histogram.
I'm not suggestion we add an external dependency on ministat but just some numbers, e.g.,
like Jesper's custom Erlang output, see slide 8 (bottom) of http://janmidtgaard.dk/quickcheck/slides/jlouis-dtu-2017.pdf would be a very nice addition. For starters, this would probably involve making a collect-variant which is integer valued.

The text was updated successfully, but these errors were encountered:

c-cube · 2017-05-09T11:47:27Z

This is interesting, and definitely useful. Probably an argument ?stats : (string * ('a -> int)) list added to arbitrary, with the name of the statistics and the counting function would be best. Each such statistics would be collected and displayed individually (based on its name).

If I implement this, would you be willing to test it?

A few comments:

computing basic statistics (average, median, etc.) should be simple enough to be integrated directly without any dep
maybe the stats could be grouped by buckets, for the histogram
a horizontal histogram, as found somewhere in the stdlib (I think) is easy to generate. See below.

Histogram printing

The snippet below displays the bucket statistics of a hashtable (using containers and sequence):

#use "topfind";;
#require "sequence";;
#require "containers";;

let tbl = CCHashtbl.of_seq Sequence.(1 -- 10000 |> zip_i |> zip);;
let hist = Hashtbl.((stats tbl).bucket_histogram);;

let max = Sequence.of_array hist |> Sequence.max |> CCOpt.get_or ~default:0

let () =
  Array.iteri
    (fun i n ->
       let m = n * 20 / max in
       Printf.printf "%-5d: %-22s %10d\n" i (CCString.repeat "*" m) n)
    hist
;;

$ ocaml truc.ml
0    : ****************             2449
1    : ********************         2936
2    : ***********                  1751
3    : *****                         736
4    : *                             255
5    :                                56
6    :                                 9

jmid · 2017-05-09T12:24:30Z

Yes, I would be happy to test it.
Also: the horizontal histogram looks great considering the limited amount of code it takes to generate it!
👍

c-cube · 2017-05-09T13:57:02Z

Here we go, with an output as follows. Can you take a look?

Collect results for test with_stats:

stats mod4:
         0: ###########                        16
         1: #########################          36
         2: ################                   24
stats num:
       0-6: #########################           9
       1-7: ######################              8
       2-8: #############                       5
       3-9: ################                    6
      4-10: #############                       5
      5-11: ################                    6
      6-12: ################                    6
      7-13: ################                    6
      8-14: #############                       5
      9-15: ################                    6
     10-16: ###########                         4
     11-17: #############                       5
     12-18: ################                    6
     13-19: ################                    6
     14-20: ###########                         4
     15-21: #############                       5
     16-22: ################                    6
     17-23: ################                    6
     18-24: ###################                 7
     19-25: ################                    6

c-cube · 2017-05-09T13:58:11Z

Wait, there are a few bugs to fix.

c-cube · 2017-05-09T14:05:46Z

Now looks like this:

stats mod4:
  num: 100, avg: 1.56
         0: #####################              24
         1: #####################              24
         2: #####################              24
         3: #########################          28
stats num:
  num: 100, avg: 62.00
       0-5: ########                            3
      6-11: #############                       5
     12-17: ########                            3
     18-23: ###########                         4
     24-29: #############                       5
     30-35: #############                       5
     36-41: ######################              8
     42-47: ###################                 7
     48-53: ##                                  1
     54-59: ######################              8
     60-65: #############                       5
     66-71: ########                            3
     72-77: ######################              8
     78-83: ################                    6
     84-89: ################                    6
     90-95: ##                                  1
    96-101: ########                            3
   102-107: #############                       5
   108-113: ###########                         4
   114-119: #########################           9
   120-125: ##                                  1
================================================================================

jmid · 2017-05-10T09:58:38Z

Ok. Looking at this now. I would prefer a consistent naming:

~print and set_print
~shrink and set_shrink
~collect and set_collect
These are all very consistent. In this light I expect ~stats and set_stats but was surprised to find that the latter is called add_stat: add instead of set and a missing s.

c-cube · 2017-05-10T10:00:24Z

It's because in this case I allow for several statistics to be collected. I agree that it's not perfect though :/

jmid · 2017-05-10T10:01:49Z

Ah, I see now it is singular. A set_stats binding would be welcome though for consistency.
Perhaps two bindings: set_stat and set_stats?

c-cube · 2017-05-10T10:07:09Z

set_stat does not make sense, but set_stats can be done.

jmid · 2017-05-10T10:14:49Z

Ah, it has the semantics of cons'ing a stat-entry. Sorry, in that case I agree. set_stats would still make sense though (with the semantics of overwriting with the given list).

jmid · 2017-05-10T10:22:45Z

I just tried:

# let t = Test.make (add_stat ("len",List.length) (list int)) (fun _ -> true);;
val t : QCheck.Test.t = QCheck.Test.Test <abstr>
# QCheck_runner.run_tests ~verbose:true [t];;
random seed: 245450799
generated; error;  fail; pass / total -     time -- test name
[✓] ( 100)    0 ;    0 ;  100 /  100 --     0.0s -- anon_test_1

+++ Collect ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Collect results for test anon_test_1:

stats len:
  num: 100, avg: 372.07
     0-393: #######################################################          83
   394-787: ####                                                              7
  788-1181: ###                                                               5
  1182-1575:                                                                   0
  1576-1969:                                                                   1
  1970-2363:                                                                   0
  2364-2757:                                                                   0
  2758-3151:                                                                   0
  3152-3545:                                                                   0
  3546-3939:                                                                   0
  3940-4333:                                                                   0
  4334-4727:                                                                   0
  4728-5121:                                                                   1
  5122-5515:                                                                   0
  5516-5909:                                                                   0
  5910-6303: #                                                                 2
  6304-6697:                                                                   0
  6698-7091:                                                                   0
  7092-7485:                                                                   0
  7486-7879:                                                                   1
  7880-8273:                                                                   0
================================================================================
success (ran 1 tests)
- : int = 0

It works very nicely! A few observations:

I was first surprised by the many empty lines printed and only subsequently realized that there were actually scarse upper entries
The indentation of the 100's is off by one. I just tried with ~count:10000 and got a similar indentation mismatch at the last entry 10000-10499:
min, max, median, stddev, would be warmly welcome :-)

c-cube · 2017-05-10T11:06:32Z

I'm not used to statistics on day-to-day programming; would you be interested in writing median and stddev?

c-cube · 2017-05-10T11:11:18Z

About the indentation, it's a problem of limiting width, I believe. I'll increase the limit.

jmid · 2017-05-10T11:42:56Z

I can have a look at median and stddev at some point, but I have too much on the TODO list right now. I already promised you some feedback for the new function generators and the statistics came up in preparing the release of our effect-driven program generator. Your contributions and QCheck is highly appreciated! I'm sorry my own contributions are limited to complaints and feature requests at this point :-)

c-cube · 2017-05-10T11:44:04Z

Ok, no problem.

c-cube · 2017-05-10T11:53:26Z

Should be fixed in last commit.

c-cube · 2017-05-10T11:53:54Z

stats mod4:
  num: 100, avg: 1.52, stddev: 11.44, median 1, min 0, max 3
                0: #################################################                25
                1: ###################################################              26
                2: #########################################                        21
                3: #######################################################          28
stats num:
  num: 100, avg: 57.68, stddev: 353.53, median 55, min 0, max 120
              0-5: ######################                                            4
             6-11: ######################                                            4
            12-17: #######################################################          10
            18-23: ######################                                            4
            24-29: ######################################                            7
            30-35: ###########################                                       5
            36-41: ###########################                                       5
            42-47: ################                                                  3
            48-53: #################################                                 6
            54-59: ######################################                            7
            60-65: #################################                                 6
            66-71: ######################                                            4
            72-77: ################                                                  3
            78-83: ###########                                                       2
            84-89: #####                                                             1
            90-95: ######################################                            7
           96-101: ######################################                            7
          102-107: ###########################                                       5
          108-113: #################################                                 6
          114-119: ################                                                  3
          120-125: #####                                                             1

c-cube · 2017-05-10T11:54:35Z

Not sure about stddev though…

c-cube · 2017-05-10T12:04:29Z

@jmid can you elaborate on your program generator? Sounds interesting! :-)

jmid · 2017-05-10T12:19:07Z

This is great - you are the man! For the standard deviation, I think you are just missing a division in line 1273 by num before the square root. Following the Wikipedia link in the code I found
https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation and
https://en.wikipedia.org/wiki/Bessel%27s_correction
which suggests a division by num - 1 instead (but I'm not a statistician either).

jmid · 2017-05-10T12:22:57Z

Re. program generator: I'll send you an email

CHANGES: ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files

@spewspews

…0.14) CHANGES: ## 0.14 - modify `int_range` to make it accept ranges bigger than `max_int`. - less newline-verbose stats - add `int{32,64}` shrinkers to arbitrary gens - add `int{32,int64}` shrinkers - move to ounit2 for `QCheck_ounit` ## 0.13 - make counter private - Add debug shrinking log - fix: small fix related to stdlib/pervasives - feat: add flatten combinators in `gen` ## 0.12 - fix singleton list shrinking - feat: add `Gen.char_range` and `Gen.(<$>)` (credit @spewspews) ## 0.11 - Add `QCheck.Gen.{string_of,string_readable}` - fix `int_bound` bound inclusiveness problem - change implementation of `int_bound` to generate values using `Random.State.int` for `bound < 2^30` - add weighted shuffled lists generator - add `float_range` to generate a floating-point number in the given range (inclusive) - add `float_bound_inclusive` and `float_bound_exclusive` to generate floating-point numbers between 0 and a given bound ## 0.10 - `Shrink`: decompose Shrink.list into Shrink.list_spine and Shrink.list_elems - `Gen.fix` has a more general and useful type - update README to include `Rely` section (qcheck now available for reason-native!) - Fix stat printing - speed-up list shrinker - Better int shrinking - core: modify proba distributions again, add `big_nat` - feat: add `small_array`, modify distributions - print number of warnings in runner's summary - refactor: modify type of results to make them more accurate - feat: warn/fail if too many tests passed only b/c precondition failed ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files

@spewspews

…0.14) CHANGES: ## 0.14 - modify `int_range` to make it accept ranges bigger than `max_int`. - less newline-verbose stats - add `int{32,64}` shrinkers to arbitrary gens - add `int{32,int64}` shrinkers - move to ounit2 for `QCheck_ounit` ## 0.13 - make counter private - Add debug shrinking log - fix: small fix related to stdlib/pervasives - feat: add flatten combinators in `gen` ## 0.12 - fix singleton list shrinking - feat: add `Gen.char_range` and `Gen.(<$>)` (credit @spewspews) ## 0.11 - Add `QCheck.Gen.{string_of,string_readable}` - fix `int_bound` bound inclusiveness problem - change implementation of `int_bound` to generate values using `Random.State.int` for `bound < 2^30` - add weighted shuffled lists generator - add `float_range` to generate a floating-point number in the given range (inclusive) - add `float_bound_inclusive` and `float_bound_exclusive` to generate floating-point numbers between 0 and a given bound ## 0.10 - `Shrink`: decompose Shrink.list into Shrink.list_spine and Shrink.list_elems - `Gen.fix` has a more general and useful type - update README to include `Rely` section (qcheck now available for reason-native!) - Fix stat printing - speed-up list shrinker - Better int shrinking - core: modify proba distributions again, add `big_nat` - feat: add `small_array`, modify distributions - print number of warnings in runner's summary - refactor: modify type of results to make them more accurate - feat: warn/fail if too many tests passed only b/c precondition failed ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files

@spewspews

…0.14) CHANGES: ## 0.14 - modify `int_range` to make it accept ranges bigger than `max_int`. - less newline-verbose stats - add `int{32,64}` shrinkers to arbitrary gens - add `int{32,int64}` shrinkers - move to ounit2 for `QCheck_ounit` ## 0.13 - make counter private - Add debug shrinking log - fix: small fix related to stdlib/pervasives - feat: add flatten combinators in `gen` ## 0.12 - fix singleton list shrinking - feat: add `Gen.char_range` and `Gen.(<$>)` (credit @spewspews) ## 0.11 - Add `QCheck.Gen.{string_of,string_readable}` - fix `int_bound` bound inclusiveness problem - change implementation of `int_bound` to generate values using `Random.State.int` for `bound < 2^30` - add weighted shuffled lists generator - add `float_range` to generate a floating-point number in the given range (inclusive) - add `float_bound_inclusive` and `float_bound_exclusive` to generate floating-point numbers between 0 and a given bound ## 0.10 - `Shrink`: decompose Shrink.list into Shrink.list_spine and Shrink.list_elems - `Gen.fix` has a more general and useful type - update README to include `Rely` section (qcheck now available for reason-native!) - Fix stat printing - speed-up list shrinker - Better int shrinking - core: modify proba distributions again, add `big_nat` - feat: add `small_array`, modify distributions - print number of warnings in runner's summary - refactor: modify type of results to make them more accurate - feat: warn/fail if too many tests passed only b/c precondition failed ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files

c-cube self-assigned this May 9, 2017

c-cube added the enhancement label May 9, 2017

c-cube closed this as completed in 4e4773c May 9, 2017

c-cube added a commit that referenced this issue May 10, 2017

compute more statistics (#30)

2dd5208

c-cube mentioned this issue Sep 18, 2018

[new release] qcheck (52e4193) ocaml/opam-repository#12644

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature wish: statistics over classifiers #30

Feature wish: statistics over classifiers #30

jmid commented May 9, 2017

c-cube commented May 9, 2017

jmid commented May 9, 2017

c-cube commented May 9, 2017

c-cube commented May 9, 2017

c-cube commented May 9, 2017

jmid commented May 10, 2017

c-cube commented May 10, 2017

jmid commented May 10, 2017

c-cube commented May 10, 2017

jmid commented May 10, 2017

jmid commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

jmid commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

jmid commented May 10, 2017

jmid commented May 10, 2017

Feature wish: statistics over classifiers #30

Feature wish: statistics over classifiers #30

Comments

jmid commented May 9, 2017

c-cube commented May 9, 2017

Histogram printing

jmid commented May 9, 2017

c-cube commented May 9, 2017

c-cube commented May 9, 2017

c-cube commented May 9, 2017

jmid commented May 10, 2017

c-cube commented May 10, 2017

jmid commented May 10, 2017

c-cube commented May 10, 2017

jmid commented May 10, 2017

jmid commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

jmid commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

c-cube commented May 10, 2017

jmid commented May 10, 2017

jmid commented May 10, 2017