-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Progress feedback to user #33
Comments
This makes sense indeed, although it's on a per-runner basis. I'll probably need to modify a bit the callbacks because the test events would become more fine grained. |
This should not block 0.6, actually. |
I think @Gbury might want to help on this :-) |
Sure, i'll have a look as soon as possible. But from what I remember, it would probably require to add some hooks to the existing test functions. |
Don't worry, 0.6 is not even merged in opam yet, it's not urgent :-) |
Sorry for the delay. I had some other hard deadlines that have stressed me. I just downloaded the upgrade and it is very nice - Thanks! |
I have tried it out both by running individual tests from the top-level and by simply #load'ing the function tester into the top-level. The little timer (updated underway) is very cool.
It still says 2.0 seconds in the last line even though it has now taken something like 10min. This is a side-effect of the function shrinker's behaviour I believe. It seems it has taken 2.0s to shrink 137 steps before we hit this point. As such it is hard to blame on the visualization, unless we want to include some form of scheduler that can pause shrinking, update the timer, and resume shrinking (probably overkill). |
Indeed the current code updates the timer at each successful shrink step, so if at one point, finding the next smaller counter-example takes a lot of time, nothing will be updated for a while. I think it's a reasonable compromise: it is possible to add another hook to update the timer for each potential counter-example tried, but maybe that might be a bit too much ? Also, that could degrade performances, though I haven't tried it yet. @c-cube do you think it might be worth adding an event for each value tried during shrinking ? Finally, note that this lack of update can also happen at almost every step (though it might be less frequent); I personally have some code where testing the property actually takes a lot of time. With the current code, there's not much that can be done about that since most time is spent in the user's code. |
Such an event (at each shrink attempt) might be possible, yeah, as long as we don't change the |
That was my idea, I'll try that and see what happens. |
Ok, so I just implemented the new event, and happened upon a seed for which single shrink steps of |
If I
With this seed Finally: Printing both numbers would be welcome from here. |
During shrinking is printed |
This is wonderful, thanks!
|
Just tried the About the newlines printed: the only relevant change I can think of is that in verbose mode, when printing the definitive results of a test, we first clear the line using a string like Lastly, I agree that the two names for the shrinking events are not ideal currently, @c-cube do you agree on |
I don't really have an idea on that, sorry. "shrunk" indeed sounds correct. @jmid if your terminal is not wide enough, it might print newlines… |
Renaming done and pushed. |
I cannot recreate the issue with single tests such as
However it happens consistently when I
The thing is I didn't experience this behaviour with the previous version. Perhaps the many events are stress testing my terminal unintentionally. I tried running it under tcsh instead of bash with the same behaviour. I also tried invoking the ocaml top-level directly, without my usual ledit-prefixing bash alias
In all cases I still get the same behaviour when I |
As @c-cube said, the problem might be terminal width. Indeed the new output need more width, since we print more information, also there is then a need to clear a line of these information before printing the definitive line of a test, so if any of these lines exceed the width of your terminal, I don't really know what happens, but if lines are wrapped, then they may not be cleared by a |
@c-cube is right indeed. My width was 110 characters. I tried to bump it up (it is only an 13" laptop) so I had to reduce the font size. I can now recreate the behaviour up to width 139. As such, the patch requires console width 140 or more. We may consider whether this is a reasonable demand. (insert grandpa-story about how terminals were 80 characters wide when I was young) :-) |
I put 140 space in the clearing line simply because with the last commit, 100 was not enonugh anymore (because of the additional numbers printed). Actually, this may be a symptom of a greater problem, namely the unknown length of the test name (indeed, for tests with a sufficiently long name, the clearing line would not be enough anymore). The question then becomes: do we try and implement a smart function to adjust printing based on the terminal width, so that we correctly handle test name of any length and terminal of any width, (but that would probably require some work and adding a dependency on a terminal lib, which might not be very practical), or do we simply try and adjust the current printer to work well under reasonable conditions ? |
First of all: I'm very happy with this new shrinking feedback. From this alone we now have an understanding of why shrinking functions may take long (many failed shrinking attempts). So many thanks again for this! I thought a bit more about it. I believe we can use the column width more wisely. Here I am influenced by Edward Tufte who proposed to increase the data-ink ratio ("spend less ink showing more") in tables, charts, etc. Looking at, e.g.,
there are several opportunities for removing needless characters (parens, semicolons, dashes, ...) to increase the "data-character ratio". Here's the outcome of an attempt at doing so:
We could even go further and remove the square brackets around the colored unicode character.
|
CHANGES: ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files
CHANGES: ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files
CHANGES: ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files
…0.14) CHANGES: ## 0.14 - modify `int_range` to make it accept ranges bigger than `max_int`. - less newline-verbose stats - add `int{32,64}` shrinkers to arbitrary gens - add `int{32,int64}` shrinkers - move to ounit2 for `QCheck_ounit` ## 0.13 - make counter private - Add debug shrinking log - fix: small fix related to stdlib/pervasives - feat: add flatten combinators in `gen` ## 0.12 - fix singleton list shrinking - feat: add `Gen.char_range` and `Gen.(<$>)` (credit @spewspews) ## 0.11 - Add `QCheck.Gen.{string_of,string_readable}` - fix `int_bound` bound inclusiveness problem - change implementation of `int_bound` to generate values using `Random.State.int` for `bound < 2^30` - add weighted shuffled lists generator - add `float_range` to generate a floating-point number in the given range (inclusive) - add `float_bound_inclusive` and `float_bound_exclusive` to generate floating-point numbers between 0 and a given bound ## 0.10 - `Shrink`: decompose Shrink.list into Shrink.list_spine and Shrink.list_elems - `Gen.fix` has a more general and useful type - update README to include `Rely` section (qcheck now available for reason-native!) - Fix stat printing - speed-up list shrinker - Better int shrinking - core: modify proba distributions again, add `big_nat` - feat: add `small_array`, modify distributions - print number of warnings in runner's summary - refactor: modify type of results to make them more accurate - feat: warn/fail if too many tests passed only b/c precondition failed ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files
…0.14) CHANGES: ## 0.14 - modify `int_range` to make it accept ranges bigger than `max_int`. - less newline-verbose stats - add `int{32,64}` shrinkers to arbitrary gens - add `int{32,int64}` shrinkers - move to ounit2 for `QCheck_ounit` ## 0.13 - make counter private - Add debug shrinking log - fix: small fix related to stdlib/pervasives - feat: add flatten combinators in `gen` ## 0.12 - fix singleton list shrinking - feat: add `Gen.char_range` and `Gen.(<$>)` (credit @spewspews) ## 0.11 - Add `QCheck.Gen.{string_of,string_readable}` - fix `int_bound` bound inclusiveness problem - change implementation of `int_bound` to generate values using `Random.State.int` for `bound < 2^30` - add weighted shuffled lists generator - add `float_range` to generate a floating-point number in the given range (inclusive) - add `float_bound_inclusive` and `float_bound_exclusive` to generate floating-point numbers between 0 and a given bound ## 0.10 - `Shrink`: decompose Shrink.list into Shrink.list_spine and Shrink.list_elems - `Gen.fix` has a more general and useful type - update README to include `Rely` section (qcheck now available for reason-native!) - Fix stat printing - speed-up list shrinker - Better int shrinking - core: modify proba distributions again, add `big_nat` - feat: add `small_array`, modify distributions - print number of warnings in runner's summary - refactor: modify type of results to make them more accurate - feat: warn/fail if too many tests passed only b/c precondition failed ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files
…0.14) CHANGES: ## 0.14 - modify `int_range` to make it accept ranges bigger than `max_int`. - less newline-verbose stats - add `int{32,64}` shrinkers to arbitrary gens - add `int{32,int64}` shrinkers - move to ounit2 for `QCheck_ounit` ## 0.13 - make counter private - Add debug shrinking log - fix: small fix related to stdlib/pervasives - feat: add flatten combinators in `gen` ## 0.12 - fix singleton list shrinking - feat: add `Gen.char_range` and `Gen.(<$>)` (credit @spewspews) ## 0.11 - Add `QCheck.Gen.{string_of,string_readable}` - fix `int_bound` bound inclusiveness problem - change implementation of `int_bound` to generate values using `Random.State.int` for `bound < 2^30` - add weighted shuffled lists generator - add `float_range` to generate a floating-point number in the given range (inclusive) - add `float_bound_inclusive` and `float_bound_exclusive` to generate floating-point numbers between 0 and a given bound ## 0.10 - `Shrink`: decompose Shrink.list into Shrink.list_spine and Shrink.list_elems - `Gen.fix` has a more general and useful type - update README to include `Rely` section (qcheck now available for reason-native!) - Fix stat printing - speed-up list shrinker - Better int shrinking - core: modify proba distributions again, add `big_nat` - feat: add `small_array`, modify distributions - print number of warnings in runner's summary - refactor: modify type of results to make them more accurate - feat: warn/fail if too many tests passed only b/c precondition failed ## 0.9 - add `qcheck-ounit` sublibrary - use environment variables to configure `qcheck-alcotest` tests - alcotest backend for qcheck - make `qcheck.ounit` tests verbose by default - make `qcheck` is a compatibility library, depends on `qcheck-core` - split lib into `qcheck` and `qcheck.ounit` - add `TestResult.is_success` helper - give access to list of instances in test results - allow setting `time_between_msg` in runner - chore: remove submodule - chore: add travis support - doc: explanations about qcheck.ounit runners - doc: update readme ## 0.8 - migrate to jbuilder - fix warnings - add some useful functions - update oasis files (close c-cube/qcheck#48) - update copyright header (closes c-cube/qcheck#47) ## 0.7 - switch to BSD license, make it more explicit (close c-cube/qcheck#43) - improve multi-line message printing in ounit (closes c-cube/qcheck#46) - fix complexity of `add_stat` - allow negative entries in statistics (see c-cube/qcheck#40) - add a way for tests to report messages to the user (see c-cube/qcheck#39) - add `QCheck.Shrink.int_aggressive` and make default int shrinker faster - shrinker for `map_keep_input` - add `QCheck.set_gen`, missing for some reason - more compact verbose output (see c-cube/qcheck#33) - better handling of dynamic progress line - Add colors to checkmarks in verbose mode - improve statistics display for runner - recover exception of shrunk input - print status line before the solving starts ## 0.6 - add `find_example` and `find_example_gen` to synthesize values from properties (see c-cube/qcheck#31) - add `QCheck.gen` for accessing the random generator easily - colorful runners, with `--no-colors` to disable them - add more generator (for corner cases) - better generation of random functions (see c-cube/qcheck#8), using `Observable` and an efficient internal representation using heterogeneous tuples, printing, and shrinking. deprecate old hacks. - add statistics gathering and display (see c-cube/qcheck#30) - better printing of Tuple - improve `Shrink.{array,list}` (see c-cube/qcheck#32) - Change asserts to raise `Invalid_arg` (following the doc), and update doc - Change `Gen.{int_bount,int_range}` to support up to 2^62 ## 0.5.3.1 - fix regression in runner output (print results of `collect`) - update the `@since` tags ## 0.5.3 - missing char in `Gen.char` (close c-cube/qcheck#23) - add `test` and `doc` to opam - add `small_list` generator - add `~long_factor` to tests and runner, for long tests - add more examples in readme, better doc for runners - improved reporting when running qcheck tests - add `Test.get_count` on test cells ## 0.5.2 - Add cli option for backtraces in `QCheck_runner` - Add test case for raising exception - Better handling of backtraces - All tests now have a name - Add step function called on each instance in a test - make `small_int` a deprecated alias to `small_nat` - add `small_signed_int` - remove some warnings - use safe-string, and fix related bug - Add long tests options to `QCheck_runner` - Add `length` specification for `to_ounit2_test` - Added paragraph in README about long tests ## 0.5.1 - document exceptions - add `small_nat`, change `small_int` semantics (close c-cube/qcheck#10) - add `QCheck.assume_fail` - add `QCheck.assume`; explain preconditions a bit (close c-cube/qcheck#9) - Polish documentation - Added quad support uniformly ## 0.5 - merge back from `qtest`: big changes in API, shrinking, use `'a arbitrary` type that combines printer, generator, shrinker, etc. (see git log) - merlin file - reorganize sources, `_oasis`, `.merlin`, etc. ## 0.4 - bugfix in `fix_fuel` - if verbose enabled, print each test case - add `QCheck.run_main` - `QCheck_ounit.~::` - add `(>:::)` - add `qcheck_ounit ml{lib,dylib}` - trivial ounit integration - make `test_cell.name` optional - `Arbitrary.fix_fuel(_gen)`: add a recursive case - `Arbitrary.fix_fuel_gen`, similar to `fix_fuel` but threading a state bottom-up to make choices depend on the path - `Arbitrary.fail_fix` to fail in a fixpoint - helper cases for `Arbitrary.fix_fuel` ## 0.3 - get rid of submodule `generator` - `Arbitrary.fix_fuel`, to generate complex recursive structures - new combinators (infix map, applicative funs, shuffle) - remove generator/Generator, and a deprecation warning - output of printers of lists/arrays now parsable by ocaml toplevel ## 0.2 - integrate Gabriel Scherer's `Generator` into `QCheck` - add `|||` - add `Prop.raises` - print the faulty instance in case of error (if a printer is available) - some combinators for `QCheck.Arbitrary` - `QCheck.mk_test` takes more arguments ## 0.1 - oasis based build system - source files
Currently, an interactive counter in the UI counts up the number of generated inputs. This works beautifully for succeeding tests and fast shrinkers.
For occasional slow tests (complex properties and exponential(?) shrinkers) the UI provides less feedback. For example, for the
check-fun
tests when they hang onfold_left fold_right
I just get this:Because of the order of the tests I know that
fold_left fold_right
is next, and hence that it must be causing it, but from the UI alone I do not knowPerhaps the first can simply be addressed by a suitable
flush
of the output channel.For the second just printing
Found counterexample, shrinking
(or something like) would be nice to get a feel for the framework's progress. A fancier version could even display some progress measure (or bar) of the shrinking phase.The text was updated successfully, but these errors were encountered: