Level test is still flaky

[alfert]

The level tests are still flaky when running in Travis. The former attempts to solve this issue (#82, #85, #86) help a bit, but I assume that some ground-breaking work needs to be done.

My basic assumption for today is that the PropEr tutorial is not a good starting point for structuring and setting up the properties, since it uses the old and obsolete approach of forall_sa instead of forall_targeted and exists. We need to investigate into this further and apply learnings from targeted_path_test.exs and targeted_tree_test.exs.

[evnu]

@alfert Can you link a failing travis run?

[alfert]

https://travis-ci.org/alfert/propcheck/builds/490418694 Von unterwegs gesendet

…

Am 08.02.2019 um 09:10 schrieb Magnus ***@***.***>: @alfert Can you link a failing travis run? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[evnu]

For future reference:

  1) property Target PBT Level 1 with forall_targeted and proper-derived nf (PropCheck.Test.LevelTest)
     test/level_tpbt_test.exs:99
     Property Elixir.PropCheck.Test.LevelTest.property Target PBT Level 1 with forall_targeted and proper-derived nf() should fail, but succeeded for all test data :-(
     code: nil
     stacktrace:
       (propcheck) lib/properties.ex:119: PropCheck.Properties.handle_check_results/3
       test/level_tpbt_test.exs:99: (test)

[evnu]

So the question seems to be why it may take more than 50k to find a successful path through the maze, right? It might make sense to also generate a heatmap of paths traveled, similar to the ones presented in the PropEr tutorial.

[alfert]

The heat map is interesting thingy. It is a visualisation of the statistics usually done by collect() and its friends, but which cannot be used by PropEr's TPBT implementation. Also the examples from Fred Hebert's book would be way nicer because such a statistic could replace the printout of all generated data.

At best, such a statistic would be a part of PropEr. If we do it in PropCheck only, than I assume we need to start some GenServer collecting the data and generate a ref for each run. In contrast to regular properties, we would to explicitly send the data to server process. Perhaps we can hide this in the forall_targed etc macros.

[alfert]

Solved with #88