dumper-fuzzy barfs a hairball now and then

[GerHobbelt]

25-dumper-fuzzy.js sporadically barfs a hairball like the one shown below.

Can anyone give a few tips/hints how to track this down without perhaps ripping fast-check open and inspecting the entrails there?

     Property failed after 69 tests
{ seed: 1955227325, path: "68:2:2:2:1:3:2:2:4:3:3:3:3:3:3:4:4:3:3:3:3:4:5:3:10:10:9:8:8:8:8:8:8:8:9:9:9:9:9:9:9:9", endOnFailure: true }
Counterexample: [{"":{"$,":{}}},{"flowLevel":0}]
Shrunk 41 time(s)
Got error: AssertionError [ERR_ASSERTION]: { '': { '$': null, null: {} } } deepStrictEqual { '': { '$,': {} } }

Stack trace: AssertionError [ERR_ASSERTION]: { '': { '$': null, null: {} } } deepStrictEqual { '': { '$,': {} } }
    at W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\test\25-dumper-fuzzy.js:44:16
    at Property.predicate (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\fast-check\lib\check\property\Property.generated.js:34:105)
    at Property.run (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\fast-check\lib\check\property\Property.generic.js:19:31)
    at runIt (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\fast-check\lib\check\runner\Runner.js:84:32)
    at check (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\fast-check\lib\check\runner\Runner.js:175:11)
    at Object.assert (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\fast-check\lib\check\runner\Runner.js:179:15)
    at Context.<anonymous> (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\test\25-dumper-fuzzy.js:38:8)
    at callFn (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\mocha\lib\runnable.js:387:21)
    at Test.Runnable.run (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\mocha\lib\runnable.js:379:7)
    at Runner.runTest (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\mocha\lib\runner.js:535:10)
    at W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\mocha\lib\runner.js:653:12
    at next (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\mocha\lib\runner.js:447:14)
    at W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\mocha\lib\runner.js:457:7
    at next (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\mocha\lib\runner.js:362:14)
    at Immediate.<anonymous> (W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\node_modules\mocha\lib\runner.js:425:5)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)

Hint: Enable verbose mode in order to have the list of all failing values encountered during the run
  Error: Property failed after 69 tests
  { seed: 1955227325, path: "68:2:2:2:1:3:2:2:4:3:3:3:3:3:3:4:4:3:3:3:3:4:5:3:10:10:9:8:8:8:8:8:8:8:9:9:9:9:9:9:9:9", endOnFailure: true }
  Counterexample: [{"":{"$,":{}}},{"flowLevel":0}]
  Shrunk 41 time(s)
  Got error: AssertionError [ERR_ASSERTION]: { '': { '$': null, null: {} } } deepStrictEqual { '': { '$,': {} } }

  Stack trace: AssertionError [ERR_ASSERTION]: { '': { '$': null, null: {} } } deepStrictEqual { '': { '$,': {} } }
      at W:\Users\Ger\Projects\sites\library.visyond.gov\80\lib\js\js-yaml\test\25-dumper-fuzzy.js:44:16
      at Property.predicate (node_modules\fast-check\lib\check\property\Property.generated.js:34:105)
      at Property.run (node_modules\fast-check\lib\check\property\Property.generic.js:19:31)
      at runIt (node_modules\fast-check\lib\check\runner\Runner.js:84:32)
      at check (node_modules\fast-check\lib\check\runner\Runner.js:175:11)
      at Object.assert (node_modules\fast-check\lib\check\runner\Runner.js:179:15)
      at Context.<anonymous> (test\25-dumper-fuzzy.js:38:8)

  Hint: Enable verbose mode in order to have the list of all failing values encountered during the run
      at Object.throwIfFailed (node_modules\fast-check\lib\check\runner\utils\RunDetailsFormatter.js:108:11)
      at Object.assert (node_modules\fast-check\lib\check\runner\Runner.js:183:31)
      at Context.<anonymous> (test\25-dumper-fuzzy.js:38:8)

[puzrin]

https://github.com/dubzzz/fast-check/blob/master/documentation/1-Guides/Tips.md#replay-after-failure is this what you are asking for?

[GerHobbelt]

👍

That should do it for now; I do wonder why the fuzzy test generator only triggers this once in a while, but I guess it's tickling an edge case there, maybe in test generation, maybe in the library itself.
Time to find out...

[puzrin]

I think fuzzy data is pseudo-random, and some rare edge case may not happen every time.

[GerHobbelt]

Indeed, it's a game of change every time, but when I filed this issue, the reported test failure (bug) was only happening very rarely, about once every 20 test runs (make test).

I increased the number of fuzzer runs (GerHobbelt@587b02c) not to have a guarantee but merely to increase the likelyhood that any bug triggered by the fuzzer would get reported on a make test run.

When a weird bug is reported more often it's more likely to be investigated than a fluke that happens rarely -- I know I shouldn't treat error reports like that, but sometimes I was working on something else and then the bug would turn up, but it would quickly scroll off screen and I would have a WTF moment. 😉

Now, at 3000 rounds, I feel more certain that the many edges of js-yaml get tested more thoroughly and will be reported often enough that I will notice them, even when I'm really looking for other things.

Let's say I'm trying to mix the human condition with the technical reality of (pseudo)random-driven test suites like 25-dumper-fuzzy. 😄