Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing/quick: add support for shrinking #9282

Open
ghost opened this issue Dec 12, 2014 · 5 comments

Comments

@ghost
Copy link

commented Dec 12, 2014

The testing/quick package automatically generates test cases to be checked against a property. However, it often generates complicated values which obscure the source of test failures. Shrinking is a feature from the original QuickCheck which eases this burden by "simplifying" a failed test case as much as possible.

Evan Shaw gave a presentation on testing/quick at Gopher Summerfest, and started work on adding shrinking to the package in the stdlib. I could't find any official activity related to it, though.

I think he got a good start on the problem, but I also think the interface definition for Shrinker would be significantly simpler like this:

// A Shrinker can produce simpler values with its same type.
type Shrinker interface {
    // Shrink returns a closure which successively yields simpler values of its type.
    Shrink() func() reflect.Value
}

Also, because testing/quick generates random values for its test cases, I believe the behavior of quick.Check() can be changed to support shrinking without breaking the Go compatibility guidelines.

I can work on this feature if the Go team decides it's a good idea.

@josharian

This comment has been minimized.

Copy link
Contributor

commented Dec 12, 2014

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc removed release-none labels Apr 10, 2015

@spacejam

This comment has been minimized.

Copy link

commented Dec 13, 2017

testing/quick stands to benefit greatly from this functionality. As it stands, you can't use it for automatic generation of regression tests, as other quickcheck implementations allow for with shrinking. Without shrinking, it is so much more cumbersome to figure out what actually caused an issue.

Many people understand quickcheck's utility for testing things like decoder(encoder(val)) == val and for this, testing/quick is adequate, but when shrinking enters the equation, the testing library becomes far more powerful, and capable of testing complex interactions. You can do things like generate a slice of events/operations/requests to apply to a system, and if the property fails to hold, the shrinker can:

  1. drop out individual elements of the slice
  2. re-run the property test on the shrunken input
  3. if the shrunken slice still fails, GOTO 1
  4. if dropping more elements fails to cause the property to be violated, we have probably just found a concise, more easily debuggable sequence of events/operations/requests that will trip up your system.

I really like using other quickcheck implementations to generate a sequence of external requests and network connectivity issues that then get fed into a simulated distributed system with simulated network + clocks. When the generated input fails, the shrinker reduces it dramatically, and the (potentially quite complex) issue becomes far more understandable.

Further, shrinkers automatically generate regression tests for you! Turn the failing non-deterministic sequence found by quickcheck into a regression test for deterministically watching for similar failures in the future.

Does anyone NOT think that would make a great gophercon talk? :P

@spacejam

This comment has been minimized.

Copy link

commented Dec 13, 2017

I'd like to volunteer to build this, since it's been a few years since (deleted OP) proposed and volunteered.

Seriously, this is so useful that sometimes I get away with only writing a single test for a low-impact interactive system, and after shrinking the testing library just tells me what I need to fix with reasonable thoroughness and clarity.

What does the team think?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Dec 13, 2017

Thanks for looking into this.

What does the team think?

I think I would like to see a specific API proposal.

@spacejam

This comment has been minimized.

Copy link

commented Dec 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.