This repository holds the example code and some notes for a talk I gave at the Simula bi-weekly tools-meetup on Property-based testing and the Hypothesis testing framework on Friday September 28th, 2018.
As the author of Hypothesis points out in an article on this very question, it is hard to give a crisp definition that captures everything associated with the term "property-based testing". The gist of it is to try to test that certain properties of your program holds by automatically (and often randomly) generating example instances where the property should hold. It frees you from the tedium of coming up with (counter)examples in the form of unit tests, and helps you find adversarial examples that break your code.
I recommend reading the aforementioned article if you want to explore the term "property-based testing" more.
Hypothesis is a property-based tesing framework mainly targeting Python but also other platforms. It helps you write property-based tests by giving you lots of options for automatically generating example data of different kinds (even numpy arrays and pandas dataframes!). It integrates nicely with your existing unit testing framework, making it easy to gradually adopt Property-based testing.
Hypothesis is being used by several companies and open source projects. The payment service company Stripe uses it to extensively test their machine learning model training pipeline, for example.
The talk revolved around a few code examples where I showcased different aspects of using Hypothesis.
The first example discussed an example borrowed from Hillel Wayne's excellent talk: A function that finds the maximal product of two elements from a list of integers. I show how quickly Hypothesis can find an example that breaks a "clever" implementation.
The second example discusses the challenge of checking whether CountVectorizer from scikit-learn behaves as expected when tokenizing on chars. In test_encoder.py I check whether CountVectorizer can handle any unicode input. In test_restricted, I check whether CountVectorizer can cope when explicitly asked not to lowercase its inputs and receiving a much more restricted set of unicode letters.
The third example shows a simple way of writing an extended unit test for the autograd differentiation library: We use autograd to find the derivative of np.sin, and then check whether it behaves like np.cos for a bunch of input arrays.
Either do pip install -r requirements.txt or do pip install pytest hypothesis autograd sklearn.
While researching Hypothesis and Property-based testing I benefitted greatly from the following:
- The official Hypothesis documentation
- Alex Chan's talk on Hypothesis and AFL
- Hillel Wayne's talk on going beyind unit tests with property-based testing and contracts
- Scott W. Found's article on Choosing properties for property-based testing