Proof-of-concept for implementation of property-based testing using fast-check

[naripok]

Summary

This issue proposes the implementation of property-based testing and specification using fast-check.
Property-based testing is a type of generative testing (fuzzing) used to assure properties about the behaviour of the implementation's API.
It differs from example-based testing in the sense that, while example-based testing assures that the implementation can handle certain hand-tailored examples without failing, property-based testing assures that the implementation behaviours as desired for (supposedly) any input it could take.

From fast-check documentation:

A property can be summarized by: for any (a, b, ...) such as precondition(a, b, ...) holds, property(a, b, ...) is true.
Property-based testing frameworks try to discover inputs (a, b, ...) causing property(a, b, ...) to be false. If they find such, they have to reduce the counterexample to a minimal counterexample.

Advantages

Generative testing has several advantages over manual test writing, e.g.:

• It reduces the runtime errors occurrences drastically, as it provides a wider path coverage by testing against possible inputs not thought by the developers;
• It favours and stimulates code reusability (where the generators are usually composed by smaller ones and shared);
• It increases development velocity, as thinking about properties is faster than trying to think about enough relevant test inputs;
• It obliges the maintenance of the test suite up-to-date (as it has much better case coverage and will show errors easily on implementations out of spec);
• It serves as a good base for implementation optimization and refactoring of code. If you know you have some code functioning as it should, you can compare implementations throughout the input space assuring the outputs matches;
• It shifts the mentality of the developer from 'trying to come up with cases where the implementation breaks' to 'specifying the desired behaviour of the API';

Here is what It looks like testing one of our methods:

import * as fc from 'fast-check';
import { evaluateCondition } from '../../src/engine/evaluate';
import {
  EngineCondition,
  QueryFilterComparisonType,
  CosineSimilarityFilter,
  PageView,
} from '../../types';

// Implementation of the generators mimicking
// the type definitions.
// This generators would be shared throughout
// the test suite. Write once; use everywhere.

// NumberArray
const vectorGen = fc.array(fc.float(), 128, 128)

// VectorQueryValue
const queryValueGen = fc.record({
  vector: vectorGen,
  threshold: fc.float()
})

// AudienceQueryDefinition
const queryArrayGen = fc.array(
  fc.record({
    featureVersion: fc.integer(),
    queryProperty: fc.constant('topicDist'),
    queryFilterComparisonType:
      fc.constant(QueryFilterComparisonType.COSINE_SIMILARITY),
    queryValue: queryValueGen
  })
)

// EngineCondition<AudienceDefinitionFilter>
const cosineSimilarityConditionGen  =
  fc.record({
  filter: fc.record({
    any: fc.boolean(),
    queries: queryArrayGen,
  }),
  rules: fc.array(
    fc.record({
      reducer: fc.record({
        name: fc.constant('count')
      }),
      matcher: fc.record({
        name: fc.constant('ge'),
        args: fc.constant(1),
      })
    })
  )
})

// PageView[]
const pageViewsGen = fc.array(
  fc.record({
    ts: fc.integer(),
    features: fc.record({
      topicDist: fc.record({
        version: fc.integer(),
        value: vectorGen,
      })
    })
  })
)

describe('Engine evaluateCondition method behaviour', () => {
  fc.assert(
    fc.property(
      cosineSimilarityConditionGen,
      pageViewsGen,
      (
        condition,
        pageViews,
      ) => {

        const result = evaluateCondition(
          condition as EngineCondition<CosineSimilarityFilter>,
          pageViews as PageView[]
        )

        // should always returns a boolean value and not throw errors
        expect(typeof result).toBe('boolean')

        // the below should pass, but in the actual implementation
        // using rules.every, whenever the rule
        // array is empty, the evaluateCondition
        // will return true
        //
        // this is a good case for demonstrating the
        // runner's shrinking capabilities anyways.

        // if there is no pageViews to match, don't match at all
        if (pageViews.length === 0) {
          expect(result).toBe(false)
        }

        // if there is no queries, don't match at all
        if (condition.filter.queries.length === 0) {
          expect(result).toBe(false)
        }

        // TODO specify remaining desired behaviours
      })
  );
});

The implementation of the specs should not interfere with the current testing suite, and it can be gradually adopted.
However, it would, at some point, renders the manual case checking nearly useless, at which point it could be discarded.

If the presented generator syntax does not please, JSON syntax can also be used with the help of json-schema-fast-check.

Disadvantages

• Incurs a shift in thinking pattern and can be problematic for new developers to grok, at first.

References

• fast-check
• fast-check-examples
• CodeSandbox usage demonstration
• Awesome FastCheck