Factory Audit

Checking how various object factories perform in Django with respect to creating valid instances out of the box.

See the full write-up (with pretty diagram).

Definitions

• Factory libraries

  The following factory libraries have been explored:

  • Django Fakery
  • Factory Boy
  • Factory Djoy
  • Hypothesis[django]
  • Mixer
  • Model Mommy

  Disclosure: Factory Djoy is my factory library. It's a thin wrapper around Factory Boy which does the hard work.

• Models

  Two factories have been created with each factory library:

  • ItemFactory: to create and save instances of plant.models.Item, a test model defined in the 'plant' app.
  • UserFactory: to create and save instances of the default django.contrib.auth User Model.

• Grading

  Each factory is graded based on how its default configuration behaves.

  The gradings are based on the definition of "valid". Valid instances are ones which will pass Django's full_clean and not raise a ValidationError. For example, using the ItemFactory a generated item passes validation with:

  item = ItemFactory()
  item.full_clean()

  The gradings are:

  • 🔴 RED - Factory creates invalid instances of the model and saves them to the database.
  • 💛 YELLOW - Factory raises an exception and does not save any instances. Preferably this would be a ValidationError, but I've also allowed IntegrityError here.
  • 💚 GREEN - Factory creates multiple valid instances with no invalid instances created or skipped. Running factory n times generates n valid instances.

Results

Library	ItemFactory	UserFactory
Django Fakery	🔴 RED	💛 YELLOW
Factory Boy	🔴 RED	🔴 RED
Factory Djoy	💛 YELLOW	💚 GREEN
Hypothesis[django]	🔴 RED	🔴 RED
Mixer	💚 GREEN	💚 GREEN
Model Mommy	💛 YELLOW	💚 GREEN

For more detailed notes about each factory including some of the grey areas around grading please see Notes about each library in the full write-up.

Development

To install and run:

$ git clone git@github.com:jamescooke/factory_audit.git
$ cd factory_audit
$ make venv
$ . venv/bin/activate
$ make install
$ cd factory_audit/
$ make test

Contributions

Please add your own factory library and run it through the tests - pull requests very welcome.

Please help me write a better wrapper for Hypothesis! I hypothesise that it is possible!