Optimizing Django REST Framework performance with django-auto-prefetching
Django-auto-prefetching is a library that automatically prefetches data from the database when using the django-rest-framework
tldr: django-auto-prefetching is a library that automatically optimizes your endpoints by using
select_related to fetch the correct objects
from the database when using django-rest-framework. You can find it on PyPI and Github
Django REST Framework and the n+1 problem
The Django REST Framework(DRF) is a framework for quickly building robust REST API’s. However when fetching models with nested relationships we run into performance issues. DRF becomes slow.
This isn’t due to DRF itself, but rather due to the n+1 problem.
When we have a model, say
ItalianChef with a relationship like
FavouriteDish, and we want to fetch all italian chefs with their corresponding favourite dishes.
This means we potentially make a lot of queries as DRF first goes to fetch all the Chefs, and then for each of those chefs, it separately fetches their favourite dish.
This means that if there’s twenty chefs, we’ll end up making twenty separate database calls for favourite dishes.
Django has a built in solution to this problem,
which tells the ORM what related objects you’re going to need. This means instead of doing a bunch of database calls, we can just do one.
There’s two issues with this however - it’s can hard to track exactly what relationships you’ll end up traversing,
and it’s time consuming to manually write
not to mention keeping them updated as code elsewhere changes.
The goal of this library is to, as effortlessly and as painlessly as possible, make sure you don’t suffer performance problems due to n+1 issues. We’re starting in the small with automatic prefetching for DRF serializers.
What does it do?
Django-auto-prefetching has a ModelViewSet mixin, which looks at the fields any given serializer uses, and automatically calculates the correct
This means that one line of code is all of the optimization many of your views will ever need!
Is it production ready?
django-auto-prefetching is being used internally at the danish startup reccoon, in a Django codebase of about 20k lines of code. We had a lot of endpoints that weren’t optimized, and we’ve seen a general 30-40% speedup in response times across our whole API, just by inheriting our ViewSets from AutoPrefetchMixin
I think there's many more improvements to be had in automatic prefetching, and I think it could potentially solve a lot of pain points in Django.
With more engineering effort it should be possible to trace the lifespan of models coming from a specific
QuerySet, and automatically prefetch the related
objects needed. It's probably never going to be perfect, but Django is very smart in a lot of aspects,
and it seems wild that we're still writing this part by hand.