Browse files

Fixed #20877 -- added a performance optimization guide

  • Loading branch information...
1 parent 4db2752 commit dc8f95b639f9ebe65d2188cb3e4f04f9fd68e384 @evildmp evildmp committed Sep 20, 2013
Showing with 403 additions and 22 deletions.
  1. +8 −0 docs/index.txt
  2. +2 −2 docs/ref/utils.txt
  3. +0 −16 docs/topics/cache.txt
  4. +8 −4 docs/topics/db/optimization.txt
  5. +1 −0 docs/topics/index.txt
  6. +384 −0 docs/topics/performance.txt
8 docs/index.txt
@@ -233,6 +233,14 @@ regions:
* :doc:`"Local flavor" <topics/localflavor>`
* :doc:`Time zones </topics/i18n/timezones>`
+Performance and optimization
+There are a variety of techniques and tools that can help get your code running
+more efficiently - faster, and using fewer system resources.
+* :doc:`Performance and optimization overview <topics/performance>`
Python compatibility
4 docs/ref/utils.txt
@@ -495,8 +495,8 @@ Atom1Feed
For cases like this, use the ``django.utils.functional.allow_lazy()``
decorator. It modifies the function so that *if* it's called with a lazy
- translation as the first argument, the function evaluation is delayed until it
- needs to be converted to a string.
+ translation as one of its arguments, the function evaluation is delayed
+ until it needs to be converted to a string.
For example::
16 docs/topics/cache.txt
@@ -1162,22 +1162,6 @@ Example::
.. _`Cache-Control spec`:
-Other optimizations
-Django comes with a few other pieces of middleware that can help optimize your
-site's performance:
-* ``django.middleware.http.ConditionalGetMiddleware`` adds support for
- modern browsers to conditionally GET responses based on the ``ETag``
- and ``Last-Modified`` headers.
-* :class:`django.middleware.gzip.GZipMiddleware` compresses responses for all
- modern browsers, saving bandwidth and transfer time. Be warned, however,
- that compression techniques like ``GZipMiddleware`` are subject to attacks.
- See the warning in :class:`~django.middleware.gzip.GZipMiddleware` for
- details.
12 docs/topics/db/optimization.txt
@@ -88,7 +88,8 @@ of parentheses, but will call callables automatically, hiding the above
Be careful with your own custom properties - it is up to you to implement
+caching when required, for example using the
+:class:`~django.utils.functional.cached_property` decorator.
Use the ``with`` template tag
@@ -111,10 +112,11 @@ For instance:
* At the most basic level, use :ref:`filter and exclude <queryset-api>` to do
filtering in the database.
-* Use :class:`F expressions <django.db.models.F>` to do filtering
- against other fields within the same model.
+* Use :class:`F expressions <django.db.models.F>` to filter
+ based on other fields within the same model.
-* Use :doc:`annotate to do aggregation in the database </topics/db/aggregation>`.
+* Use :doc:`annotate to do aggregation in the database
+ </topics/db/aggregation>`.
If these aren't enough to generate the SQL you need:
@@ -233,6 +235,8 @@ queryset``.
+.. _overuse_of_count_and_exists:
Don't overuse ``count()`` and ``exists()``
1 docs/topics/index.txt
@@ -26,6 +26,7 @@ Introductions to all the key parts of Django you'll need to know:
+ performance
384 docs/topics/performance.txt
@@ -0,0 +1,384 @@
+Performance and optimization
+This document provides an overview of techniques and tools that can help get
+your Django code running more efficiently - faster, and using fewer system
+Generally one's first concern is to write code that *works*, whose logic
+functions as required to produce the expected output. Sometimes, however, this
+will not be enough to make the code work as *efficiently* as one would like.
+In this case, what's needed is something - and in practice, often a collection
+of things - to improve the code's performance without, or only minimally,
+affecting its behavior.
+General approaches
+What are you optimizing *for*?
+It's important to have a clear idea what you mean by 'performance'. There is
+not just one metric of it.
+Improved speed might be the most obvious aim for a program, but sometimes other
+performance improvements might be sought, such as lower memory consumption or
+fewer demands on the database or network.
+Improvements in one area will often bring about improved performance in
+another, but not always; sometimes one can even be at the expense of another.
+For example, an improvement in a program's speed might cause it to use more
+memory. Even worse, it can be self-defeating - if the speed improvement is so
+memory-hungry that the system starts to run out of memory, you'll have done
+more harm than good.
+There are other trade-offs to bear in mind. Your own time is a valuable
+resource, more precious than CPU time. Some improvements might be too difficult
+to be worth implementing, or might affect the portability or maintainability of
+the code. Not all performance improvements are worth the effort.
+So, you need to know what performance improvements you are aiming for, and you
+also need to know that you have a good reason for aiming in that direction -
+and for that you need:
+Performance benchmarking
+It's no good just guessing or assuming where the inefficiencies lie in your
+Django tools
+<>`_ is a very
+handy tool that provides insights into what your code is doing and how much
+time it spends doing it. In particular it can show you all the SQL queries your
+page is generating, and how long each one has taken.
+Third-party panels are also available for the toolbar, that can (for example)
+report on cache performance and template rendering times.
+Third-party services
+There are a number of free services that will analyse and report on the
+performance of your site's pages from the perspective of a remote HTTP client,
+in effect simulating the experience of an actual user.
+These can't report on the internals of your code, but can provide a useful
+insight into your site's overall performance, including aspects that can't be
+adequately measured from within Django environment. Examples include:
+* `Yahoo's Yslow <>`_
+* `Google PageSpeed <>`_
+There are also several paid-for services that perform a similar analysis,
+including some that are Django-aware and can integrate with your codebase to
+profile its performance far more comprehensively.
+Get things right from the start
+Some work in optimization involves tackling performance shortcomings, but some
+of the work can simply be built in to what you'd do anyway, as part of the good
+practices you should adopt even before you start thinking about improving
+In this respect Python is an excellent language to work with, because solutions
+that look elegant and feel right usually are the best performing ones. As with
+most skills, learning what "looks right" takes practice, but one of the most
+useful guidelines is:
+Work at the appropriate level
+Django offers many different ways of approaching things, but just because it's
+possible to do something in a certain way doesn't mean that it's the most
+appropriate way to do it. For example, you might find that you could calculate
+the same thing - the number of items in a collection, perhaps - in a
+``QuerySet``, in Python, or in a template.
+However, it will almost always be faster to do this work at lower rather than
+higher levels. At higher levels the system has to deal with objects through
+multiple levels of abstraction and layers of machinery.
+That is, the database can typically do things faster than Python can, which can
+do them faster than the template language can::
+ # QuerySet operation on the database
+ # fast, because that's what databases are good at
+ my_bicycles.count()
+ # counting Python objects
+ # slower, because it requires a database query anyway, and processing
+ # of the Python objects
+ len(my_bicycles)
+ # Django template filter
+ # slower still, because it will have to count them in Python anyway,
+ # and because of template language overheads
+ {{ my_bicycles|length }}
+Generally speaking, the most appropriate level for the job is the lowest-level
+one that it is comfortable to code for.
+.. note::
+ The example above is merely illustrative.
+ Firstly, in a real-life case you need to consider what is happening before
+ and after your count to work out what's an optimal way of doing it *in that
+ particular context*. The database optimization documents describes :ref:`a
+ case where counting in the template would be better
+ <overuse_of_count_and_exists>`.
+ Secondly, there are other options to consider: in a real-life case, ``{{
+ my_bicycles.count }}``, which invokes the ``QuerySet`` ``count()`` method
+ directly from the template, might be the most appropriate choice.
+Often it is expensive (that is, resource-hungry and slow) to compute a value,
+so there can be huge benefit in saving the value to a quickly accessible cache,
+ready for the next time it's required.
+It's a sufficiently significant and powerful technique that Django includes a
+comprehensive caching framework, as well as numerous other opportunities to
+make use of caching.
+:doc:`The caching framework </topics/cache>`
+Django's :doc:`caching framework </topics/cache>` offers very significant
+opportunities for performance gains, by saving dynamic content so that it
+doesn't need to be calculated for each request.
+For convenience, Django offers different levels of cache granularity: you can
+cache the output of specific views, or only the pieces that are difficult to
+produce, or even an entire site.
+Implementing caching should not be regarded as an alternative to improving code
+that's performing poorly because it has been written badly. It's one of the
+final steps towards producing well-performing code, not a shortcut.
+Other opportunities for caching
+Beyond the caching framework, Django offers other smaller pieces of caching
+It's common to have to call a class instances's method more than once. If
+that function is expensive, then doing so can be wasteful.
+Using the ``@cached_property`` decorator saves the value returned by a
+property; the next time the function is called on that instance, it will return
+the saved value rather than re-computing it. Note that this only works on
+methods that take ``self`` as their only argument and that it changes the
+method to a property.
+:class:`` appends a
+content-dependent tag to the filenames of :doc:`static files
+</ref/contrib/staticfiles>` to make it safe for browsers to cache them
+long-term without missing future changes - when a file changes, so will the
+tag, so browsers will reload the asset automatically.
+Understanding laziness
+*Laziness* is a strategy complementary to caching. Caching avoids
+recomputation by saving results; laziness delays computation until it's
+actually required.
+Laziness allows us to refer to things before they are instantiated, or even
+before it's possible to instantiate them. This has numerous uses.
+For example, :ref:`lazy translation <lazy-translations>` can be used before the
+target language is even known, because it doesn't take place until the
+translated string is actually required, such as in a rendered template.
+Laziness is also a way to save effort by trying to avoid work in the first
+place. That is, one aspect of laziness is not doing anything until it has to be
+done, because it may not turn out to be necessary after all. Laziness can
+therefore have performance implications, and the more expensive the work
+concerned, the more there is to gain through laziness.
+Python provides a number of tools for lazy evaluation, particularly through the
+:py:term:`generator` and :py:term:`generator expression` constructs. It's worth
+reading up on laziness in Python to discover opportunities for making use of
+lazy patterns in your code.
+Laziness in Django
+Django is itself quite lazy. A good example of this can be found in the
+evaluation of ``QuerySets``. :ref:`QuerySets are lazy <querysets-are-lazy>`.
+Thus a ``QuerySet`` can be created, passed around and combined with other
+``QuerySets``, without actually incurring any trips to the database to fetch
+the items it describes. What gets passed around is the ``QuerySet`` object, not
+the collection of items that - eventually - will be required from the database.
+On the other hand, :ref:`certain operations will force the evaluation of a
+QuerySet <when-querysets-are-evaluated>`. Avoiding the premature evaluation of
+a ``QuerySet`` can save making an expensive and unnecessary trip to the
+Django also offers an :meth:`~django.utils.functional.allow_lazy` decorator.
+This allows a function that has been called with a lazy argument to behave
+lazily itself, only being evaluated when it needs to be. Thus the lazy argument
+- which could be an expensive one - will not be called upon for evaluation
+until it's strictly required.
+:doc:`Database optimization </topics/db/optimization>`
+Django’s database layer provides various ways to help developers get the best
+performance from their databases. The :doc:`database optimization documentation
+</topics/db/optimization>` gathers together links to the relevant
+documentation and adds various tips that outline the steps to take when
+attempting to optimize your database usage.
+Other database-related tips
+Enabling :ref:`persistent-database-connections` can speed up connections to the
+database accounts for a significant part of the request processing time.
+This helps a lot on virtualized hosts with limited network performance, for example.
+HTTP performance
+Django comes with a few helpful pieces of :doc:`middleware </ref/middleware>`
+that can help optimize your site's performance. They include:
+Adds support for modern browsers to conditionally GET responses based on the
+``ETag`` and ``Last-Modified`` headers.
+Compresses responses for all modern browsers, saving bandwidth and transfer
+time. Note that GZipMiddleware is currently considered a security risk, and is
+vulnerable to attacks that nullify the protection provided by TLS/SSL. See the
+warning in :class:`~django.middleware.gzip.GZipMiddleware` for more information.
+Third-party HTTP tools
+There are numerous third-party Django tools and packages available, notably
+ones that are able to "minify" and compress HTML, CSS and JavaScript.
+Template performance
+Note that:
+* using ``{% block %}`` is faster than using ``{% include %}``
+* heavily-fragmented templates, assembled from many small pieces, can affect
+ performance
+The cached template loader
+Enabling the :class:`cached template loader
+<django.template.loaders.cached.Loader>` often improves performance
+drastically, as it avoids compiling each template every time it needs to be
+Using different versions of available software
+It can sometimes be worth checking whether different and better-performing
+versions of the software that you're using are available.
+This may be helpful, but is unlikely to solve a serious performance problem.
+You won't usually gain performance advantages that are better than marginal.
+.. note::
+ It's worth repeating: **reaching for alternatives to software you're
+ already using is very rarely the answer to performance problems**.
+Newer is often - but not always - better
+It's fairly rare for a new release of well-maintained software to be less
+efficient, but the maintainers can't anticipate every possible use-case - so
+while being aware that newer versions are likely to perform better, don't
+simply assume that they always will.
+This is true of Django itself. Successive releases have offered a number of
+improvements across the system, but you should still check the real-world
+performance of your application, because in some cases you may find that
+changes mean it performs worse rather than better.
+Newer versions of Python, and also of Python packages, will often perform
+better too - but measure, rather than assume.
+.. note::
+ Unless you've encountered an unusual performance problem in a particular
+ version, you'll generally find better features, reliability, and security
+ in a new release and that these benefits are far more significant than any
+ performance you might win or lose.
+Alternatives to Django's template language
+For nearly all cases, Django's built-in template language is perfectly
+adequate. However, if the bottlenecks in your Django project seem to lie in the
+template system and you have exhausted other opportunities to remedy this, a
+third-party alternative may be the answer.
+`Jinja2 <>`_ can offer performance improvements,
+particularly when it comes to speed.
+Alternative template systems vary in the extent to which they share Django's
+templating language.
+.. note::
+ *If* you experience performance issues in templates, the first thing to do
+ is to understand exactly why. Using an alternative template system may
+ prove faster, but the same gains may also be available without going to
+ that trouble - for example, expensive processing and logic in your
+ templates could be done more efficiently in your views.
+Alternative software implementations
+It *may* be worth checking whether Python software you're using has been
+provided in a different implementation that can execute the same code faster.
+However, most Django performance problems in well-written code are typically
+not to be found at the Python execution level, but rather in inefficient
+database querying, caching, and templates (and if you're relying on
+poorly-written Python code, your performance problems are very unlikely to be
+solved by having it execute faster).
+Avoid using C implementations of Python libraries or non-standard Python
+implementations like `PyPy <>`_ in search of performance gains,
+unless you are sure they are appropriate for your application. Any gains are
+likely to be small, and compatibility issues are common.

0 comments on commit dc8f95b

Please sign in to comment.