Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 87 lines (62 sloc) 3.272 kb
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
1 Djangobench
2 ===========
3
048fe9e @jacobian Initial refactoring into a proper package.
jacobian authored
4 A harness and a set of benchmarks for measuring Django's performance over
5 time.
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
6
7 Running the benchmarks
8 ----------------------
9
048fe9e @jacobian Initial refactoring into a proper package.
jacobian authored
10 Here's the short version::
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
11
048fe9e @jacobian Initial refactoring into a proper package.
jacobian authored
12 mkvirtualenv --no-site-packages djangobench
5510f26 @jacobian Support for benchmarking within a git repo.
jacobian authored
13 pip install -e git://github.com/jacobian/djangobench.git#egg=djangobench
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
14 svn co http://code.djangoproject.com/svn/django/tags/releases/1.2/ django-control
15 svn co http://code.djangoproject.com/svn/django/trunk django-experiment
048fe9e @jacobian Initial refactoring into a proper package.
jacobian authored
16 djangobench
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
17
048fe9e @jacobian Initial refactoring into a proper package.
jacobian authored
18 Okay, so what the heck's going on here?
19
20 First, ``djangobench`` doesn't test a single Django version in isolation --
21 that wouldn't be very useful. Instead, it benchmarks an "experiment" Django
22 against a "control", reporting on the difference between the two and
23 measuring for statistical significance.
24
25 So to run this, you'll need two complete Django source trees. By default
26 ``djangobench`` looks for directories named ``django-control`` and
27 ``django-experiment`` in the current working directory, but you can change
28 that by using the ``--control`` or ``--experiment`` options.
29
30 Now, because you need two Django source trees, you can't exactly install
31 them: ``djangobench`` works its magic by mucking with ``PYTHONPATH``.
32 However, the benchmarks themselves need access to the ``djangobench``
33 module, so you'll need to install it.
34
5510f26 @jacobian Support for benchmarking within a git repo.
jacobian authored
35 If you're feeling fancy, you can use one of them there newfangled DVCSes instead
36 and test against a single repository containing branches::
37
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
38 git clone git://github.com/django/django.git
39 djangobench --vcs=git --control=1.2 --experiment=master
40
5510f26 @jacobian Support for benchmarking within a git repo.
jacobian authored
41 Git's the only supported VCS right now, but patches are welcome.
42
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
43 At the time of this writing Django's trunk hasn't significantly diverged
44 from Django 1.2, so you should expect to see not-statistically-significant
45 results::
46
47 Running 'startup' benchmark ...
48 Min: 0.138701 -> 0.138900: 1.0014x slower
49 Avg: 0.139009 -> 0.139378: 1.0027x slower
50 Not significant
51 Stddev: 0.00044 -> 0.00046: 1.0382x larger
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
52
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
53 Writing new benchmarks
54 ----------------------
55
56 Benchmarks are very simple: they're a Django app, along with a settings
22561f1 @jacobian Made trials something that each benchmark script handles; added support ...
jacobian authored
57 file, and an executable ``benchmarks.py`` that gets run by the harness. The
58 benchmark script needs to honor a simple contract:
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
59
22561f1 @jacobian Made trials something that each benchmark script handles; added support ...
jacobian authored
60 * It's an executable Python script, run as ``__main__`` (e.g. ``python
61 path/to/benchmark.py``). The subshell environment will have
62 ``PYTHONPATH`` set up to point to the correct Django; it'll also have
63 ``DJANGO_SETTINGS_MODULE`` set to ``<benchmark_dir>.settings``.
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
64
22561f1 @jacobian Made trials something that each benchmark script handles; added support ...
jacobian authored
65 * The benchmark script needs to accept a ``--trials`` argument giving
66 the number of trials to run.
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
67
22561f1 @jacobian Made trials something that each benchmark script handles; added support ...
jacobian authored
68 * The output should be simple RFC 822-ish text -- a set of headers,
69 followed by data points::
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
70
22561f1 @jacobian Made trials something that each benchmark script handles; added support ...
jacobian authored
71 Title: some benchmark
72 Description: whatever the benchmark does
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
73
22561f1 @jacobian Made trials something that each benchmark script handles; added support ...
jacobian authored
74 1.002
75 1.003
76 ...
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
77
22561f1 @jacobian Made trials something that each benchmark script handles; added support ...
jacobian authored
78 The list of headers is TBD.
79
80 There's a couple of utility functions in ``djangobench.utils`` that assist
81 with honoring this contract; see those functions' docstrings for details.
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
82
048fe9e @jacobian Initial refactoring into a proper package.
jacobian authored
83 The existing benchmarks should be pretty easy to read for inspiration. The
84 ``query_delete`` benchmark is probably a good place to start.
ff2bd52 @jacobian Hey, look, working code!
jacobian authored
85
340bfe5 @sebleier Small update to example DVCS usage
sebleier authored
86 **Please write new benchmarks and send me pull requests on Github!**
Something went wrong with that request. Please try again.