collective.funkload

Complex functional load testing and benchmarking

collective.funkload provides some extensions of Funkload, a web performance testing and reporting tool. These extensions provide flexible yet simple ways to:

• run benchmarks of multiple test scenarios
• run these benchmarks against multiple setups
• generate comparisons between those setups

All of the console scripts provided by collective.funkload provide a "--help" option which documents the command and its options.

Table of Contents

• The collective.funkload Workflow
  • 1. Develop test scenarios in Python eggs as with unittest
  • 2. Benchmark the baseline setup
  • 3. Benchmark the other setups
  • 4. Build the HTML and differential reports and the matrix index
  • 5. Examine the results using the matrix index
  • 6. Repeat as changes are made
• fl-run-bench

The collective.funkload Workflow

1. Develop test scenarios in Python eggs as with unittest

Funkload test cases can be generated using the funkload test recorder and then placed in a Python egg's "tests" package as with normal unittest test cases.

These test cases should be developed which reflect all of the application's supported usage patterns. Take care to balance between separating tests by usage scenario (anonymous visitors, read access, write access, etc.) and keeping the number of tests low enough to scan results.

2. Benchmark the baseline setup

If there is no single baseline, such as when comparing multiple setups to each other, then the term baseline here might be slightly inaccurate. The important part, however, is to establish that the test cases successfully cover the usage scenarios.

Use the fl-run-bench provided by collective.funkload using zope.testing.testrunner semantics to specify the tests to run and the "--label" option to specify a label indicating the benchmark it the baseline (or some other label if baseline isn't appropriate):

$ fl-run-bench -s foo.loadtests --label=baseline
Running zope.testing.testrunner.layer.UnitTests tests:
  Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
========================================================================
Benching FooTestCase.test_FooTest
========================================================================
...
Bench status: **SUCCESS**
  Ran # tests with 0 failures and 0 errors in # minutes #.### seconds.
Tearing down left over layers:
  Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.

Use "fl-build-report --html" to build an HTML report from the XML generated by running the benchmark above:

$ fl-build-report --html FooTest-bench-YYYYMMDDThhmmss.xml
Creating html report ...done:
file:///.../test_ReadOnly-YYYYMMDDThhmmss-baseline/index.html

Examine the report details. If the test cases don't sufficiently cover the application's supported useage patterns, repeat steps 1 and 2 until the test cases provide sufficient coverage.

3. Benchmark the other setups

In turn, deploy each of the setups. This procedure will be dictated by the application. If using different buildout configurations, for example, deploy each configuration:

$ ...

Then use the same fl-run-bench command as before (or adusted as needed for the setup) giving a differen "--label" option dsignating the setup:

$ fl-run-bench -s foo.loadtests --label=foo-setup
Running zope.testing.testrunner.layer.UnitTests tests:
  Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
========================================================================
Benching FooTestCase.test_FooTest
========================================================================
...
Bench status: **SUCCESS**
  Ran # tests with 0 failures and 0 errors in # minutes #.### seconds.
Tearing down left over layers:
  Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.

Repeat this step for each setup.

4. Build the HTML and differential reports and the matrix index

Use the "fl-build-label-reports" command with the "--x-label" and "--y-label" options to automatically build all the HTML reports, the differential reports based on the labels, and an index matrix to the HTML and differential reports. The "fl-build-label-reports" script will use a default title and sub-title based on the labels but may specified using the "--title" and "--sub-title" options. Arbitrary text or HTML may also be included on stdin or using the "--input" option:

$ echo "When deciding which setup to use..." | \
fl-build-label-reports --x-label=baseline --y-label=foo-setup \
--y-label=bar-setup --title="Setup Comparison"
--sub-title="Compare setups foo and bar against baseline"
Creating html report ...done:
file:///.../test_ReadOnly-YYYYMMDDThhmmss-baseline/index.html
Creating html report ...done:
file:///.../test_ReadOnly-YYYYMMDDThhmmss-foo-label/index.html
Creating html report ...done:
file:///.../test_ReadOnly-YYYYMMDDThhmmss-bar-label/index.html
Creating diff report ...done:
/.../diff_ReadOnly-YYYYMMDDT_hhmmss-foo-label_vs_hhmmss-baseline/index.html
Creating diff report ...done:
/.../diff_ReadOnly-YYYYMMDDT_hhmmss-bar-label_vs_hhmmss-baseline/index.html
Creating report index ...done:
file:///.../index.html

Both the "--x-label" and "--y-label" options may be given multiple times or may use Python regular expressions to create an MxN matrix of differential reports. See the "fl-build-label-reports --help" documentation for more details.

5. Examine the results using the matrix index

Open the index.html generated by the last command to survey the HTML reports and differential reports.

6. Repeat as changes are made

As changes are made in your application or setups or to test new setups, repeat steps 3 and 4. When step 4 is repeated by running "fl-build-label-reports" adjusting the "--x-label" and "--y-label" options as appropriate, new HTML and differential reports will be generated as appropriate for the new load test benchmark results and the matrix index will be updated.

fl-run-bench

The scripts that Funkload installs generally require that they be executed from the directory where the test modules live. While this is appropriate for generating test cases with the Funkload recorder, it's often not the desirable behavior when running load test benchmarks. Additionally, the argument handling for the benchmark runner doesn't allow for specifying which tests to benchmark using the zope.testing.testrunner semantics, such as specifying modules and packages with dotted paths, as one is often wont to do when working with setuptools and eggs.

To accommodate this usage pattern, the collective.funkload package provides a wrapper around the Funkload benchmark runner that handles dotted path arguments gracefully. Specifically, rather than pass *.py file and TestCase.test_method arguments, the "fl-bench-runner" provided by collective.funkload supports zope.testing.testrunner semantics for finding tests with "-s", "-m" and "-t".

>>> from collective.funkload import bench
>>> bench.run(defaults, (
...     'test.py -s foo -t test_foo '
...     '--cycles 1 --url http://bar.com').split())
t...
Benching FooTestCase.test_foo...
* Server: http://bar.com...
* Cycles: [1]...