Home
Welcome to the python benchmark harness documentation on the right-hand side you can find the legend which can bring you quickly to the topic of your choice.
A micro/macro benchmark is an act of testing the performance of a framework, algorithm, routine, or application to make sure it performs as expected. python benchmark-harness is therefore a benchmarking framework for the Python programming language that allows you to create your benchmarks for your code. The scope of what you want to encompass within your benchmark and how you want to define it is completely up to the user of this framework. But the main goal of a benchmark is to stress your logic with a synthetic workload and collect measurements about how well it is performing.
It is recommended however to keep a majority of your benchmark test runs within a short duration so you will be provided with some performance feedback faster and more often this makes debugging problems more comfortable. Keeping your benchmarks relatively short in overall runtime will also allow making it relatively a bit easier to integrate alongside your functional unit tests within a CI/CD pipeline.
Within this framework, you can create, compare, analyze, save, and visualize the performance of your code in an automated way by integrating it into a unit test framework. Giving you the option to have an automated self-defined "performance test" of your performance-critical parts of the application you are developing.
Allowing you to spot performance degradations more early within the software development life cycle. Helping you to deliver a more performant end product to your users. All of these options packed within this framework are documented within this wiki letting you start leveraging the aid of the performance benchmarks within the Python programming language.
The installation for the latest official release of the python-micro-benchmarks package is easy and straightforward. You can download the latest version directly from the package index using pip. This can be done with the following command:
pip install python-micro-benchmarksTo install the latest version of python-micro-benchmarks, you need first to obtain a copy of the source. You can do that by cloning the git repository:
git clone git://github.com/JoeyHendricks/python-micro-benchmarks.gitTo check that python-micro-benchmarks has been properly installed, type python from your shell. Then at the Python prompt, try to import python-micro-benchmarks, and check the installed version:
>>> import Benchmarking
>>> Benchmarking.__version__
1.0.1The micro-benchmark object is an overarching control scheme for controlling both the non-intrusive and intrusive benchmarking/profiling efforts. It is used to define test metadata, access raw test results, and execute non-intrusive benchmarks. It is possible to import this object from the package the following way: from Benchmarking import micro_benchmark as mb. Once imported it is possible to start accessing the properties to define your benchmarking test case and run your own custom benchmarks that are automatically profiled and stored in a database.
Consider the following graphic it displays a high-level overview of what relevant data can be accessed by an end-user:
With the above in mind using the benchmark object it is possible to execute two types of benchmarks that are defined and controlled in a similar way within the benchmarking object:
-
Non-intrusive benchmarks: These do not need any changes to your code and they can be defined by the use of a JSON template given to the run method of the benchmarking object. The benefit of this non-intrusive benchmark is that it can be more easily defined, maintained, and integrated into a unit test framework. Besides that managing benchmark meta like test-id and test case can be more easily managed from a more unit test approach to your benchmarks.
-
Intrusive benchmark: These require that the "collect_measurements" decorator is used on the method or function you wish to benchmark. This means for your benchmarks to work your target function needs to be instrumented, have a test case name defined, and the "enabled" boolean needs to be set to true so measurements will be collected otherwise your benchmark will fail.
The main benefit of this decorator is that it provides even more flexibility to how you want to design your benchmark or what you want to benchmark. Besides that, it is useful for testing purposes it also provides an extra layer of logging within production as an extra logging tool that can be turned on when desired. This in term will add some additional overhead to determine if this acceptable is, a balancing game that would need to be carefully weighed.
Both intrusive and non-intrusive benchmarking have them up and downsides however for most pre-production situations it is would be recommended to use the non-intrusive method as it is easier the use and to maintain it also provides relatively the same value besides. In the subsequent sections, the non-intrusive method of benchmarking with this framework will be further explained.
It is possible to automatically detect regression while comparing two benchmarks with each other this is done ---