Visualization // Documentation
Benchmark any (Python) function, store (as csv or Excel), read and visualize the results with only a few lines of code!
Table of Contents:
The general structure of a benchmark script is as follows:
- you have some algorithm
- you want to test the algorithm by varying over a set of inputs
- you assess quantities of interest (i.e. some performance metric) to the output of the algorithm
This can be implemented as a Python function:
def benchmark_function(input):
result = algorithm(input)
return {"name_performance_metric_1": performance_metric_1(result),"name_performance_metric_2": performance_metric_1(result),...}
In order to benchmark your algorithm you simply need to call the above function to all sets of inputs. Furthermore, the output of an algorithm is rarely deterministic. Therefore, you want to repeat evaluating your algorithm at a given input in order to statistical analyze the performance. All of this and storing your results is taken care of by the Benchmark class in pybe.benchmark. Lets look at a concrete example.
Lets say you have an optimization algorithm implemented in Python which takes as inputs
- a function to be optimized and
- the number of runs.
You want to evaluate the optimizer on a certain test function and benchmark how well the optimizer performs for specific number of runs. For this you have a performance metric which can be called to the output of the optimization and returns a real number (float).
Then, your benchmark function looks as follows:
def benchmark_function(number_of_runs):
result_optimizer = optimizer(test_function,number_of_runs)
return {"name_performance_metric": performance_metric(result_optimizer)}
Lets say you want to benchmark your optimization algorithm for number of runs 10,100 and 1000. In order to statistical analyse the performance, you evaluate 200 times at each input. Now, you can simply benchmark your optimization algorithm by using the pybe Benchmark class.
from pybe.benchmark import Benchmark
benchmark = Benchmark()
benchmark(function=benchmark_function,inputs=[10,100,1000],number_runs=200,name="name_of_my_optimization_algorithm")
Drag the resulting name_of_my_optimization_algorithm.csv into the Dashboard and thats it!
The official release is available at PyPi:
pip install pybe
You can clone this repository by running the following command:
git clone https://github.com/nicolaipalm/pybe
cd pybe
pip install
In order to benchmark a Python function you only need to implement the function and specify some data of the benchmark.
from pybe.benchmark import Benchmark
from pybe.wrappers import timer
import time
benchmark = Benchmark() # initialize pybe's benchmark class
@timer # additionally track the time needed in each iteration
def test_function(i: int):
time.sleep(0.1)
return {"name_of_output": i} # specify the output in a dictionary
# benchmark test_function on inputs [1,2,3] and evaluate each input 10 times
benchmark(test_function,
name="test_benchmark", # set the name of the benchmark
inputs=[1, 2, 3],
store=True, # store the benchmark results
number_runs=10)
Look at the test_benchmark.csv file in your directory!
You can view the results also directly in Python or write them to an Excel or csv file
print(benchmark.inputs, benchmark.name_outputs) # print inputs and names of outputs
print(benchmark.result) # print results as stored in benchmark.csv
benchmark.to_excel(name="my_results") # write results as excel
benchmark.to_csv(name="my_results") # write results as csv
You can read any of the benchmark results by simply initializing the benchmark class with parameter the .yaml benchmark file path
benchmark = Benchmark(benchmark_file_path)
The structure of the resulting csv is supposed to be very intuitive:
- each row represents one call of the benchmarked function with
- one column for the input
- one column with the name of the benchmark
- one column for each output
For example:
- the function has two outputs: time and value
- is benchmarked at inputs 10 and 100
- has name Hello
- is evaluated once for each input
Then, the resulting csv/Excel has the following structure:
value | time | Input | Name | |
---|---|---|---|---|
0 | 0.1 | 1 | 10 | hello |
1 | 0.05 | 20 | 100 | hello |