BlackBoxOptimizationBenchmarking.jl

A Julia implementation of the Black-Box-Optimization-Benchmarking (BBOB) functions.

Benchmark results

The average sucess rate (meaning the optimizer reached the minimum + 1e-6) in function of the number of objective function evaluations :

Since some global optimizers have poor final convergence, they were chained into a Nelder-Mead using 10% of the objective function evaluation budget.

The average sucess rate across the dimension of the function:

The total relative run time of each optimizer

Note that the Python algorithms are called from Julia, which might cause some overhead.

Functions

Functions can be accessed as BlackBoxOptimizationBenchmarking.F1, which returns a BBOBFunction with fields f containing the function itself, f_opt its minimal value, and x_opt its minimizer, i.e. f(x_opt) = f_opt.

Functions can be listed using list_functions():

julia> BlackBoxOptimizationBenchmarking.list_functions()
20-element Array{BlackBoxOptimizationBenchmarking.BBOBFunction,1}:
 Sphere                                           
 Ellipsoidal Function                             
 Discus Function                                  
 Bent Cigar Function                              
 Sharp Ridge Function                             
 Different Powers Function                        
 Rastrigin Function                               
 Weierstrass Function                             
 Schaffers F7 Function                            
 Schaffers F7 Function, moderately ill-conditioned
 Composite Griewank-Rosenbrock Function F8F2      
 Ellipsoidal                                      
 Schwefel Function                                
 Rastrigin                                        
 Buche-Rastrigin                                  
 Linear Slope                                     
 Attractive Sector                                
 Step Ellipsoidal Function                        
 Rosenbrock Function, original                    
 Rosenbrock Function, rotated

A benchmark for a single optimizer and function can be run with:

benchmark(optimizer::Any, f::BBOBFunction, run_lengths, Ntrials, dimensions, Δf)

Or for a collection of optimizers with:

benchmark(optimizers::Vector{T}, funcs, run_lengths, Ntrials, dimensions, Δf)

See test/runtests.jl

The optimizer must implement the methods optimize, minimum and minimizer, see

scripts/optimizers_interface.jl

Generating new instance of the functions

To avoid overfiting and test if algorithms are robust with respect to rotations of the error function, rotation matrices are randomly generated the first time the package is used.

If needed new rotations can be generated by running the following:

@eval BlackBoxOptimizationBenchmarking begin
    @memoize function Q(D)
        r = randn(D); r = r/norm(r)
        Q = [r nullspace(Matrix(r'))]
    end
    @memoize function R(D)
        r = randn(D); r = r/norm(r)
        R = [r nullspace(Matrix(r'))]
    end
end

Reference:

http://coco.lri.fr/downloads/download15.01/bbobdocfunctions.pdf