ARRDE vs BIPOP-aCMA-ES

[aabbtree77]

If someone is interested in the comparison of the two, check out our discussion on Minion Issue 11. It might provide some food for thought about benchmarks and various algorithms.

It seems that the ARRDE groks CEC2017 F24 in D=20 completely, but it's a bit of a harder case for BIPOP-aCMA-ES if one wants to go below 4% relative error, while BBOB2009 F24 in D=40 is just the opposite case.

At least the ARRDE is also capable to solve a badly scaled rotated ellipsoid, so this calls for an attention ;).

[aabbtree77]

Pay attention that many tougher functions of CEC2017, e.g. F21-F30 (but not all) induce ill-conditioning/rescaling warnings issued by BIPOP-aCMAES.

[aabbtree77]

Functions such as CEC2017 F24 are interesting in that they are "compositions" defined as

           m
f(x) = Σ wᵢ(x) · (fᵢ(Tᵢ(x)) + bᵢ)
          i=1

where every fi:

• is a different benchmark function,
• every one has its own shift,
• every one has its own rotation,
• every one has its own scaling,
• weights depend on distance to each optimum.

So while moving through space you smoothly transition from, say,

• Rastrigin
• into Schwefel
• into Rosenbrock
• into Bent Cigar

without ever being told where the boundaries are.

In particular, F24 is composed of:

Function	Local Condition Number	Main Difficulty
Ackley	≈ 1 (near optimum)	Smooth basin with many local minima farther away
Elliptic	≈ 10⁶	Extreme anisotropy (severe ill-conditioning)
Griewank	≈ 1	Oscillatory landscape with weak multimodality
Rastrigin	≈ 1	Very dense distribution of local minima

[aabbtree77]

1. Does anybody work on improving CMAES (BIPOP-aCMAES) on CEC-2017 F20-F30, or people gave up and it is a DE territory now?

2. What is the state of the art CMAES now, is it still BIPOP restarts? Does anybody use reinforcement learning to adapt step sizes, is it still CSA, or TPA now?

3. For someone willing to run pycma on these tougher CEC-2017 functions, what are some non-default parameters to try to have some hope reaching/beating the ARRDE?

I am using

sigma0 = 20.0 (as everything is bounded to -100, 100 on CEC-2017)

    xbest, es = cma.fmin2(
        objective,
        np.zeros(D),
        #lower + (upper - lower) * np.random.rand(D),
        sigma0,
        options={
            "bounds": [lower, upper],
            "maxfevals": BUDGET,
            "seed": 20260818,
            "verb_disp": 0,
            "verb_log": 0,
            "verb_filenameprefix": None,
            "popsize": None,
        },
        restarts=np.inf,
        bipop=True,
    )

as the best case of the CMAES, but on these difficult CEC-2017 multimodals it just gets into say f=2500 in the first 1M evals on F24 CEC-2017 and then absolutely no progress up to 200M evals. fopt = 2400, the ARRDE gets into f = 2400 in 200M evals.

Notice that one can also run the ARRDE with 10B evals if needed on a local i7 PC bought 15 years ago and get results in 24 hours, while pycma is Python and limited to say 200M evals, which is still a lot. The problem is that it does not do much after 1M evals, so one wonders if there is something to try for very large budgets.

If there is a total radio silence on this thread for a year or so, assume everyone is pushing DEs and CMAES is a dead horse now.

[aabbtree77]