Particle swarm getting stuck in local minima on corner of optimisation domain

[xtalax]

I have been trying to use Optimization.jl with Optim.jl to perform global optimization with ParticleSwarm(), with MethodOfLines.jl generating the data for the loss, and running the forward problem.

I have run in to an issue where the solver gets locked in to only evaluating the upper bound of all parameters, after a few iterations.

Is this the right place for this issue?

My code is somewhat difficult to produce an MWE from, so I thought to ask if this is a known issue before gutting it for debugging fodder.

My optimization setup:

...

LOSS = []                              # Loss accumulator
PARS = []                              # parameters accumulator
using Plots
cb = function (θ, l) #callback function to observe training
    append!(LOSS, l)
    append!(PARS, [θ])
    println("loss = ", l, "; θ = ", θ)
    false
end

println("Generate OptimizationFunction...")
@time opf = OptimizationFunction(loss, Optimization.AutoZygote())

println("Generate OptimizationProblem...")
@time opprob = OptimizationProblem(opf, initp)

_ = loss([1.0,1.0], [])

println("Optimizing...")
@time result = Optimization.solve(opprob, OptimizationOptimJL.ParticleSwarm(lower = [0.0, 20.0], upper = [1.0, 100.0], n_particles=100), maxiters=500, callback=cb)

The output of the run so far:

Optimizing...
loss = 18.22959044129359; θ = [1.0, 30.0]
loss = 11.263023755822093; θ = [0.9909603966537504, 99.77047480421588]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
loss = 9.926057679334475; θ = [1.0, 100.0]
...

Should I post full code?

[thompsonmj]

I think I'm having a related issue for which I opened a Discourse topic.

[pkofod]

Do you know that the solution is not correct?

[thompsonmj]

@pkofod Yep! I mapped out the cost space to be sure that there's a global min in the bounded search space.

Passing along the suggestion that fixed my issue: set abstol and reltol lower for the discretized PDE system solve. I set abstol=1e-64 to convince the optimizer to find the right parameters, but higher values work too where the default is 1e-6.

[thompsonmj]

@xtalax Might be helpful to see rest of the code too ... the optimizer throws parameters to the max if there's a mismatch between the discretized PDE system solve and what's used for comparison in the loss function. e.g. I got mixed up with this for a bit by forgetting to include the saveat for this solve within the loss function.

[pkofod]

Please reopen if you think this is an Optim problem and not a solve problem