Surrogate Model Improvements

[schmoelder]

#45 implements a surrogate model of the optimization problem. The aim of that PR is a MVP, allowing us to explore further options.

This issue here collects some further ideas to improve surrogate modelling in CADET-Process.

SurrogateModel

• Catch classifier error for only feasible/infeasible points (e.g. write lambda function that takes only x and returns
• an array of only true or only false respectively

Improve find_x0

• Handle situations where only one sided neighbor if found (e.g. extrapolate using gradient from two points to the left/right)
• Use multiple points to interpolate (e.g. 5 nearest neighbors)
• Benchmark performance of optimization with starting values (find_x0 vs just some random point)
• Exclude points that violate nonlinear constraints
• The usage of GP for extrapolation in the initial value search is okay.
• Mixing different approaches is acceptable, but it causes ongoing issues.
• using inverted GPs are helpful, but again break down at in regions where the problem has not been sampled yet
• (Optional) Reduce overfitting in the training process through cross-validation.

minimize / fill_gaps

• write tests

Future

Use GP of minimum boundary as an objective for a global optimization approach

Fit GP to the minimum boundary and use only the real simulator for individual optimization steps. This falls into the category of decision-making and combines Ax and conditioning.

1. There is the conditioning approach, which is computationally intensive: $min f_{min}(x_0=x_0')$, where Ax tries different $x_0'$ and improves the surrogate function of the minimum boundary piece by piece. A suitable trade-off between exploration and exploitation needs to be found.

2. Alternatively, following Michael's approach: How can we formulate an objective so that it learns conditional minima for all $x_0'$ (find the optimal path of minima)?
   
   The exact form of the objective needs to be clarified.

Optimize the functional relation of f_{min}(x_0) with a surrogate model or at least return their relationship as a GP

Optimize the GP for the minimum boundary itself on a GP surrogate of the objective space. This would reduce the number of necessary steps. This falls into the category of post-processing visualization.

1. This could be improved by non-uniformly sampling over the parameter space using a GP. Essentially, an alternative 'find_minimum' approach would be written.
2. Ideally, a GP for the functional relationship between $f_{min}(x_0)$ and $x_0$ would be fitted. This can be done either on the uniform grid points or directly using approach 1. The advantage is that the uncertainty of the surrogate model would be indicated.