What questions should we investigate with benchmarking?

[SRFU-NN]

Now that we have benchmarking, we want to use it to investigate specific and well-defined questions. Suggest them here, and vote for the ones you think we should start with.

[SRFU-NN]

Naive Expected Improvement vs. fudged Expected Improvement (Xi), vs. fudged expected improvement with changing Xi

[SRFU-NN]

Benchmark on a physically relevant model system (Pd catalysed coupling? Kinetics model?)

[dk-teknologisk-mon]

I have a lot of things I think are interesting to investigate, but I think this thread will drown if I post them separately, so I have grouped individual questions according to topic below:

Starting phase:

1. Is there a "good" fixed number, N, of initial experiments?
2. Should N depend on problem dimensionality?
3. Should N be dynamic and instead be determined by the responses spanning some threshold? E.g. if you know your scale (and you do if you have a goal), should you just do space-filling until your results span at least 20 % of your scale? Or some signal:noise threshold?
4. Is there a practical difference between naive LHS, maximin LHS and golden ratio sampling?

Replication:
5. What happens if you replicate periodically?
6. What is a "good" replication frequency if there are benefits to it?
7. Does it make sense to replicate among the initial points, or later on? E.g. does it help to include X center points among the initial points?
8. Is it better to focus replication on points of high signal? E.g. if the initial points are all essentially 0, should you then wait with replicating?

Termination:
9. Should you validate your expected minimum before stopping? If yes, the settings will change after your validation run, so "how little" a change should cause you to stop?
10. Should you validate more than once? 3 times? 5? 10?

Acquisition and model hyperparameters:
11. Comparison of acquisition functions: Naive EI (xi = 0), fudged EI (xi = |goal - best data|, or |goal - expected minimum|), UCB (varied kappa-values), maximal variance (UCB with kappa = inf), more modern acq functions like noisy EI and knowledge gradient.
12. When multi-run experimentation is favored by practical circumstances, how big should N >1 be for it to speed up your completion time? Is N = 2 enough or a waste? Will large N per batch typically result in overshooting the needed amount of runs by N (or multiple N)?
13. Our acquisition functions use ´n_restarts_optimizer = 5´, which means it starts from 5 random points. Would more points help? If yes, under what circumstances? And how many? 50? 500?
14. How much and when do noise bounds matter?
15. How much and when do length-scale bounds matter?
16. I've talked to some people who suggest changing acq. function along the way, using heuristics like "max variance (or space filling) until the 95 % CI of the max-var point no longer overlaps with the 95 % CI of the EI point". How well does this perform compared to naive EI?

System complexity and generalizability:
For many of the above questions, I personally suspect that many of the conclusions may be sensitive to problem difficulty (measured as random ERT). I think it will be important to investigate each on a range of different ERT values (e.g. 10, 100, 1000, 10000). I'm also unsure whether we should include more functions that have stronger local minima (e.g. a log-version of Branin).