Fix TSEMO

[marcosfelt]

This is a PR for tracking improvements to TSEMO (issue #2). It is a work in progress.

I've made one change to the hypervolume reference, but I think it's still not the same as the Bradford paper.

• Test functions (DTLZ2, vmlop)
• Correct reference based on Bradford et al.
• Set up spectral sampling
• Do some ablation experiments on fixed vs ideal-anti-ideal difference and random rate.

[marcosfelt]

I've discovered a lot of subtle things about TSEMO. I want to note them here:

• For the training of the GPs (i.e., hyperparameter optimisation), at least 100 optimisation restarts are needed to consistently find the optimal log likelihood, especially with small data. You also need to constrain the hyperparameters and add priors so they won't train to extremely large or small values early on.
• Sepctral sampling is very temperamental and slow (it scales with m^1/2). It will randomly fail due to problems with the singular value decomposition (SVD). Therefore, I have to implement "retries" in the code to restart the spectral sampling in these cases. Also, you need at least 1500 sampling points to get good results and 4000 preferably. For the code itself, I went with pyrff instead of my own code, but it requires saving down the sampled functions to disk and then copying them back up.
• You need a good implementation of NSGA-II. I was working with platypus for a while but then realized it was not performing well and switched to pymoo.

[marcosfelt]

I'm done with TSEMO now. I don't think the tests will pass because one of the Gryffin tests seems to have a problem. However, otherwise, things should be working now.

[jangerit]

Great! Being able to use TSEMO in python now is really cool.

About Gryffin: for the test functions we implemented it seems like Gryffin only gets close the optimal points but does not find those exactly. That's why the tests sometimes fail. Also, the Gryffin tests take quite long. Maybe we should comment them out in test_strategies.py or rather just look whether the code runs without errors but exclude the full optimization runs (this would be comment out lines: 597, 601, 673, 677 in test_strategies.py).`

[marcosfelt]

I think it's probably best to do the latter...maybe we can just run a few iterations of TSEMO and Gryffin to see if the best function values are improving. We'll have to rely on running tests elsewhere to confirm they perform correctly on benchmarks.

…

On Wed, 15 Jul 2020 at 00:13, Jan Rittig ***@***.***> wrote: Great! Being able to use TSEMO in python now is really cool. About Gryffin: for the test functions we implemented it seems like Gryffin only gets close the optimal points but does not find those exactly. That's why the tests sometimes fail. Also, the Gryffin tests take quite long. Maybe we should comment them out in test_strategies.py or rather just look whether the code runs without errors but exclude the full optimization runs (this would be comment out lines: 597, 601, 673, 677). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGF3ORZCUZYFYI7NOMXVMVTR3TRDBANCNFSM4M6BKMKQ> .

[jangerit]

maybe we can just run a few iterations of TSEMO and Gryffin to see if the best function values are improving

That's a good idea! We should also do this for SOBO, I think. This should reduce the time for the pytest runs a lot.