Failure of CBayesianOptimisationTest/testMaximumExpectedImprovement on one machine

[droberts195]

A community member who is trying to run the C++ unit tests observed a failure that may indicate that we have some undefined behaviour somewhere in the Bayesian optimisation code.

The test was run using ./ml_test --run_test=CBayesianOptimisationTest/testMaximumExpectedImprovement in the maths library unit tests and the failure output is:

Running 1 test case...
2020-07-29 08:26:12,204855 UTC [15652] DEBUG CTestObserver.cc@23 +------------------------------------------------------------+
2020-07-29 08:26:12,204924 UTC [15652] DEBUG CTestObserver.cc@24 |  CBayesianOptimisationTest/testMaximumExpectedImprovement  |
2020-07-29 08:26:12,204936 UTC [15652] DEBUG CTestObserver.cc@25 +------------------------------------------------------------+
2020-07-29 08:26:12,651002 UTC [15652] DEBUG CBayesianOptimisationTest.cc@316 % improvement BO = 60.0783, % improvement RS = 35.4576
2020-07-29 08:26:13,092090 UTC [15652] DEBUG CBayesianOptimisationTest.cc@316 % improvement BO = 12.7077, % improvement RS = 0
2020-07-29 08:26:13,506686 UTC [15652] DEBUG CBayesianOptimisationTest.cc@316 % improvement BO = 8.06812, % improvement RS = 4.17033
2020-07-29 08:26:14,370432 UTC [15652] DEBUG CBayesianOptimisationTest.cc@316 % improvement BO = 92.8491, % improvement RS = 71.9993
2020-07-29 08:26:14,720186 UTC [15652] DEBUG CBayesianOptimisationTest.cc@316 % improvement BO = 39.7416, % improvement RS = 14.5358
2020-07-29 08:26:15,231838 UTC [15652] DEBUG CBayesianOptimisationTest.cc@316 % improvement BO = 80.4244, % improvement RS = 59.4046
2020-07-29 08:26:15,740521 UTC [15652] DEBUG CBayesianOptimisationTest.cc@316 % improvement BO = 0, % improvement RS = 24.69
CBayesianOptimisationTest.cc(318): fatal error: in "CBayesianOptimisationTest/testMaximumExpectedImprovement": critical check improvementBopt > improvementRs has failed [0 <= 0.24690042419103589]
2020-07-29 08:26:15,740848 UTC [15652] INFO  CTestObserver.cc@35 Unit test timing - CBayesianOptimisationTest/testMaximumExpectedImprovement took 3535ms

*** 1 failure is detected in the test module "lib.maths"

When I run this test locally it passes and we haven't seen it fail in our CI.

Example output from a successful run is in https://discuss.elastic.co/t/elastic-machine-learning-code-source/242847/6

The failure output is in https://discuss.elastic.co/t/elastic-machine-learning-code-source/242847/7 together with the hardware specs of the machine it was run on. Apparently no ML code was changed.

We should run this test under valgrind and also carefully check the code it covers for possible undefined behaviour in case it could cause a crash.

[edsavage]

The test was run under valgrind using the following commands

cd ml-cpp
make -j 8
cd lib/maths/unittest
make -j 8
valgrind ./ml_test --run_test=CBayesianOptimisationTest/testMaximumExpectedImprovement

The test completed successfully with no errors reported from valgrind.

[tveasey]

BO test is sensitive to platform dependent variation in numerics. That said I made a number of improvements to the robustness since this issue was reported, for example #2144 and #2147. Since this test is not failing on any of our target platforms, I don't think it makes sense to keep it open any longer.