Prevent division by zero in GPR when y_train is constant

[afonari]

This is merged PR of two PRs: #18388, #19361.
This fixes: #18318.

[ogrisel]

Also please fix the linting problem reported by the CI and expand the test by testing with multi-target data: a Y matrix where one column is a constant 2. for instance and the other is random normal data:

n_samples = X.shape[0]
rng = np.random.RandomState(0)
Y = np.concatenate([
    rng.normal(size=(n_samples, 1)),  # non-constant target
    np.full(shape=(n_samples, 1), fill_value=2)  # constant target
], axis=1)

[ogrisel]

Thanks for improving the tests. LGTM. Just a few more suggestions below.

I am no GPR specialist so I would appreciate it if others (e.g. @jaburke166 @boricles, @sobkevich, @jmetzen, @plgreenLIRU) could have a look.

[afonari]

Added a commented test as discussed.

[ogrisel]

LGTM.

[afonari]

Can this be merged?

[cmarmo]

Last detail, then LGTM.

[sobkevich]

Thanks for improving the tests. LGTM. Just a few more suggestions below.

I am no GPR specialist so I would appreciate it if others (e.g. @jaburke166 @boricles, @sobkevich, @jmetzen, @plgreenLIRU) could have a look.

Hi,
with the current approach I had problems with test test_predict_cov_vs_std(kernel) if constant y is added to it.
If this test is run for constant y there will be:

(test_predict_cov_vs_std[kernel1-y1])
kernel = RBF(length_scale=1), y = array([1., 1., 1., 1., 1., 1.])

    @pytest.mark.parametrize('kernel,y',
                             list(product(kernels, [y, y_with_zero_std])))
    def test_predict_cov_vs_std(kernel, y):
        if sys.maxsize <= 2 ** 32 and sys.version_info[:2] == (3, 6):
            pytest.xfail("This test may fail on 32bit Py3.6")
    
        # Test that predicted std.-dev. is consistent with cov's diagonal.
        gpr = GaussianProcessRegressor(kernel=kernel).fit(X, y)
        y_mean, y_cov = gpr.predict(X2, return_cov=True)
        y_mean, y_std = gpr.predict(X2, return_std=True)
>       assert_almost_equal(np.sqrt(np.diag(y_cov)), y_std)
E       AssertionError: 
E       Arrays are not almost equal to 7 decimals
E       
E       Mismatch: 100%
E       Max absolute difference: 0.00182264
E       Max relative difference: inf
E        x: array([6.5100511e-06, 4.4185900e-06, 4.1690509e-06, 4.8057573e-06,
E              5.8756981e-06])
E        y: array([1.5440852e-03, 1.8270599e-03, 0.0000000e+00, 1.0005933e-05,
E              1.4597007e-03])

But I really don't know if it is important to save this equality for case where y is constant

[afonari]

with the current approach I had problems with test test_predict_cov_vs_std(kernel) if constant y is added to it.

Note that for the fixed kernel: RBF(length_scale=1.0, length_scale_bounds="fixed") it passes. I also tried settings length_scale_bounds="fixed" to other kernels from the kernels list and those pass too. So maybe it is expected to fail for variable length kernels.

[sobkevich]

with the current approach I had problems with test test_predict_cov_vs_std(kernel) if constant y is added to it.

Note that for the fixed kernel: RBF(length_scale=1.0, length_scale_bounds="fixed") it passes. I also tried settings length_scale_bounds="fixed" to other kernels from the kernels list and those pass too. So maybe it is expected to fail foar variable length?

Maybe)

[afonari]

Anything to be done here?

[afonari]

Ping.

[glemaitre]

I merge main in the branch. LGTM. I just move the code of the test regarding multitarget in the related PR.

[afonari]

Thanks a lot everyone!

[glemaitre]

The bug regarding the covariance and standard deviation is solved here: #19939