GP bug for noise_learn = false underprediction of uncertainty

[odunbar]

Purpose

Fix the noise_learn = false bug greatly underpredicting uncertainty. In Lorenz example, the following two should be similar. with noise_learn=false and true:

co-author with @lm2612

In this PR

• Initial bugfix to correct the regularization noise vs white kernel
• Tightened runtests thanks to improved prediction,
• new alg_reg_noise optional argument to GP to set the regularization when noise_learn=true (removing magic_number)
• Lorenz example - with SKLJL() option
  

[codecov]

Codecov Report

Merging #143 (48598e3) into master (ad38cb5) will increase coverage by 0.55%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #143      +/-   ##
==========================================
+ Coverage   88.15%   88.71%   +0.55%     
==========================================
  Files           4        4              
  Lines         380      381       +1     
==========================================
+ Hits          335      338       +3     
+ Misses         45       43       -2     

Impacted Files	Coverage Δ
src/GaussianProcess.jl	93.00% <100.00%> (+2.09%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad38cb5...48598e3. Read the comment docs.

[odunbar]

bors try

[bors]

try

Build succeeded:

• test-linux
• docs-build
• test-macos
• test-windows
• buildkite/calibrateemulatesample-ci

[lm2612]

Looks good with SKLJL.
When I test with GPJL and noise_learn=false, I'm finding the predicted covariance matrix y_var to now be around 2x Γy. Is it possible that this was already included for GPJL?

[odunbar]

The predicted covariance (when returned with transform_to_real=true) will be the sum of 2 contributions

1. one from the GP kernel approximation (e.g. the fact that we use an RBF to approximate something)
2. one from the observational noise.

So in theory the y_var must be >= observational noise. The amount that it is bigger will be due to the error in the GP approximation.

does it return something similar with noise_learn = true? This will fix contribution 2. = observational noise, and only learns contribution 1. If they are similar, this would indicate that it's just not a great approximation if they are not similar then maybe there is an issue (note they are unlikely to be exactly the same).

I did try to do something in the unit tests like this - (see the added file)

[odunbar]

Ignore what i said: from the GaussianProcess.jl source code I found this:
thus they add the "lognoise" (i.e. the regularization noise) back in during prediction

function predict_y(gp::GPE, x::AbstractMatrix; full_cov::Bool=false)
    μ, σ2 = predict_f(gp, x; full_cov=full_cov)
    if full_cov
        npred = size(x, 2)
        return μ, σ2 + ScalMat(npred, exp(2*gp.logNoise))
    else
        return μ, σ2 .+ exp(2*gp.logNoise)
    end
end

Thanks! I will update accordingly

[lm2612]

Ok great! I forgot to say in my previous comment I was referring to the Lorenz example

[odunbar]

OK

• I've moved the magic_number to be an input argument for GP.
• I've tightened up the new unit tests, so now predicted means, and variances, with noise_learn and without have to be within 5% of each other (was 50% before - hence why it missed the bug!)

[lm2612]

Great, I get virtually the same results for both GPJL and SKLJL now, thanks!

[odunbar]

bors r+

[bors]

Build succeeded:

• buildkite/calibrateemulatesample-ci
• docs-build
• test-linux
• test-macos
• test-windows