VISH

[drsaadhamid]

No description provided.

[vdutor]

Could you write down in math what the gradients look like? That said, I don't think we need the gradients of funk hecke wrt the lengthscale and variance. For the lengthscale you can simply multiply the inputs X before evaluating the spherical harmonics and the variance can be accounted for by multiplying the eigenvalues with it.

[drsaadhamid]

The gradient wrt the variance is just the funk hecke integral.
The gradient wrt the lengthscale is the funk hecke integral with the shape function replaced by the derivative of the shape function wrt to the lengthscale, i.e. v \int d/dl s(t, l) f(t) dt, where f(t) is the product of the Gegenbauer polynomial and the weighting function wrt which the Gegenbauers are orthogonal and v is the variance.

[vdutor]

Why do you need this functionality. Minor: maybe a more descriptive name would be clear_eigenvalue_cache_onchange?

[drsaadhamid]

To avoid having to recompute funk-hecke integrals. A call to the eigenvalues method is made every time the Kuu matrix is required, which means every time a prediction is required. If we didn't cache the eigenvalues then we will do a lot of redundant computation whilst optimising the acquisition function, since we need to recompute the posterior variance at every step.

[vdutor]

why would you want to do that?

[drsaadhamid]

So that the posterior variance at the data locations goes down to the likelihood variance. This is necessary for the acquisition function to work properly. The variance of the unwarped GP is combination of the posterior mean and posterior variance of the warped GP. If the posterior variance doesn't get sufficiently small, the acquisition function will keep selecting the peak of the mean function. (You can see this for yourself if you run the experiment but change line 38 of the config file vish.yaml from "spherical_matern" to "chord_matern".)

I appreciate that this issue is specific to using VISH for PI, so maybe you don't want it changed in this repo? If so, we could fork it, or just work in a different branch? We could also just keep using the SphericalMatern kernel, which is now working fine.

[vdutor]

minor: I would place this in a utils.py.

[vdutor]

minor: rename file to covariances.py?

[vdutor]

Are you still using this? If not, best to remove the file from the PR.

[drsaadhamid]

Not using it anymore; will remove.

[vdutor]

I would refactor some of this code and have it as an integration test too in tests.

[drsaadhamid]

Will do

[drsaadhamid]

I added a model for SGPR in vish/__init__.py. You were right that simply setting the optimal variational parameters for the SVGP model wasn't a good idea because the matrix that needs to be inverted is not well conditioned. "Rotating" (more of a scaling since Kuu is diagonal) by the cholesky decomposition of Kuu does help in this regard.

I've also added a script to compare the posterior variances at data points between a TruncatedChordMatern kernel and a ChordMatern kernel. The difference between the two is that the prior variance is computed using a truncated mercer decomposition for the TruncatedChordMatern, and using the ordinary Matern equations for the ChordMatern. When the likelihood variance is allowed to be optimised the posterior variance at data points is below the likelihood variance for all kernels. (I've added a pytest that ensures this in tests/test_vish_gpr.py.) However, we need to keep the likelihood variance small if we want to use the VISH model with the uncertainty sampling acquisition function or we will keep making acquisitions at the same location. If we fix it to 0.01 or below (in practice we'll want it quite small, say 1e-4), then notice that the posterior variance at data points stays below the likelihood variance for the TruncatedChordMatern, but not for the ChordMatern.