Roadmap for JuliaGaussianProcesses

[theogf]

As proposed in Slack we should establish a concrete plan on how the JuliaGaussianProcesses ecosystem should look like.
I think having discussions in a third-party repo (the general website) is probable the most convenient?

Here is a quick list with things coming out of my mind:

• There is a strong discussion to keep AbstractGPs an interface package, and move a lot of the concrete implementation to another package
• Maybe the most important point: we need a stronger documentation with a lot of concrete examples and usecases
• KernelFunctions is slowly reaching maturity and should get to version 1.0 soon
• @willtebbutt already moved Stheno and TemporalGPs to JuliaGaussianProcesses, I will try to do the same with AugmentedGaussianProcesses at some point.
• As mentionned here Bayesian Optimization Example Stheno.jl#126 or here Bayesian Optimisation theogf/AugmentedGaussianProcesses.jl#88, Bayesian Optimization is quite requested maybe we should have our own implementation or connect it better with BayesianOptimization.jl
• GPLikelihoods is being left alone and sad, we should probably finish some more concrete implementation
• Going with the previous point we need to bring some algorithms such as EP or VI for inference
• We need a central point where all this is explained, and that is the original objective of this repo/website
• I need to finish writing InducingPoints.jl

Please don't hesitate to add things on your mind :)

[theogf]

Some additional points more detailed in JuliaGaussianProcesses/AbstractGPs.jl#104
AbstractGPs was meant as an interface package, eventually hosting basic a basic GP version.
However it slowly became more than that and more dramatically there is no clear documentation on what the interface should be.

[willtebbutt]

Quick remark before I forget: we should consider factoring out the ColVecs, RowVecs, MOInput stuff etc into a separate package. It doesn't really belong in KernelFunctions.jl -- for example, I've got multi-output things that I want to do in Stheno that don't need to know anything about the multi-output kernels that live in KernelFunctions.jl, but they do want to have access to the types that we use to talk about the inputs to multi-output GPs.

[theogf]

Related issue JuliaGaussianProcesses/KernelFunctions.jl#394

[willtebbutt]

As mentionned here JuliaGaussianProcesses/Stheno.jl#126 or here theogf/AugmentedGaussianProcesses.jl#88, Bayesian Optimization is quite requested maybe we should have our own implementation or connect it better with BayesianOptimization.jl

The BayesOpt question is an interesting one. While it's generally the case that people use GPs as the model for the objective function when doing BayesOpt, it's not that this has to be the case. I think the kind of solution that I'd like to see would be one that is model-agnostic in principle, but which supports the AbstractGPs API well in practice. i.e. we could just slot a GP in and expect it to work, but you could also use a DeepGP or GPAR or something else.

[willtebbutt]

But overall I think that the most important points are potentially:

1. Formalising the AbstractGPs API -- I think that there might actually be two APIs lurking in there, or a couple of different levels at which its helpful to implement the API, because e.g. Stheno.jl and TemporalGPs.jl implement it in different ways (TemporalGPs.jl doesn't implement cov, rather it implements logpdf, rand, and marginals etc directly). Maybe we want a distinction between the user-facing API and the developer-facing API or something?
2. Approximate inference algorithms for non-Gaussian likelihoods + associated API. Also recommendations as to which parts of the AbstractGPs API you can expect different packages to support. For example, if you implement an approximate inference algorithm and expect it to work with TemporalGPs.jl, you don't ever get to call cov.
3. Examples and documentation. We really need to stress-test the API and the models before we try and tag version 1 or KernelFunctions and AbstractGPs etc, and we need to explain the different.

[willtebbutt]

@yebai @st-- @theogf @rossviljoen this feels like an appropriate place to assemble the feature table we discussed in today's meeting.

Approximation techniques.

Disclaimer: this table doesn't distinguish between different ways of parametrising the different approximations, or the various different ways you can try to optimise the variational approximation (for example). This table represents the first iteration of what will probably be an on-going effort to characterise the relationship between different packages.

Disclaimer 2: I'm fairly uncertain about the capabilities of R-INLA. I've included it because my understanding is that it's incredibly popular with people who do spatial statistics, and it has interesting capabilities that are somewhat orthogonal to what most ML packages offer.

Disclaimer 3: I've left some question marks where I'm not entirely sure about something.

Non-Gaussian Likelihoods

Approximation Technique	JuliaGPs	GaussianProcesses.jl	GPML	GPFlow	GPyTorch	R-INLA
Exact (Cholesky)	✅	✅	✅	✅	✅	✅
Laplace	✅	❌	✅	❌	❌	✅
INLA	❌	❌	❌	❌	❌	✅
Variational Bayes	✅	✅	✅	✅	✅	❌
Expectation Propagation	❌	❌	✅	❌	❌	❌
Power EP	❌	❌	✅	❌	❌	❌
HMC	✅	✅	✅	✅	❌	❌
ESS	✅	✅	❌ (?)	❌ (?)	❌	❌

Scaling GPs

Approximation Technique	JuliaGPs	GaussianProcesses.jl	GPML	GPFlow	GPyTorch	R-INLA
Exact (Cholesky)	✅	✅	✅	✅	✅	?
Conjugate Gradients etc	❌	❌	❌	✅	✅	❌
VFE (collapsed)	✅	❌	✅	✅	❌	❌
FITC	❌	✅	✅	❌	❌	❌
Sparse PEP	❌	❌	✅	❌	❌	❌
VFE (Hensman-style)	✅	❌	❌	✅	✅	❌
State-space	✅	❌	✅	❌	❌	❌
SPDE (Lindgren 2011)	❌	❌	❌	❌	❌	✅
SKI	❌	❌	✅	❌	✅	❌

Note that we ought to distinguish between using natural gradients with VFE, but this table doesn't.
I've left off a number of sparse approximations.

Updating this table

Please update this directly if you want to add stuff / correct stuff.

[yebai]

thanks, @willtebbutt - this is really helpful!

[yebai]

Any reason for not including GPy here?

[willtebbutt]

No, I just forgot. Same with GPStuff. Someone should add them.

[devmotion]

I removed the question mark for JuliaGPs + ESS since EllipticalSliceSampling is used successfully in the AbstractGPs tutorial. Please add it back (or probably better add it as a comment) if anything is missing there and the integration has to be improved.