GSoC_2016_project_large_gps

Wu Lin edited this page Jul 21, 2016 · 61 revisions
Clone this wiki locally

Large-Scale Gaussian Processes

Polish, update, and extend Shogun's framework for Gaussian Processes, with a focus on large-scale problems and sparse variational inference.

Mentors

Difficulty & Requirements

Medium

You need know:

Description

Following our previous successful project on variational learning for Big Data, we attempt to bring Shogun's Gaussian Processes (GP) to Big Data land. From a high level perspective, this means that the goal is to implement established methodology on how to scale up GPs to be able to process hundreds of thousands of points.

Details

We want to focus on the following sub-tasks:

  • Variational inference for (full) GP
  • Variational inference for sparse GP
  • Stochastic variational inference for sparse GP
  • Applications

The project will start from the existing code base, which already contains a huge amount of work. We aim to fill the gaps in the above methods:

  • Implement all standard methods that are not yet in the framework
  • Benchmark existing methods against competing implementations (such as GPStuff, GPflow, GPML, etc)
  • Improve efficiency if necessary. Make explicit use of multi-core computation.
  • Unify/Refactor gradient computations within Shogun (Note that the new framework should be easy to be extended to using AutoDiff)

Waypoints and initial work

  • Understanding the inference procedure and refactoring existing framework with the help from mentors
  • Make beautiful demos and benchmarks under the new framework
  • Stochastic variational inference for GPs (optional)

Refactoring existing framework

Variational Gaussian inference (Suggested Roadmap)

  • base class for computing gradient of Evidence Lower BOund (ELBO) wrt variaitonal variables
  • base class for computing gradient of ELBO wrt hyper-parameters in likelihoods, mean functions, and co-variance/kernel functions
  • (for full GP) classes for computing gradient wrt variaitonal variables using existing hand-implemented codes
  • (for full GP) classes for computing gradient wrt hyper-parameters using existing hand-implemented codes
  • Benchmarks and notebooks for demos
  • base class for MC samplers
  • classes for using existing MC samplers
  • (for sparse GP) classes for computing gradient wrt variaitonal variables using existing hand-implemented codes
  • (for sparse GP) classes for computing gradient wrt hyper-parameters using existing hand-implemented codes
  • classes for HMC samplers from Stan (optional)
  • base class for model selection (eg, Bayesian OPT) (optional)

Optional

  • MCMC inference
  • Deep GP

Why this is cool

Our primary goal is to scale up GPs to use them in "big data" applications. GPs are becoming more and more popular for big data because they provide both accurate predictions, tell us how confident we should be about our prediction (aka uncertainty quantification) and also whether we have selected the right model (aka model selection). With big data comes big noise, so these issues are becoming ever more relevant. Recently GPs are a popular choice beyond regression and classification for a wide range of appliations from numerical optimization to recommendation system and even to deep networks. The main bottleneck in these applications is scalability. Therefore, we want you to write easy-to-use, scalable code which will help the machine learning community to use GPs.

Useful ressources

Get back to the main projects page.