Sergey Lisitsyn edited this page Feb 21, 2015 · 17 revisions
Clone this wiki locally

Hip Deep learning

Taking last year's project to a new level.


Difficulty & Requirements

Pretty difficult. Requires:

  • C++ programming
  • Understanding what neural nets are and how do we learn them
  • OpenCL programming is a plus
  • Understanding Of inference in graphical models, variational inference is a plus
  • Experience with Stochastic Optimization is a plus


Last year we had an introductory project completed by Khaled Nasr on deep learning which introduced most essential components to use neural networks with Shogun. The content of that project was to implement most common feedforward architectures, autoencoders and boltzmann machines. This year we wish to devote more time to deep directed generative models with a Bayesian treatment of parameter learning. Generative models are immensely powerful for many tasks and using deep networks to parametrize exponential family distributions in the form of sigmoid belief networks is an old idea that recently has received a lot of attention with astounding results. We aim to include the newest results on that field in Shogun to facilitate many applied settings. The project will include implementation of variational inference machinery for graphical models with factors given by MLPs, optimization methods and applications. The first part is heavy on mathematics and will require good understanding of probability theory and graphical models. The second part consists of including various stochastic optimizers into shogun, as just stochastic gradient decent is not enough nowadays to train complex models well. The third part will consist of delivering convincing demonstrations of speed, performance and scalability of the implemented models in a series of applications. Given time, we will also offer the chance to explore novel developments in this field.


Waypoints and initial work

  • First of all, you'd have to dive into the existing goods. We believe the easiest way to do that is to improve the code through small iterative changes that improve code quality
  • Training neural nets with CPU only is really slow and we're actually pretty near to have GPU training in Shogun so the next step is to finish up GPU training and finally train some deep net with a GPU


  • We expect this project to consider active research problems so probably you can even come up with something really-really new!

Why this is cool

Deep learning is everywhere - it is on the bleeding edge of research. Most importantly, in recent attempts deep learning is getting the benefit of a bayesian treatment and the goal of this project is to marry deep nets and variational inference in the spirit of Reverend Bayes.


Useful resources

Entrance level tasks