DeepLearning-Resources

An awesome list of deep learning resources

Table of Contents

• Tutorials / Courses

• Talks / Videos

• Books

• Recent Papers

• Recent Interesting Blog Posts

• Papers

• Fundamentals

• Practical Tricks

• Autoencoders

• Convolutional Neural Networks

• Recurrent Neural Networks

• Image Classification

• Visualization and Inversion

• Object Detection

• Semantic Segmentation

• Optimization

• Regularization

• Reinforcement Learning

• Architectures and other Applications

• Parallelizing Deep Networks

• Neural Program Synthesis and Language Grounding

• Resources and References

Tutorials and Courses

• Generative Adversarial Networks by Ian Goodfellow, NIPS 2016 Tutorial.
• Deep Learning for Computer Vision by Rob Fergus, NIPS 2013 tutorial.
• Convolutional Neural Networks for Visual Recognition by Fei-Fei Li, Andrej Karpathy and Justin Johnson, 2016. [Stanford]
• Deep Learning for Natural Language Processing by Richard Socher. [Stanford]
• Neural Networks for Machine Learning by Geoffrey Hinton, 2012. [Coursera]
• Lecture notes from Geoffrey Hinton's classes on neural networks 2013.
• Deep Learning Course by Nando de Freitas, 2015. [Oxford]
• Deep Learning Course [CILVR lab, NYU]
• Deep Learning by Vincent Vanhoucke and Arpan Chakraborty, 2016. [Udacity/Google]
• A Deep Learning Tutorial: From Perceptrons to Deep Networks by Ivan Vasilev.

Talks and Videos

• The Next Generation of Neural Networks by Geoffrey Hinton at GoogleTechTalks 2007.
• Recent Developments in Deep Learning by Geoffrey Hinton, UBC 2013.
• The Unreasonable Effectiveness of Deep Learning by Yann LeCun, Johns Hopkins University 2014.
• Deep Learning of Representations by Yoshua Bengio at GoogleTechTalks 2012.
• A Beginners Guide to Deep Neural Networks by Natalie Hammel and Lorraine Yurshansky, Google Research Blog 2015.
• Deep Learning RNNaissance with Dr. Juergen Schmidhuber at NYC ML Meetup 2014.
• Deep Learning: Theoretical Motivations by Yoshua Bengio, DLSS 2015.
• Deep Learning for Vision: Tricks of the trade Facebook, Bay Area Vision Meeting 2013.
• Deep Learning, Self-Taught Learning and Unsupervised Feature Learning by Andrew Ng, Graduate Summer School: Deep Learning, Feature Learning 2013.

Books

• Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
• Deep Learning survey in Nature by Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.

Recent Papers

2016-12

* [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://arxiv.org/pdf/1612.01887v1.pdf) J Lu, C Xiong, D Parikh, R Socher - arXiv preprint arXiv:1612.01887, 2016

Recent Interesting Blog Posts

• Yes You Should Understand Backprop by Andrej Karpathy
• New Perspectives on Adversarial Training (NIPS 2016 workshop) by Ferenc Huszár
• The major advancements in Deep Learning in 2016
• Demystifying Deep Reinforcement Learning by Tambet Matiisen
• Attention and Memory in Deep Learning and NLP
• Attention and Augmented Recurrent Neural Networks by Chris Olah and Shan Carter
• Why Randomness is Important for Deep Learning by Eric Jang
• Introduction to Neural Machine Translation with GPUs by Kyunghyun Cho
• An overview of gradient descent optimization algorithms by Sebastian Ruder

Programming

Tensorflow

• Deep Learning and Tensorflow by Hvass laboratories
• Sirajology by Siraj Raval

Papers

Fundamentals

• Efficient BackProp, LeCun YA, Bottou L, Orr GB, Müller KR. InNeural networks: Tricks of the trade 2012.
• Learning representations by back-propagating errors, The original article on back-propagation, DE Rumelhart, GE Hinton, RJ Williams - Cognitive modeling, 1988.
• Minimizing description length in an unsupervised neural network, Hinton GE, Zemel RS. Minimizing description length in an unsupervised neural network. Preprint. 1997.
• Dimensionality Reduction by Learning an Invariant Mapping, Hadsell R, Chopra S, LeCun Y, CVPR 2006.

Practical Tricks

• On the Importance of Initialization and Momentum in Deep Learning, Sutskever I, Martens J, Dahl G, Hinton G, ICML 2013.
• Understanding the Difficulty of Training Deep Feedforward Neural Networks, Glorot X, Bengio Y, In International conference on artificial intelligence and statistics 2010.
• Practical Recommendations for Gradient-Based Training of Deep Architectures, Y Bengio - Neural Networks: Tricks of the Trade, 2012 - Springer.
• Improving Neural Networks by Preventing Co-adaptation of Feature Detectors, Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR, arXiv preprint arXiv:1207.0580, 2012.
• Learning Deep Architectures for AI, Y Bengio - Foundations and trends in Machine Learning, 2009.

Autoencoders

• Reducing the dimensionality of data with neural networks, GE Hinton, RR Salakhutdinov - Science, 2006.
• Disentangling factors of variation for facial expression recognition, S Rifai, Y Bengio, A Courville, P Vincent, M Mirza, ECCV 2012.
• Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA, JMLR 2010.

Convolutional Neural Networks

• Gradient-Based Learning Applied to Document Recognition, LeCun Y, Bottou L, Bengio Y, Haffner P, Proceedings of the IEEE. 1998.
• Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis, Simard PY, Steinkraus D, Platt JC, ICDAR 2003.
• Network in Network, M Lin, Q Chen, S Yan - arXiv preprint arXiv:1312.4400, 2013.

Recurrent Neural Networks

• Training Recurrent Neural Networks, Ilya Sutskever, PhD Thesis, 2012.
• Supervised Sequence Labelling with Recurrent Neural Networks, Textbook, Studies in Computational Intelligence, Springer, 2012.
• Training and Analyzing Deep Recurrent Neural Networks, M Hermans, B Schrauwen, NIPS 2013.
• Generating Sequences With Recurrent Neural Networks, A Graves - arXiv preprint arXiv:1308.0850, 2013.

Image Classification

• ImageNet Classification with Deep Convolutional Neural Networks, Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, NIPS 2012.
• Very Deep Convolutional Networks for Large-Scale Image Recognition, K Simonyan, A Zisserman - arXiv preprint arXiv:1409.1556, 2014.
• Going Deeper with Convolutions, Szegedy, Christian and Liu, Wei and Jia, Yangqing and Sermanet, Pierre and Reed, Scott and Anguelov, Dragomir and Erhan, Dumitru and Vanhoucke, Vincent and Rabinovich, Andrew, CVPR 2015.
• Deep Residual Learning for Image Recognition, K He, X Zhang, S Ren, J Sun - arXiv preprint arXiv:1512.03385, 2015.
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian, ICCV 2015.

Visualization and Inversion

• Visualizing and Understanding Convolutional Networks, MD Zeiler, R Fergus, ECCV 2014.
• Understanding Deep Image Representations by Inverting Them, A Mahendran, A Vedaldi, CVPR 2015.
• Visualizing and Understanding Recurrent Networks, A Karpathy, J Johnson, FF Li - arXiv preprint arXiv:1506.02078, 2015.

Object Detection

• Fast R-CNN, R Girshick, ICCV 2015.
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, S Ren, K He, R Girshick, J Sun, NIPS 2015.
• Region-based Convolutional Networks for Accurate Object Detection and Segmentation, R Girshick, J Donahue, T Darrell, J Malik, PAMI 2016.

Semantic Segmentation

• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs, Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL, arXiv preprint arXiv:1412.7062. 2014.
• Fully Convolutional Networks for Semantic Segmentation, J Long, E Shelhamer, T Darrell, CVPR 2015.
• Semantic Image Segmentation via Deep Parsing Network, Z Liu, X Li, P Luo, CC Loy, X Tang, ICCV 2015.
• Conditional Random Fields as Recurrent Neural Networks, Zheng S, Jayasumana S, Romera-Paredes B, Vineet V, Su Z, Du D, Huang C, Torr PH, ICCV 2015.

Optimization

• Deep learning via Hessian-free optimization, Martens J, ICML 2010.
• On optimization Methods for Deep Learning, Ngiam J, Coates A, Lahiri A, Prochnow B, Le QV, Ng AY, ICML 2011.
• Adam: A Method for Stochastic Optimization, D Kingma, J Ba - arXiv preprint arXiv:1412.6980, 2014.
• On the saddle point problem for non-convex optimization, R Pascanu, YN Dauphin, S Ganguli, Y Bengio - arXiv preprint arXiv:1405.4604, 2014
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, Dauphin YN, Pascanu R, Gulcehre C, Cho K, Ganguli S, Bengio Y, NIPS 2014.
• The Loss Surfaces of Multilayer Networks, Choromanska A, Henaff M, Mathieu M, Arous GB, LeCun Y, arXiv preprint arXiv:1412.0233. 2014.
• Gradient-based Hyperparameter Optimization through Reversible Learning, D Maclaurin, D Duvenaud, RP Adams - arXiv preprint arXiv:1502.03492, 2015.

Regularization

• Rectified Linear Units Improve Restricted Boltzmann Machines, Nair V, Hinton GE, ICML 2010.
• Maxout Networks, Goodfellow IJ, Warde-Farley D, Mirza M, Courville A, Bengio Y, arXiv preprint arXiv:1302.4389. 2013.
• Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R, JMLR 2014.
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, S Ioffe, C Szegedy - arXiv preprint arXiv:1502.03167, 2015.

Reinforcement Learning

• Human-level control through deep reinforcement learning, Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Nature 2015.
• Playing Atari with Deep Reinforcement Learning, Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M, arXiv preprint arXiv:1312.5602. 2013.

Architectures and other Applications

• Neural network Zoo, Fjodor van Veen, Asimov Institute
• DeepFace: Closing the Gap to Human-Level Performance in Face Verification, Taigman Y, Yang M, Ranzato MA, Wolf L, CVPR 2014.
• Large Scale Distributed Deep Networks, Dean J, Corrado G, Monga R, Chen K, Devin M, Mao M, Senior A, Tucker P, Yang K, Le QV, Ng AY, NIPS 2012.
• Predicting Parameters in Deep Learning, Denil M, Shakibi B, Dinh L, de Freitas N, NIPS 2013
• Distilling the Knowledge in a Neural Network, G Hinton, O Vinyals, J Dean, arXiv preprint arXiv:1503.02531, 2015.
• Spatial Transformer Networks, Jaderberg M, Simonyan K, Zisserman A, NIPS 2015.
• Generative Adversarial Networks, Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y, NIPS 2014.
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, A Radford, L Metz, S Chintala, arXiv preprint arXiv:1511.06434, 2015.
• Semi-Supervised Learning with Ladder Networks, A Rasmus, M Berglund, M Honkala, H Valpola, T Raiko, NIPS 2015.
• Highway Networks, Srivastava RK, Greff K, Schmidhuber J, arXiv preprint arXiv:1505.00387, 2015.
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition, He K, Zhang X, Ren S, Sun J, PAMI 2015.

Parallelizing Deep Networks

• One weird trick for parallelizing convolutional neural networks, A Krizhevsky - arXiv preprint arXiv:1404.5997, 2014.
• Large Scale Distributed Deep Networks, Dean, Jeffrey, et al. Advances in neural information processing systems. 2012.
• Coarse Grain Parallelization of Deep Neural Networks, Tallada, M. G. In Proceedings of the 21st ACM SIGPLAN 2016.

Neural Program Synthesis and Language Grounding

• DEEPCODER: LEARNING TO WRITE PROGRAMS
• Program Synthesis using Natural Language
• Programming with a Differentiable Forth Interpreter
• Unsupervised Learning by Program Synthesis
• Differentiable Programs with Neural Libraries
• The symbol grounding problem
• Language to Logical Form with Neural Attention
• Deep Neural Networks with Massive Learned Knowledge
• Harnessing Deep Neural Networks with Logic Rules
• End-To-End Memory Networks
• Reinforcement Learning for Mapping Instructions to Actions
• Parsing with Compositional Vector Grammars

Resources and References

• Deep Learning Summer School, Montreal 2015.
• LISA reading List
• Deep learning resources
• Awesome Deep Vision - a curated list of deep learning resources for computer vision.
• Awesome Deep Learning
• Language Grounding to Vision and Control