Welcome to Deep Learning Scratch Arena.

What is Deep Learning Scratch Arena?

My goal is to provide high quality Scratch Implementations of the fundamentals of Deep Learning and its applications, with interactive well documentated jupyter notebooks. All notebooks come along with implementations using Tensorflow, MXNet and Pytorch.

Why Scratch Implemantation?

• It helps us to understand the core working of an algorithm better.
• We could try to improve an existing algorithm both for accuracy and efficiency.
• We could implement our own version of an existing algorithm with some new features via experiments.
• We could even invent new algorithms that could outperform previous algorithms achieving SOTA(state of the art) results.
• Thus we could actually contribute to the field.

Watch https://youtu.be/l9RWTMNnvi4?t=130, The God Father of AI Geoffrey Hinton said "If you want to understand a really complicated device like brain, then you should build one".

Same goes for the Deep Learning technologies, if you really (like reeaally) want to understand such technologies then you should build them. Indeed Keras exists, but these kind of frameworks which takes not more than 5 lines to build a model actually hides everything from you, that makes you completely unknown what's going on under the hood. The purpose of such frameworks is rapid deployment and production, they are not meant to be used by people who are beginner in the field. If you already understand the technology, then be my guest and use such frameworks like a champ, otherwise if you are a beginner in the field......... you better listen to The God Father.

List of topics implemented from scratch.

Intro to Deep Learning.

• Understanding Percepron - Linear Regression
• Binary Classification - Logistic Regression
• Multiclass Classification - Softmax Regression
• The Modern Deep Learning - Neural Networks

Activation Functions.

• Sigmoid
• Tanh
• ArcTan
• Softsign
• ISRU
• ISRLU
• PLU
• ReLU
• SoftPlus
• Leaky ReLU
• PReLU
• RReLU
• ELU
• SELU
• Swish
• Bent Identity

Gradient Descent Variants.

• Momentum Optimizer
• Nesterov Accelerated Gradient Optimizer
• AdaGrad Optimizer
• AdaDelta Optimizer
• RMSProp Optimizer
• Adam Optimizer
• AMSGrad Optimizer
• Nadam Optimizer
• AdaMax Optimizer

Learning Rate Decay Methods.

• Time Decay
• Step Decay
• Exponential Decay
• Cyclic Decay

Preventing Overfitting.

• L1 Regularization
• L2 Regularization
• Dropout
• DropConnect

Making Deep Learning Learn Faster.

• Batch Normalization
• Layer Normalization

Recurrent Neural Networks.

• Recurrent Neural Networks (RNN)
• Long Short Term Memory Cell (LSTM)
• Gated Recurrent Unit (GRU)
• Sequence to Sequence
• Multilayer Recurrent Neural Networks
• Bidirectional Recurrent Neural Networks

Convolutional Neural Networks.

• Understanding Convolutional
• Convolution with Padding
• Convolution with Strides
• Max Pooling
• Average Pooling
• Understanding Deconvolution
• Deconvolution with Strides
• Deconvolution with Padding

Generative Models.

• Autoencoder
• Denoising Autoencoder
• Sparse Autoencoder
• Variational Autoencoder
• Generative Adversarial Networks
• Deep Convolutional GAN
• Conditional GAN
• AC GAN
• LS GAN
• Info GAN
• Cyclic GAN. (To be implemented)
• Disco GAN. (To be implemented)
• Bi GAN. (To be implemented)
• W GAN. (To be implemented)

Computer Vision

• LeNet
• AlexNet
• VGG
• Network in Network
• Inception Network
• ResNet
• DenseNet
• Data Augmentation

Applications of Computer Vision

• Image Classification
• Transfer Learning
• Image Denoising
• Image Resolution Inhancer. (To be implemented)
• Image Reconstruction. (To be implemented)
• Image Colorization. (To be implemented)
• Object Detection. (To be implemented)
• Objection Segmentation. (To be implemented)
• Style Transfer. (To be implemented)
• Text to Image. (To be implemented)
• Image Captioning. (To be implemented)

Natural Language Processing

• Word Embeddings. (To be implemented)
• Embedding Matrix. (To be implemented)
• Word to Vec. (To be implemented)
• GloVe. (To be implemented)
• Negative Sampling. (To be implemented)
• Attention Mechanism. (To be implemented)

Applications of Natural Language Processing

• Sentiment Analysis. (To be implemented)
• Named Entity Recognition. (To be implemented)
• Machine Translation. (To be implemented)
• Text Summarization. (To be implemented)
• Visual Question Answering. (To be implemented)
• Image Captioning. (To be implemented)
• Music Generation. (To be implemented)

How to run provided notebooks?

It is recommended to run notebooks locally on your computer only if you have GPU(CUDA) support, otherwise it'd be quit painfull :O

If you don't have the GPU, then follow the steps as bellow:-

• Go to https://colab.research.google.com
• Sign In using your google account.
• Clone the repository by running git clone
• From the menu top left, Go to "File -> Upload notebook -> Choose File".
• Then select the notebook you want to run located in your computer and click "Open".
• Wait for a while.
• When notebook is successfully loaded, from the menu top left go to "Runtime -> Change runtime type".
• Select GPU as your "Hardware accelerator".
• Click "SAVE".
• Now you are ready to roll.

What are some good resources to learn Deep Learning?

Here are some very useful resources I've curated so far. First we will learn some basic fundamentals of Deep Learning, then we will learn the core math required to understand a bit complex Deep Learning approaches, so watch and read in the following order.

1. Deep Learning For Beginners by Andrew Ng

   Its a full course for beginners by Prof Andrew Ng, this course provides the glimps to the world of AI.

2. Deep Learning For Intermediates by Andrew Ng

   Its a full course for intermediates by Prof Andrew Ng, this course provides techniques to improve Deep Learning models.

3. Convolutional Neural Networks by Andrew Ng

   Its a full course on Convolutional Neural Networks by Prof Andrew Ng, this course provides great explanations and examples for the topic with great simplicity and helpful guides.

4. Recurrent Neural Networks by Andrew Ng

   Its a full course on Recurrent Neural Networks by Prof Andrew Ng, this course provides explanations to why we need sequence models and what are their use cases and how do they work.

5. An Outstanding course for Linear Algebra by Prof W. Gilbert Strange

   This course is by far my best course on Linear Algebra on the planet. Prof Strange has done a phenomenal job, just watch his videos and I bet you'd see a rainbow with a unicorn.

6. An Another Outstanding Course on Linear Algera By 3Blue1Brown

   This course introduces the "Essence of linear algebra" series, aimed at animating the geometric intuitions underlying many of the topics taught in a standard linear algebra course.

7. Course on Calculus by 3Blue1Brown

   What might it feel like to invent calculus?

8. A Great Course on Deep Learning by MIT

   Its a full course on Deep Learning by MIT, this course provides great explanations to various deep learning topics like GANs... etc. Note: only watch first 4 lectures because afterwards they are explaining an another AI field know as Reinforcement Learning, which we will cover later.

9. Deep Learning Course by Stanford

   This course covers introduction to deep learning and its applications like Adversarial Attacks, Interpretability and Health Care... etc. Note: skip the Reinforcement Learning part for now, we will cover that later.

10. An Awesome Book on Deep Learning

    This book is an interactive deep learning book with codes, maths, and discussions, for beginners as well as for experts. It provides crystal clear explaination for all the topics along with hands on coding tutorials. If you got question regarding the topic then you can ask it to the forum as well.

11. Another Great Book on Deep Learning by Ian Goodfellow

    This book introduces to Deep Learning by first teaching the crucial math required for research in Deep Learning, for example:-

    • Linear Algebra
    • Probability
    • Numerical Computation... etc

    This book would definitely help you to build the foundational understanding of mathematics for research in Deep Learning.

TODO

• Implement all above algorithms from scratch in Tensorflow.
• Implement all above algorithms from scratch in Pytorch.
• Provide resources to learn topics mentioned above.
• Add more explanations to notebooks for better understanding.

Author

Rishik Mourya