Introduction to Deep Learning (11-785)

About the Course

Introduction to Deep Learning (11-785) is a comprehensive course offered at Carnegie Mellon University that focuses on the fundamentals of deep neural networks and their applications to various AI tasks. The course is designed to provide students with a solid foundation in deep learning, preparing them for advanced academic settings and giving them a significant advantage in the industrial job market.

Course Content

The course covers a wide range of topics related to deep learning, including but not limited to:

• Feedforward Neural Networks
• Convolutional Neural Networks
• Recurrent Neural Networks
• Generative Adversarial Networks
• Transformer Networks

Assignments

Each assignment in this course is divided into two parts:

Part 1: Recreating Parts of PyTorch

In the first part of each assignment, students are tasked with recreating various components of the PyTorch deep learning framework from scratch. This hands-on approach helps students gain a deeper understanding of the underlying mechanics of deep learning models and the operations that power them.

Part 2: Kaggle Competition

The second part of each assignment involves participating in a Kaggle competition. Students are required to apply their knowledge to build and train deep learning models to solve real-world problems. This practical experience not only reinforces the concepts learned in class but also provides valuable experience in model development and evaluation.

Homework 1:

Part 1:

Create building blocks for a Multi-Layer Perceptron (MLP) and implement Mean Squared Error (MSE), Cross Entropy loss functions, Stochastic Gradient Descent (SGD), and Batch Normalization.

Part 2:

Build a Multi-Layer Perceptron (MLP) that can effectively recognize and label the phoneme states in the training data.

Homework 2:

Part 1:

Create the building blocks for a Convolutional Neural Network (CNN) from scratch. This includes implementing forward and backward propagation through convolutional layers, pooling layers, and resampling layers.

Part 2:

Build and train a CNN for face recognition tasks

Homework 3:

Part 1:

Implement the building blocks for a Recurrent Neural Network (RNN) from scratch. This includes implementing forward and backward propagation through RNN layers, LSTM layers, and GRU layers.

Part 2:

Build and train an RNN for language modeling tasks

Homework 4:

Part 1:

Complete the building blocks for a Transformer Network from scratch. This includes implementing the self-attention mechanism. Additionally, complete the next token prediction and sequence generation functions.

Part 2:

Build and train a Transformer Network for sequence-to-sequence tasks