This project was completed as a part of the Honors portion of the Sequence Models Course on Coursera.
Credit to DeepLearning.AI and the Coursera platform for providing the course materials and guidance.
In this notebook, my primary objective is to explore and understand the Transformer architecture, a sophisticated neural network designed to take advantage of parallel processing, leading to significant improvements in the training process. Throughout this assignment, I will gain valuable insights into various aspects of the Transformer architecture. Firstly, I will learn how to create positional encodings, which play a crucial role in capturing sequential relationships within the data. This will allow the model to better understand the order and context of input sequences.
Next, I will delve into the concept of scaled dot-product self-attention, which involves calculating attention weights between different words in a sentence. This mechanism helps the model focus on relevant words and learn meaningful representations. Additionally, I will implement masked multi-head attention, a fundamental building block of the Transformer. This allows the model to selectively attend to different parts of the input, enabling it to handle variable-length sequences efficiently. Finally, I will build and train a complete Transformer model, harnessing its parallel processing capabilities to expedite the training process and achieve improved performance.
