Material for the class: Information-Theoretic Analysis of Language Models
Fall 2022/3, School of Computer Science, Reichman University
In this semi-seminar style class, we use tools from information theory to analyze and explore language modern models that are based on transformer neural networks. The goal of this class is to provide mathematical background for data compression and apply learned concepts to develop novel applications in natural language processing. This class is in a semi-seminar format: the instructor will provide a mathematical premier for data compression from information theory. Participants will prepare and deliver classes on subsequent topics, including live demonstrations of code snippets.
- notebooks: jupyter notebooks from lectures and student presentations
- data: datasets used in class and/or in home assignments
- reading: additioanl reading material