Natural Language Processing with Deep Learning

Taking together Stanford CS224n course with support of DeepPavlov team.

Time: весенний семестр 2020 года, вторник, 19:00

Location: Учебный центр 1С, Москва, Дмитровское шоссе, д.9 (метро "Тимирязевская"), аудитория 9235 (2 этаж).

News: https://t.me/dlinnlp2020spring

Chat: https://t.me/dlinnlp_discuss

Forum: https://forum.deeppavlov.ai/c/schools-hackatons/Deep-Learning-in-NLP/41

Course Structure

• weekly quizes
  • Quiz 1
  • Quiz 2
  • Quiz 3
• up to 5 practical hometasks (Jupyter Notebook), to be announced.
  • Assignment 1
  • Assignment 2
  • Assignment 3
• course project (obligatory), to be announced.

Spring 2020 syllabus

Week 1. Word Vector Representations (11.02.2020)

• Word Vector Representations (word2vec)
• Word Vectors and Word Senses (0:00-38:40, 58:00-1:20:00)
• Additional materials:
  • CS224n: Natural Language Processing with Deep Learning
  • CS224n: Word Vector Representations, word2vec
  • Word2Vec Tutorial - The Skip-Gram Model
  • Efficient Estimation of Word Representations in Vector Space
  • Distributed Representations of Words and Phrases and their Compositionality

Week 2. Neural Networks. Backpropagation (18.02.2020)

• CS224n: Word Vectors and Word Senses (38:40-58:00)
• Enriching Word Vectors with Subword Information
• CS231n: Backpropagation, Neural Networks 1
• Additional materials:
  • CS224n: GloVe: Global Vectors for Word Representation
  • fastText

Week 3. Neural Networks. Initialization and Normalization

• CS231n: Neural Networks
• CS231n: Lecture Notes
• CS231n: Lecture Notes

Week 4. Neural Networks. Optimization

• CS231n: Neural Networks
• CS231n: Lecture Notes

Week 5. Recurrent Neural Networks and Language Models

• CS224n: Language Models and RNNs
• CS224n: Language Models and RNNs: Notes

Week 6. Vanishing Gradients, Fancy RNNs

• CS224n: Vanishing Gradients, Fancy RNNs
• Paper: On the difficulty of training recurrent neural networks
• Article: Understanding LSTM Networks

Week 7. Convolutional Networks for NLP

• CS224n: Convolutional Networks for NLP
• Paper: Natural Language Processing (Almost) from Scratch
• Paper: Comparative Study of CNN and RNN for Natural Language Processing
• Paper: Convolutional Neural Networks for Sentence Classification

Week 8. Translation, Seq2Seq, Attention

• CS224n: Translation, Seq2Seq, Attention
• Lecture Notes
• Paper: Neural Machine Translation by Jointly Learning to Align and Translate
• Paper: Effective approaches to attention-based neural machine translation
• Article: Attention? Attention! by Lilian Weng

Approximate Syllabus

Week 6. Deep contextualized word representations

Week 7. Translation, Seq2Seq, Attention

Week 8. Contextual Word Embeddings

Week 9. Question Answering

Week 10. Natural Language Generation

Project Proposals

• The BERT-based Schema-Guided State Tracking [paper] [paper] [pic]
• The BERT Cross-Lingual Transferability [medium] [paper]
• BERT adaptation for new languages and tasks [presentation]
• How conversational is Conversational BERT? [docs]
• Grammatical error correction [Shared Task] [paper for Russian]
• Semi-supervised morpheme segmentation [paper]
• Low-resource morphological inflection [Shared Task] (to be updated)
• More morphological inflection [Shared Task]
• Automatic data augmentation (to be updated)
• Автоматическое решение ЕГЭ [Соревнование]
• Образовательные приложения глубокого обучения (обучение иностранному или русскому языку)
• [Semeval 2020]
• [SemEval 2018]
• [SemEval 2019]
• [Taxonomy enrichment]
• Russian aspect-based sentiment analysis [Dialog-2015]

Related Courses

• CS224n: Natural Language Processing with Deep Learning [course] [youtube]
• CS231n: Convolutional Neural Networks for Visual Recognition [course] [youtube]
• Machine Learning Glossary
• Open Machine Learning Course by @yorko
• DEEP LEARNING НА ПАЛЬЦАХ
• Theoretical Deep Learning

Useful Material

• CS231n: Python & NumPy Tutorial
• 100 numpy exercises
• The Matrix Calculus You Need For Deep Learning
• DeepMath2019
• CMU Neural Nets for NLP 2020