Deep Learning for Natural Language Processing

Master course at the University of Zurich (HS18)

Content

This repository contains all exercises (including description and code in the folder code) and the summary of the lecture (including a book summary of Goldberg: Neural Network Methods for Natural Language Processing).

The summary folder contains :

• summary.md (main file with all content written in Markdown)

• summary.html (html file created from Markdown)

• summary.pdf (best format/presentation style)

The papers folder contains:

• papers.md (main file with a summary of all papers presented during the lecture written in Markdown)

• papers.html (html file created from Markdown)

• papers.pdf (best format/presentation style)

---

Summary Content

• Chapter 1: Intro

• Chapter 2: Machine Learning Pipeline

• Chapter 3: Learning Basics and Linear Models

  • Train, Validation and Test Set
  • Linear Models
  • Linear Separability
  • Loss Functions
  • Regularization
  • Gradient-based Optimization

• Chapter 4: Neural Networks

  • Activation Functions
  • Different Types of NN-Models
  • Training NNs

• Representations (Chapters 6,8,9)

  • Types of Features
  • Sparse Vector Representations
  • Encoding Categorical Features
  • Combining Dense Vectors

• Embeddings (Chapters 10,11)

  • Tips & Tricks regarding Embeddings
  • Pre-Trained Word Embeddings
    • word2vec, an example of a distributed representation algorithm
  • Char-based and Sub-Word Embeddings

• Convolutional Neural Networks (CNNs/ConvNets) (Chapter 13)

  • CNN Architecture
  • 1D Convolution over Text
  • Pooling
  • Alternative: Feature-Hashing
  • Hierarchical Convolutions
  • Issues with Deep CNNs

• Recurrent Neural Networks (RNNs) (Chapters 14,15,16)

  • The RNN Abstraction
  • RNN Training
  • Bi-Directional RNN
  • Multi-Layer (Stacked) RNNs / deep RNNs
  • RNN Applications / Usages
  • RNN Architectures
    • LSTM (Long Short-Term Memory)
    • GRU (Gated Recurrent Unit)
  • Other RNN Usages
    • Generators
    • Conditioned Generation
    • Next Step? Attention

---

Moritz Eck