Skip to content

Latest commit

 

History

History
404 lines (287 loc) · 22.3 KB

NLP.md

File metadata and controls

404 lines (287 loc) · 22.3 KB
Raw

NLP

Since 2018, pre-training has without a doubt become one of the hottest research topics in Natural Language Processing (NLP). By leveraging generalized language models like the BERT, GPT and XLNet, great breakthroughs have been achieved in natural language understanding. However, in sequence to sequence based language generation tasks, the popular pre-training methods have not achieved significant improvements. Now, researchers from Microsoft Research Asia have introduced MASS—a new pre-training method that achieves better results than BERT and GPT.

The general building blocks of their model, however, are still found in all current neural language and word embedding models. These are:

  1. Embedding Layer: a layer that generates word embeddings by multiplying an index vector with a word embedding matrix;
  2. Intermediate Layer(s): one or more layers that produce an intermediate representation of the input, e.g. a fully-connected layer that applies a non-linearity to the concatenation of word embeddings of $n$ previous words;
  3. Softmax Layer: the final layer that produces a probability distribution over words in $V$.

The softmax layer is a core part of many current neural network architectures. When the number of output classes is very large, such as in the case of language modelling, computing the softmax becomes very expensive.

Word Embedding and Language Model

Word are always in the string data structure in computer.

Language is made of discrete structures, yet neural networks operate on continuous data: vectors in high-dimensional space. A successful language-processing network must translate this symbolic information into some kind of geometric representation—but in what form? Word embeddings provide two well-known examples: distance encodes semantic similarity, while certain directions correspond to polarities (e.g. male vs. female). There is no arithmetic operation on this data structure. We need an embedding that maps the strings into vectors.

Probabilistic Language Model

Language Modeling (LM) estimates the probability of a word given the previous words in a sentence: $P(x_t\mid x_1,\cdots, x_{t-1},\theta)$. Formally, the model is trained with inputs $(x_1,\cdots, x_{t-1})$ and outputs $Y(x_t)$, where $x_t$ is the output label predicted from the final (i.e. top-layer) representation of a token $x_{t-1}$.

Neural Language Model

The Neural Probabilistic Language Model can be summarized as follows:

  1. associate with each word in the vocabulary a distributed word feature vector (a real valued vector in $\mathbb{R}^m$),
  2. express the joint probability function of word sequences in terms of the feature vectors of these words in the sequence, and
  3. learn simultaneously the word feature vectors and the parameters of that probability function.

When a word $w$ appears in a text, its context is the set of words that appear nearby (within a fixed-size window). We can use the many contexts of $w$ to build up a representation of $w$.

Attention Mechanism

Attention mechanisms in neural networks, otherwise known as neural attention or just attention, have recently attracted a lot of attention (pun intended).

An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.

Attention distribution is a probability distribution to describe how much we pay attention into the elements in a sequence for some specific task.

For example, we have a query vector $\mathbf{q}$ associated with the task and a list of input vector $\mathbf{X}=[\mathbf{x}_1, \mathbf{x}_2,\cdots, \mathbf{x}_N]$. We can select the input vector smoothly with the weight $$\alpha_i=p(z=i\mid \mathbf{X}, \mathbf{q})\ =\frac{\exp(s(\mathbf{x}i, \mathbf{q}))}{\sum{j=1}^N\exp(s(\mathbf{x}_j, \mathbf{q}))}$$ where $\alpha_i$ is the attention distribution, $s(\cdot,\cdot)$ is the attention scoring function. $z$ is the index of the vector in $\mathbf{X}$. The "Scaled Dot-Product Attention" use the modified dot-product as the scoring function.

The attention function between different input vectors is calculated as follows:

  1. Step 1: Compute scores between different input vectors and query vector $S_N$;
  2. Step 2: Translate the scores into probabilities such as $P = \operatorname{softmax}(S_N)$;
  3. Step 3: Obtain the output as aggregation such as the weighted value matrix with $Z = \mathbb{E}_{z\sim p(\mid \mathbf{X}, \mathbf{q} )}\mathbf{[x]}$.

There are diverse scoring functions and probability translation function, which will calculate the attention distribution in different ways.

Efficient Attention, Linear Attention apply more efficient methods to generate attention weights.

Key-value Attention Mechanism and Self-Attention use different input sequence as following $$\operatorname{att}(\mathbf{K, V}, \mathbf{q}) =\sum_{j=1}^N\frac{s(\mathbf{K}_j, q)\mathbf{V}j}{\sum{i=1}^N s(\mathbf{K}_i, q)}$$ where $\mathbf{K}$ is the key matrix, $\mathbf{V}$ is the value matrix, $s(\cdot, \cdot)$ is the positive similarity function.

Each input token in self-attention receives three representations corresponding to the roles it can play:

  • query - asking for information;
  • key - saying that it has some information;
  • value - giving the information.

We compute the dot products of the query with all keys, divide each by square root of key dimension $d_k$, and apply a softmax function to obtain the weights on the values as following.

Soft Attention: the alignment weights are learned and placed “softly” over all patches in the source image; essentially the same type of attention as in Bahdanau et al., 2015. And each output is derived from an attention averaged input.

  • Pro: the model is smooth and differentiable.
  • Con: expensive when the source input is large.

Hard Attention: only selects one patch of the image to attend to at a time, which attends to exactly one input state for an output.

  • Pro: less calculation at the inference time.
  • Con: the model is non-differentiable and requires more complicated techniques such as variance reduction or reinforcement learning to train. (Luong, et al., 2015)

Soft Attention Mechanism

Soft Attention Mechanism is to output the weighted sum of vector with differentiable scoring function:

$$\operatorname{att}(\mathbf{X}, \mathbf{q}) = \mathbb{E}_{z\sim p(z\mid \mathbf{X}, \mathbf{q} )}\mathbf{[x]}$$

where $p(z\mid \mathbf{X}, \mathbf{q} )$ is the attention distribution.

In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix $Q$. The keys and values are also packed together into matrices $K$ and $V$. We compute the matrix of outputs as: $$\operatorname{Attention}(Q, K, V)= [\mathbb{E}_{z\sim p(z\mid \mathbf{K}, \mathbf{Q}1)}\mathbf{[V]},\cdots,\mathbb{E}{z\sim p(z\mid \mathbf{K}, \mathbf{Q}i )}\mathbf{[V]},\cdots, \mathbb{E}{z\sim p(z\mid \mathbf{K}, \mathbf{Q}_N )}\mathbf{[V]}].$$

Hard Attention Mechanism

Hard Attention Mechanism is to select most likely vector as the output $$\operatorname{att}(\mathbf{X}, \mathbf{q}) = \mathbf{x}j$$ where $j=\arg\max{i}\alpha_i$.

It is trained using sampling method or reinforcement learning.

Sparse Attention Mechanism

The softmax mapping is elementwise proportional to $exp$, therefore it can never assign a weight of exactly zero. Thus, unnecessary items are still taken into consideration to some extent. Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability.

Sparse Attention Mechanism is aimed at generating sparse attention distribution as a trade-off between soft attention and hard attention.

Graph Attention Networks

GAT introduces the attention mechanism as a substitute for the statically normalized convolution operation.

Transformer

Representation learning forms the foundation of today’s natural language processing system; Transformer models have been extremely effective at producing word- and sentence-level contextualized representations, achieving state-of-the-art results in many NLP tasks. However, applying these models to produce contextualized representations of the entire documents faces challenges. These challenges include lack of inter-document relatedness information, decreased performance in low-resource settings, and computational inefficiency when scaling to long documents. In this talk, I will describe 3 recent works on developing Transformer-based models that target document-level natural language tasks.

Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

Transformer blocks are characterized by a multi-head self-attention mechanism, a position-wise feed-forward network, layer normalization modules and residual connectors.

BERT

Masked Language Model

GPT

gpt-2

gpt3

image-gpt

ViT

Structured State Spaces

Pre-trained Models

Llama

Predictive coding

Toolbox

Open Library

Megatron-LM

FastMoE

DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.