Understand the fundamentals of Natural Language Processing (NLP) and how to apply them to solve practical problems. This repository contains the code implementation of different models and techniques used in NLP.
- Representation of Words (One-Hot, Distributed)
- SVD for word representation
- BOW
- n gram
- Skip gram (Word2Vec)
- Prompting Methods
- GPT
• Next Sentence Prediction (NSP)(Devlin et al., 2019): A binary classification loss predicting whether two segments appear consecutively within a larger document, or are random unrelated sentences.
• Sentence Order Prediction (SOP) (Lan et al., 2020): A binary classification loss for predicting whether two sentences are in a natural or swapped order.
• Capital Word Prediction (CWP) (Liu et al., 2020b): A binary classification objective calculated over each word, predicting whether whether each word is capitalized or not.
• Sentence Deshuffling (SDS) (Liu et al., 2020b): A multi-class classification task to reorganize permuted segments.
• Sentence distance prediction (SDP) (Liu et al., 2020b) : A three-class classification task, predicting the positional relationship between two sentences (adjacent in the same document, not adjacent but in the same document, in different documents).
• Masked Column Prediction (MCP) (Yin et al., 2020): Given a table, recover the names and data types of masked columns.
• Linguistic-Visual Alignment (LVA) (Lu et al., 2019): A binary classification to Predict whether the text content can be aligned to visual content.
• Image Region prediction (IRP) (Su et al., 2020): Given an image whose partial features are masked (zeroed out), predict the masked regions.
• Replaced Token Detection (RTD) (Xiao et al., 2021): A binary classification loss predicting whether each token in corrupted input was replaced by a generative sample or not.
• Discourse Relation Prediction (DRP) (Sun et al., 2020): Predict the semantic or rhetorical relation between two sentences.
• Translation Language Modeling (TLM) (Lample and Conneau, 2019): Consider parallel sentences and mask words randomly in both source and target sentences.
• Information Retrieval Relevance (IRR) (Sun et al., 2020): Predict the information retrieval relevance of two sentences.
• Token-Passage Prediction (TPP) (Liu et al., 2020b): Identify the keywords of a passage appearing in the segment.
• Universal Knowledge-Text Prediction (UKTP) (Sun et al., 2021): Incorporate knowledge into one pre-trained language model.
• Machine Translation (MT) (Chi et al., 2021a) : Translate a sentence from the source language into the target language.
• Translation Pair Span Corruption (TPSC) (Chi et al., 2021a) : Predict the masked spans from a translation pair.
• Translation Span Corruption (TSC) (Chi et al., 2021a) : Unlike TPSC, TSC only masks and predicts the spans in one language.
• Multilingual Replaced Token Detection (MRTD) (Chi et al., 2021b): Distinguish real input tokens from corrupted multilingual sentences by a Generative Adversarial Network, where both the generator and the discriminator are shared across languages.
• Translation Replaced Token Detection (TRTD) (Chi et al., 2021b): Distinguish the real tokens and masked tokens in the translation pair by the Generative Adversarial Network.
• Knowledge Embedding (KE) (Wang et al., 2021): Encode entities and relations in knowledge graphs (KGs) as distributed representations
• Image-to-text transfer (ITT) (Wang et al., 2021): Is similar to the image caption that generates a corresponding description for the input image.
• Multimodality-to-text transfer (MTT) (Wang et al., 2021): Generate the target text based on both the visual information and the noised linguistic information.
- Stanford CS224N: NLP with Deep Learning : Course website || 2021 Course || 2021 Lectures