Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
3 contributors

Users who have contributed to this file

@Separius @zhanqiuzhang @basicv8vc
239 lines (209 sloc) 34.5 KB

Github please do not ban me!

awesome-sentence-embedding Awesome

Build Status GitHub - LICENSE

A curated list of pretrained sentence and word embedding models

Table of Contents

About This Repo

  • well there are some awesome-lists for word embeddings and sentence embeddings, but all of them are outdated and more importantly incomplete
  • this repo will also be incomplete, but I'll try my best to find and include all the papers with pretrained models
  • this is not a typical awesome list because it has tables but I guess it's ok and much better than just a huge list
  • if you find any mistakes or find another paper or anything please send a pull request and help me to keep this list up to date
  • enjoy!

General Framework

  • Almost all the sentence embeddings work like this:
  • Given some sort of word embeddings and an optional encoder (for example an LSTM) they obtain the contextualized word embeddings.
  • Then they define some sort of pooling (it can be as simple as last pooling).
  • Based on that they either use it directly for the supervised classification task (like infersent) or generate the target sequence (like skip-thought).
  • So, in general, we have many sentence embeddings that you have never heard of, you can simply do mean-pooling over any word embedding and it's a sentence embedding!

Word Embeddings

  • Note: don't worry about the language of the code, you can almost always (except for the subword models) just use the pretrained embedding table in the framework of your choice and ignore the training code
date paper citation count training code pretrained models
- WebVectors: A Toolkit for Building Web Interfaces for Vector Semantic Models N/A - RusVectōrēs(broken)
- Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings N/A - ChineseEmbedding
2013/01 Efficient Estimation of Word Representations in Vector Space 999+ C Word2Vec
2014/12 Word Representations via Gaussian Embedding 129 Cython -
2014/?? A Probabilistic Model for Learning Multi-Prototype Word Embeddings 27 DMTK -
2014/?? Dependency-Based Word Embeddings 541 C++ word2vecf
2014/?? GloVe: Global Vectors for Word Representation 999+ C GloVe
2015/06 From Paraphrase Database to Compositional Paraphrase Model and Back 0 Theano PARAGRAM
2015/06 Non-distributional Word Vector Representations 39 Python WordFeat
2015/06 Sparse Overcomplete Word Vector Representations 70 C++ -
2015/?? Topical Word Embeddings 142 Cython
2015/?? SensEmbed: Learning Sense Embeddings for Word and Relational Similarity 145 - SensEmbed
2015/?? Joint Learning of Character and Word Embeddings 95 C -
2016/02 Swivel: Improving Embeddings by Noticing What's Missing 37 TF -
2016/03 Counter-fitting Word Vectors to Linguistic Constraints 116 Python counter-fitting(broken)
2016/05 Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec 29 Chainer -
2016/06 Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations 31 Go lexvec
2016/06 Siamese CBOW: Optimizing Word Embeddings for Sentence Representations 95 Theano Siamese CBOW
2016/07 Enriching Word Vectors with Subword Information 999+ C++ fastText
2016/08 Morphological Priors for Probabilistic Neural Word Embeddings 20 Theano -
2016/11 A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks 165 C++ charNgram2vec
2016/12 ConceptNet 5.5: An Open Multilingual Graph of General Knowledge 148 Python Numberbatch
2016/?? Learning Word Meta-Embeddings 20 - Meta-Emb(broken)
2017/02 Offline bilingual word vectors, orthogonal transformations and the inverted softmax 143 Python -
2017/04 Multimodal Word Distributions 30 TF word2gm
2017/05 Poincaré Embeddings for Learning Hierarchical Representations 127 Pytorch -
2017/06 Context encoders as a simple but powerful extension of word2vec 3 Python -
2017/06 Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints 60 TF Attract-Repel
2017/08 Learning Chinese Word Representations From Glyphs Of Characters 10 C -
2017/08 Making Sense of Word Embeddings 20 Python sensegram
2017/09 Hash Embeddings for Efficient Word Representations 6 Keras -
2017/10 BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages 14 Gensim BPEmb
2017/11 SPINE: SParse Interpretable Neural Embeddings 12 Pytorch SPINE
2017/?? Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics 11 C -
2017/?? Joint Embeddings of Chinese Words, Characters, and Fine-grained Subcharacter Components 16 C -
2017/?? Dict2vec : Learning Word Embeddings using Lexical Dictionaries 20 C++ Dict2vec
2017/?? AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP 31 Gensim AraVec
2018/04 Dynamic Meta-Embeddings for Improved Sentence Representations 14 Pytorch DME/CDME
2018/04 Representation Tradeoffs for Hyperbolic Embeddings 26 Pytorch h-MDS
2018/05 Analogical Reasoning on Chinese Morphological and Semantic Relations 22 - ChineseWordVectors
2018/06 Probabilistic FastText for Multi-Sense Word Embeddings 11 C++ Probabilistic FastText
2018/09 Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks 1 TF SynGCN
2018/09 FRAGE: Frequency-Agnostic Word Representation 16 Pytorch -
2018/12 Wikipedia2Vec: An Optimized Tool for LearningEmbeddings of Words and Entities from Wikipedia 1 Cython Wikipedia2Vec
2018/?? cw2vec: Learning Chinese Word Embeddings with Stroke n-gram Information 10 C++ -
2019/02 VCWE: Visual Character-Enhanced Word Embeddings 0 Pytorch VCWE
2019/05 Learning Cross-lingual Embeddings from Twitter via Distant Supervision 0 Text -

OOV Handling

Contextualized Word Embeddings

  • Note: all the unofficial models can load the official pretrained models
date paper citation count code pretrained models
- Language Models are Unsupervised Multitask Learners N/A TF
Pytorch
Keras
GPT-2
2017/08 Learned in Translation: Contextualized Word Vectors 206 Pytorch
Keras
CoVe
2018/01 Universal Language Model Fine-tuning for Text Classification 240 Pytorch ULMFit(English, Zoo)
2018/02 Deep contextualized word representations 976 Pytorch
TF
ELMO(AllenNLP, TF-Hub)
2018/04 Efficient Contextualized Representation:Language Model Pruning for Sequence Labeling 4 Pytorch LD-Net
2018/07 Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation 22 Pytorch ELMo
2018/08 Direct Output Connection for a High-Rank Language Model 7 Pytorch DOC
2018/10 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 999+ TF
Keras
Pytorch
MXNet
PaddlePaddle
TF
Keras
BERT(BERT, ERNIE)
2018/?? Improving Language Understanding by Generative Pre-Training 277 TF
Keras
Pytorch
GPT
2018/?? Contextual String Embeddings for Sequence Labeling 48 Pytorch Flair
2019/01 BioBERT: pre-trained biomedical language representation model for biomedical text mining 25 TF BioBERT
2019/01 Cross-lingual Language Model Pretraining 42 Pytorch XLM
2019/01 Multi-Task Deep Neural Networks for Natural Language Understanding 46 Pytorch MT-DNN
2019/01 Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context 47 TF
Pytorch
Pytorch
Transformer-XL
2019/03 SciBERT: Pretrained Contextualized Embeddings for Scientific Text 8 Pytorch, TF SciBERT
2019/04 Publicly Available Clinical BERT Embeddings 10 Text clinicalBERT
2019/04 ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission 4 Pytorch ClinicalBERT
2019/05 ERNIE: Enhanced Language Representation with Informative Entities 4 Pytorch ERNIE
2019/06 Pre-Training with Whole Word Masking for Chinese BERT 0 Pytorch, TF BERT-wwm
2019/06 XLNet: Generalized Autoregressive Pretraining for Language Understanding 18 TF XLNet
2019/07 ERNIE 2.0: A Continual Pre-training Framework for Language Understanding 0 PaddlePaddle ERNIE 2.0
2019/07 RoBERTa: A Robustly Optimized BERT Pretraining Approach 1 Pytorch RoBERTa

Pooling Methods

Encoders

date paper citation count code model_name
- Incremental Domain Adaptation for Neural Machine Translation in Low-Resource Settings N/A Python AraSIF
2014/05 Distributed Representations of Sentences and Documents 998 Pytorch
Python
Doc2Vec
2014/11 Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models 466 Theano
Pytorch
VSE
2015/06 Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books 261 Theano
TF
Pytorch, Torch
SkipThought
2015/11 Order-Embeddings of Images and Language 181 Theano order-embedding
2015/11 Towards Universal Paraphrastic Sentence Embeddings 238 Theano ParagramPhrase
2015/?? From Word Embeddings to Document Distances 473 C, Python Word Mover's Distance
2016/02 Learning Distributed Representations of Sentences from Unlabelled Data 227 Python FastSent
2016/07 Charagram: Embedding Words and Sentences via Character n-grams 80 Theano Charagram
2016/11 Learning Generic Sentence Representations Using Convolutional Neural Networks 34 Theano ConvSent
2017/03 Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features 131 C++ Sent2Vec
2017/04 Learning to Generate Reviews and Discovering Sentiment 153 TF
Pytorch
Pytorch
Sentiment Neuron
2017/05 Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings 35 Theano GRAN
2017/05 Supervised Learning of Universal Sentence Representations from Natural Language Inference Data 410 Pytorch InferSent
2017/07 VSE++: Improving Visual-Semantic Embeddings with Hard Negatives 39 Pytorch VSE++
2017/08 Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm 147 Keras
Pytorch
DeepMoji
2017/09 StarSpace: Embed All The Things! 47 C++ StarSpace
2017/10 DisSent: Learning Sentence Representations from Explicit Discourse Relations 35 Pytorch DisSent
2017/11 Dual-Path Convolutional Image-Text Embedding with Instance Loss 2 Matlab Image-Text-Embedding
2017/11 Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations 36 Theano para-nmt
2018/03 An efficient framework for learning sentence representations 52 TF Quick-Thought
2018/03 Universal Sentence Encoder 89 TF-Hub USE
2018/04 End-Task Oriented Textual Entailment via Deep Explorations of Inter-Sentence Interactions 2 Theano DEISTE
2018/04 Learning general purpose distributed sentence representations via large scale multi-task learning 79 Pytorch GenSen
2018/06 Embedding Text in Hyperbolic Spaces 17 TF HyperText
2018/07 Representation Learning with Contrastive Predictive Coding 70 Keras CPC
2018/08 Context Mover’s Distance & Barycenters: Optimal transport of contexts for building representations 1 Python CMD
2018/09 Learning Universal Sentence Representations with Mean-Max Attention Autoencoder 2 TF Mean-MaxAAE
2018/10 Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model 2 TF-Hub USE-xling
2018/10 BioSentVec: creating sentence embeddings for biomedical texts 8 Python BioSentVec
2018/11 Word Mover's Embedding: From Word2Vec to Document Embedding 11 C, Python WordMoversEmbeddings
2018/11 A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks 15 Pytorch HMTL
2018/12 Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond 19 Pytorch LASER
2018/?? Convolutional Neural Network for Universal Sentence Embeddings 0 Theano CSE
2019/01 No Training Required: Exploring Random Encoders for Sentence Classification 10 Pytorch randsent
2019/02 CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model 0 Pytorch CMOW
2019/07 Multilingual Universal Sentence Encoder 1 TF-Hub MultilingualUSE

Evaluation

Misc

Vector Mapping

Articles

You can’t perform that action at this time.