Awesome Pretrained Language Models

Pretrained Language Models (PLMs) have been achieved great success over many NLP tasks. And with the rapid development of PLMs, there is a need to make a list to show these succesful and remarkable language models.

Lists of Language Models

Group by companies and organizations.

• Awesome Pretrained Language Models
• Lists of Language Models
  • Al21 Labs
  • Alibaba
  • Amazon
  • Baidu
  • BigSicence
  • DeepMind
  • EleutherAI
  • Meta (Facebook)
  • Fudan University
  • Google
  • HIT & iFLYTEK
  • Huawei
  • Inspur
  • IDEA
  • JD (JINGDONG)
  • Langboat
  • Microsoft
  • Nvidia
  • OpenAI
  • OPPO
  • Tencent
  • THU (Tsinghua) & BAAI (Beijing)
  • More PLMs are coming ...
• Claims

Al21 Labs

• Jurassic-1
  • JURASSIC-1: TECHNICAL DETAILS AND EVALUATION
  • github repo

Alibaba

• PLUG (Pre-training for Language Understanding and Generation))
  • 超大规模中文生成

Amazon

• Alexa Teacher Model
  • Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
  • AlexaTM 20B
    • AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
  • github repo (model hasn't been publicly released yet, stay tuned!)

Baidu

• ERNIE family: github repo
  • ERNIE 1.0
    • ERNIE: Enhanced Representation through Knowledge Integration
  • ERNIE 2.0
    • ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
  • ERNIE 3.0
    • ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
  • ERNIE 3.0 TiTan
    • ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
  • ERNIE 3.0 Zeus
    • ERNIE 3.0 Zeus Prompt: 全球首个融合任务相关知识的千亿中文大模型
  • ERNIE-GEN
    • ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation
  • ERNIE-ViL
    • ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
  • ERNIE-ViLG
    • ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation
  • ERNIE-Gram
    • ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
  • ERNIE-M
    • ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
  • ERNIE-Doc
    • ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
  • ERNIE-Tiny
    • ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
  • ERNIE-GeoL
    • ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
  • ERNIE-SPARSE
    • ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention
  • ERNIE-Search
    • ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval

• PLATO family: github repo
  • PLATO-1
    • PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
  • PLATO-2
    • PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning
  • PLATO-XL
    • PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
  • PLATO-KAG
    • PLATO-KAG: Unsupervised Knowledge-Grounded Conversation via Joint Modeling

BigSicence

• T0 series
  • Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Hugging Face 🤗
  • github repo
• BLOOM
  • Hugging Face 🤗

DeepMind

• Gopher
  • Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• Chinchilla
  • Training Compute-Optimal Large Language Models
• Retro
  • Improving language models by retrieving from trillions of tokens
  • github repo (Non-offical implementation)

EleutherAI

Hugging Face 🤗

• GPT-Neo
  • github repo
• GPT-J
  • github repo
• GPT-NeoX
  • github repo

Meta (Facebook)

• XGLM
  • Few-shot Learning with Multilingual Language Models
  • Hugging Face 🤗
• XLM-R
  • Unsupervised Cross-lingual Representation Learning at Scale
  • github repo
  • Hugging Face 🤗
• M2M-100
  • Beyond English-Centric Multilingual Machine Translation
  • github page
  • Hugging Face 🤗
• Blender
  • Recipes for building an open-domain chatbot
  • Internet-Augmented Dialogue Generation
  • Beyond Goldfish Memory: Long-Term Open-Domain Conversation
  • BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
  • github repo
  • BlenderBot A CONVERSATIONAL AI PROTOTYPE
  • Hugging Face 🤗
• OPT
  • OPT: Open Pre-trained Transformer Language Models
  • github repo
  • Hugging Face 🤗
  • Request access to OPT-175B
• NLLB
  • No Language Left Behind: Scaling Human-Centered Machine Translation
  • github repo
  • Hugging Face 🤗
• Altas
  • Few-shot Learning with Retrieval Augmented Language Models
• PEER
  • PEER: A Collaborative Language Model

Fudan University

• CPT
  • CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
  • github repo

Google

• T5 family
  • T5 series
    • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
    • GLU Variants Improve Transformer
    • github repo
    • Hugging Face 🤗
  • ByT5
    • ByT5: Towards a token-free future with pre-trained byte-to-byte models
    • github repo
    • Hugging Face 🤗
  • ExT5
    • ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
  • LongT5
    • LongT5: Efficient Text-To-Text Transformer for Long Sequences
    • github repo
    • Hugging Face 🤗
• LaMDA
  • LaMDA: Language Models for Dialog Applications
  • github repo (Non-offical)
• FLAN
  • Finetuned Language Models Are Zero-Shot Learners
  • github repo
• GShard
  • GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Meena
  • Towards a Human-like Open-Domain Chatbot
  • github repo (Non-offical)
• PaLM
  • PaLM: Scaling Language Modeling with Pathways
  • github repo (Non-offical)
• UL2
  • Unifying Language Learning Paradigms
  • github repo
  • Hugging Face 🤗
• Minerva
  • Solving Quantitative Reasoning Problems with Language Models
  • Minerva SAMPLE EXPLORER
• Flan-PaLM
  • Scaling Instruction-Finetuned Language Models
  • Flan-T5 checkpoints

HIT & iFLYTEK

• PERT:
  • PERT: Pre-training BERT with Permuted Language Model
  • github repo

哈工大讯飞联合实验室: Joint Laboratory of HIT and iFLYTEK Research (HFL).

Huawei

• PanGu (盘古) family
  • PanGu-Alpha
    • PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
    • 启智 AI协作平台
  • PanGu-Bot
    • PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
  • PanGu-Coder
    • PanGu-Coder: Program Synthesis with Function-Level Language Modeling
• NEZHA (哪吒)
  • NEZHA: Neural Contextualized Representation for Chinese Language Understanding

Huawei-Noah offical github repo for Pretrained-Language Model.

Inspur

• Yuan 1.0
  • Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
  • github repo

IDEA

• 封神榜——中文语言预训练模型开源计划
  • Erlangshen (二郎神)
  • Wenzhong (闻仲)
  • Rangdeng (燃灯)
  • Yuyuan (余元)
  • Bigan (比干)
  • Zhouwenwang (周文王)
  • Taiyi (太乙)

IDEA refers to 粤港澳大湾区数字经济研究院.

JD (JINGDONG)

• 织女 Vega v1

Langboat

• Mengzi
  • Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese
  • github repo

Microsoft

• Megatron-Turning NLG family
• DeBERTa series
  • DeBERTa: Decoding-enhanced BERT with Disentangled Attention
  • DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
  • github repo
  • Hugging Face 🤗
• DeepNMT
  • Very Deep Transformers for Neural Machine Translation
  • github repo
• DeepNet
  • DeepNet: Scaling Transformers to 1,000 Layers
• GODEL
  • GODEL: Large-Scale Pre-Training for Goal-Directed Dialog
  • github repo
• METRO
  • METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

Nvidia

• Megatron-LM
  • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  • github repo

OpenAI

• GPT-family
  • GPT, GPT-2, GPT-3
    • GPT: Improving Language Understanding by Generative Pre-Training
    • GPT-2: Language Models are Unsupervised Multitask Learners
    • GPT-3: Language Models are Few-Shot Learners
    • GPT-2 github repo
    • GPT-2 Hugging Face 🤗
  • WebGPT
    • WebGPT: Browser-assisted question-answering with human feedback
  • InstructGPT
    • Training language models to follow instructions with human feedback
• Codex
  • Evaluating Large Language Models Trained on Code

OPPO

• OBERT

Tencent

• Motian (摩天)
• ShenZhou (神舟)
• ShenNonG (神农)
• HunYuan (混元)

THU (Tsinghua) & BAAI (Beijing)

• THUDM
  • Wenhui (文汇)
    • github repo
  • GLM-130B
    • github repo
    • demo for GLM-130B

THUDM refers to "Data Mining Research Group at Tsinghua University".

• Tsinghua AI

  • ERNIE
    • ERNIE: Enhanced Language Representation with Informative Entities
  • CPM family
    • CPM-1
      • CPM: A Large-scale Generative Chinese Pre-trained Language Model
      • github repo
    • CPM-2
      • CPM-2: Large-scale cost-effective pre-trained language models
      • github repo
      • 启智 AI协作平台

• THU-CoAI

  • EVA family
    • EVA 1.0
      • EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training
      • github project
    • EVA 2.0
      • EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training
      • github project

THU-CoAI refers to "Conversational AI groups from Tsinghua University".

• BAAI
  • Chinese-Transformer-XL
    • github repo
    • 启智 AI协作平台
  • GLM
    • GLM: General Language Model Pretraining with Autoregressive Blank Infilling
    • 启智 AI协作平台

More PLMs are coming ...

🔥Coming soon!🔥

---

Claims

Thanks for all these companies and orgnizations paying a lot of money and efforts to build and train these large models for the benefit of all human beings.

This repo is inspired by awesome pretrained Chinese NLP models.