Language Modeling Research Hub

Welcome to the Language Modeling Research Hub, a comprehensive compendium for enthusiasts and scholars delving into the fascinating realm of language models (LMs), with a particular focus on large language models (LLMs). This repository is meticulously organized to facilitate easy navigation and access to a wealth of information.

• Learning Resources: A selection of articles, courses, open-source projects, and data designed to enhance your LM knowledge.
• Practices: A showcase of my experiments and code, offering interactive demos and frameworks that highlight LM's potential.

Dive in to explore, contribute, and expand the horizons of language modeling with us!

"In theory, theory and practice are the same. In practice, they are not." --Albert Einstein

📑 Table of Contents

• 📖 Learning Resources

  • Reading List
  • Notes
  • Courses
  • Books
  • Blogs
  • Talks
  • Open-source LLMs
  • Dataset Collections
  • Evaluation Benchmarks
  • Prompt Engineering

• 💻 Practice

  • API
  • Instruction Tuning
  • Constrained Generation

• 🤝 Related Collections

📖 Learning Resources

Reading List

Reading list and related notes for LLM research, see Reading List for details.

• Key Findings

• Architecture
• Causality
• Code Learning
• Dialogue
• Efficiency
• Human Alignment
• Information Extraction
• Instruction Tuning
• Interpretability
• In Context Learning
• Knowledge Update
• Mixture of Experts (MoE)
• Non-Autoregressive Generation
• Reasoning
  • Abstract Reasoning
  • Chain of Thought
  • Symbolic Reasoning
• Retrieval
• Social

Notes

My notes for LM research, see notes for details.

• Tokenization
• Position Encoding

Courses

• Johns Hopkins University: CS 601.x71 NLP: Self-supervised Models Spring 2023/Fall 2022
• Princeton University: COS 597G: Understanding Large Language Models Fall 2022
• Stanford University: CS25: Transformers United V2 Fall 2021/Winter 2023
• Stanford University: CS 324 - Advances in Foundation Models Winter 2022/Winter 2023

Books

• Foundations of Statistical Natural Language Processing, Chris Manning and Hinrich Schütze,
• Speech and Language Processing, Dan Jurafsky and James H. Martin
• The Oxford Handbook of Computational Linguistics, Ruslan Mitkov

Blogs

• Jay Alammar's Blog
• Lil'Log (Lilian Weng's Blog)
• llm-course (Mlabonne's Blog)
• Sebastian Ruder's Blog
• Yao Fu's Blog
• 科学空间 (Jianlin Su's Blog)

Talks

• LLM-Talk, LLMs in Five Formulas, by Alexander Rush

Open Source LLMs

Collection of various open-source LLMs, see Open-source LLMs for details.

• Pretrained Model
• Multitask Supervised Finetuned Model
• Instruction Finetuned Model
  • English
  • Chinese
  • Multilingual
• Human Feedback Finetuned Model
• Domain Finetuned Model
• Open Source Projects
  • reproduce/framework
  • accelerate
  • evaluation
  • deployment/demo

Related Collections

• open-llms , A list of open LLMs available for commercial use.
• LLM-Zoo , collects information of various open- and closed-source LLMs
• FindTheChatGPTer , 汇总那些ChatGPT的开源平替们，包括文本大模型、多模态大模型等
• 中国大模型 , 旨在记录中国大模型情况
• Awesome-Chinese-LLM , 整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等

Dataset Collections

Datasets for Pretrain/Finetune/Instruction-tune LLMs, see Datasets for details.

• Pretraining Corpora
• Instruction

Related Collections

• Awesome-LLMs-Datasets , Summarize existing representative LLMs text datasets.
• LLMDataHub , A quick guide (especially) for trending instruction finetuning datasets
• Awesome Instruction Datasets , A collection of awesome-prompt-datasets, awesome-instruction-dataset, to train ChatLLM such as chatgpt 收录各种各样的指令数据集, 用于训练 ChatLLM 模型
• sft_datasets , 开源SFT数据集整理,随时补充

Evaluation Benchmarks

Collection of automatic evaluation benchmarks, see Evaluation Benchmarks for details.

• English
  • Comprehensive
  • Knowledge
  • Reason
    • Hard Mathematical, Theorem
  • Code
  • Personalization
• Chinese
  • Comprehensive
  • Safety
• Multilingual

Related Collections

• Evaluations of LLM , The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models"

• EvaluationPapers4ChatGPT , Resource, Evaluation and Detection Papers for ChatGPT

• LLMs-for-NLG-Evaluation , Awesome LLM for NLG Evaluation Papers

Prompt Engineering

Collection of tricks of writing a perfect prompt, see Prompt for details.

💻 Practice

API

LLM API demos (including mirror links), see API for details.

• openai

Instruction Tuning

1. Instruction Construct: Construct Instruction by mixture or self-instruct
2. Fine Tuning: Instruction Tuning on 4 LLM with multilingual instructions

see Instruction Tuning for details.

• Experiments
  • Datasets
    • Collection
    • Bootstrap
  • Model Cards
  • Usage
• Results

Constrained Generation

constrain LLM to generate specific answer (e.g., some open ended QA, limited vocabulary tasks), see Constrained Generate for details.

• Common method (constrain vocabulary + sample algorithm)
• Trie + Beam search (has issues currently)

🤝 Related Collections

• LLMSurvey , A collection of papers and resources related to Large Language Models
• LLMsPracticalGuide , A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers)
• Awesome-LLM , a curated list of Large Language Model
• llm-action , 本项目旨在分享大模型相关技术原理以及实战经验