Efficient Large Language Models: A Survey

Efficient Large Language Models: A Survey [arXiv] (Version 1 released on 12/06/2023; Version 2: 12/23/2023)

Zhongwei Wan¹, Xin Wang¹, Che Liu², Samiul Alam¹, Yu Zheng³, Jiachen Liu⁴, Zhongnan Qu⁵, Shen Yan⁶, Yi Zhu⁷, Quanlu Zhang⁸, Mosharaf Chowdhury⁴, Mi Zhang¹

¹The Ohio State University, ²Imperial College London, ³Michigan State University, ⁴University of Michigan, ⁵Amazon AWS AI, ⁶Google Research, ⁷Boson AI, ⁸Microsoft Research Asia

⚡News: Our survey paper has been officially accepted by TMLR 2024: [OpenReview]

@article{wan2023efficient,
  title={Efficient Large Language Models: A Survey},
  author={Wan, Zhongwei and Wang, Xin and others},
  year={2023},
  journal={arXiv preprint arXiv:2312.03863},
}

❤️ Contribution

This repository is maintained by tuidan (wang.15980@osu.edu), SUSTechBruce (wan.512@osu.edu), samiul272 (alam.140@osu.edu), and mi-zhang (mizhang.1@osu.edu). We welcome feedback, suggestions, and contributions that can help improve this survey and repository so as to make them valuable resources to benefit the entire community.

If you have any suggestions regarding our taxonomy, find any missed papers, or update any preprint arXiv paper that has been accepted to some venue, feel free to send us an email or submit a pull request using the following markdown format.

Paper Title, <ins>Conference/Journal/Preprint, Year</ins>  [[pdf](link)] [[other resources](link)].

If one preprint paper has multiple versions, please use the earliest submitted year.
Display the papers in a year descending order (the latest, the first).

📌 What is This Survey About?

Large Language Models (LLMs) have demonstrated remarkable capabilities in many important tasks and have the potential to make a substantial impact on our society. Such capabilities, however, come with considerable resource demands, highlighting the strong need to develop effective techniques for addressing the efficiency challenges posed by LLMs. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We hope our survey and this GitHub repository can serve as valuable resources to help researchers and practitioners gain a systematic understanding of the research developments in efficient LLMs and inspire them to contribute to this important and exciting field.

⚡We will actively maintain this repository and incorporate new research as it emerges.

🤔 Why Efficient LLMs are Needed?

Although LLMs are leading the next wave of AI revolution, the remarkable capabilities of LLMs come at the cost of their substantial resource demands. Figure 1 (left) illustrates the relationship between model performance and model training time in terms of GPU hours for LLaMA series, where the size of each circle is proportional to the number of model parameters. As shown, although larger models are able to achieve better performance, the amounts of GPU hours used for training them grow exponentially as model sizes scale up. In addition to training, inference also contributes quite significantly to the operational cost of LLMs. Figure 2 (right) depicts the relationship between model performance and inference throughput. Similarly, scaling up the model size enables better performance but comes at the cost of lower inference throughput (higher inference latency), presenting challenges for these models in expanding their reach to a broader customer base and diverse applications in a cost-effective way. The high resource demands of LLMs highlight the strong need to develop techniques to enhance the efficiency of LLMs. As shown in Figure 2, compared to LLaMA-1-33B, Mistral-7B, which uses grouped-query attention and sliding window attention to speed up inference, achieves comparable performance and much higher throughput. This superiority highlights the feasibility and significance of designing efficiency techniques for LLMs.

📖 Table of Content

🤖 Model-Centric Methods
🔢 Data-Centric Methods
- Data Selection
  - Data Selection for Efficient Pre-Training
  - Data Selection for Efficient Fine-Tuning
- Prompt Engineering
🧑‍💻 System-Level Efficiency Optimization and LLM Frameworks
- System-Level Efficiency Optimization
- LLM Frameworks

🤖 Model-Centric Methods

Model Compression

Quantization

Post-Training Quantization

Weight-Only Quantization

IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact, arXiv, 2024 [Paper]
OmniQuant: OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models, ICLR, 2024 [Paper] [Code]
OneBit: Towards Extremely Low-bit Large Language Models, arXiv, 2024 [Paper]
GPTQ: Accurate Quantization for Generative Pre-trained Transformers, ICLR, 2023 [Paper] [Code]
QuIP: 2-Bit Quantization of Large Language Models With Guarantees, arXiv, 2023 [Paper] [Code]
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, arXiv, 2023 [Paper] [Code]
OWQ: Lessons Learned from Activation Outliers for Weight Quantization in Large Language Models, arXiv, 2023 [Paper] [Code]
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression, arXiv, 2023 [Paper] [Code]
FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs, NeurIPS-ENLSP, 2023 [Paper]
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurlPS, 2022 [Paper] [Code]
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning, NeurIPS, 2022 [Paper] [Code]
QuantEase: Optimization-based Quantization for Language Models, arXiv, 2023 [Paper] [Code]

Weight-Activation Co-Quantization

OmniQuant: OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models, ICLR, 2024 [Paper] [Code]
Intriguing Properties of Quantization at Scale, NeurIPS, 2023 [Paper]
ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation, arXiv, 2023 [Paper] [Code]
ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats, NeurIPS-ENLSP, 2023 [Paper] [Code]
OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization, ISCA, 2023 [Paper] [Code]
RPTQ: Reorder-based Post-training Quantization for Large Language Models, arXiv, 2023 [Paper] [Code]
Outlier Suppression+: Accurate Quantization of Large Language Models by Equivalent and Optimal Shifting and Scaling, arXiv, 2023 [Paper] [Code]
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models, arXiv, 2023 [Paper]
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML, 2023 [Paper] [Code]
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers, NeurIPS, 2022 [Paper]

Evaluation of Post-Training Quantization

Evaluating Quantized Large Language Models, arXiv, 2024 [Paper]

Quantization-Aware Training

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits, arXiv, 2024 [Paper]
FP8-LM: Training FP8 Large Language Models, arXiv, 2023 [Paper]
Training and inference of large language models using 8-bit floating point, arXiv, 2023 [Paper]
BitNet: Scaling 1-bit Transformers for Large Language Models, arXiv, 2023 [Paper]
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models, arXiv, 2023 [Paper] [Code]
Compression of Generative Pre-trained Language Models via Quantization, ACL, 2022 [Paper]

Parameter Pruning

Structured Pruning

Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models, ICLR, 2024 [Paper]
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation, arXiv, 2024 [Paper]
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect, arXiv, 2024 [Paper]
NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models, arXiv, 2024 [Paper]
SliceGPT: Compress Large Language Models by Deleting Rows and Columns, ICLR, 2024 [Paper] [Code]
LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery, arXiv, 2023 [Paper]
LLM-Pruner: On the Structural Pruning of Large Language Models, NeurIPS, 2023 [Paper] [Code]
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning, NeurIPS-ENLSP, 2023 [Paper] [Code]
LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning, arXiv, 2023 [Paper]

Unstructured Pruning

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot, ICML, 2023 [Paper] [Code]
A Simple and Effective Pruning Approach for Large Language Models, arXiv, 2023 [Paper] [Code]
One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models, arXiv, 2023 [Paper]

Low-Rank Approximation

SVD-LLM: Singular Value Decomposition for Large Language Model Compression, arXiv, 2024 [Paper] [Code]
ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models, arXiv, 2023 [Paper] [Code]
Language model compression with weighted low-rank factorization, ICLR, 2022 [Paper]
TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition, arXiv, 2023 [Paper]
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation, ICML, 2023 [Paper] [Code]

Knowledge Distillation

White-Box KD

DistiLLM: Towards Streamlined Distillation for Large Language Models, arXiv, 2024 [Paper] [Code]
Towards the Law of Capacity Gap in Distilling Language Models, arXiv, 2023 [Paper] [Code]
Baby Llama: Knowledge Distillation from an Ensemble of Teachers Trained on a Small Dataset with no Performance Penalty, arXiv, 2023 [Paper]
Knowledge Distillation of Large Language Models, arXiv, 2023 [Paper] [Code]
GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models, arXiv, 2023 [Paper]
Propagating Knowledge Updates to LMs Through Distillation, arXiv, 2023 [Paper] [Code]
Less is More: Task-aware Layer-wise Distillation for Language Model Compression, ICML, 2023 [Paper]
Token-Scaled Logit Distillation for Ternary Weight Generative Language Models, arXiv, 2023 [Paper]

Black-Box KD

Zephyr: Direct Distillation of LM Alignment, arXiv, 2023 [Paper]
Instruction Tuning with GPT-4, arXiv, 2023 [Paper] [Code]
Lion: Adversarial Distillation of Closed-Source Large Language Model, arXiv, 2023 [Paper] [Code]
Specializing Smaller Language Models towards Multi-Step Reasoning, ICML, 2023 [Paper] [Code]
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes, ACL, 2023 [Paper]
Large Language Models Are Reasoning Teachers, ACL, 2023 [Paper] [Code]
SCOTT: Self-Consistent Chain-of-Thought Distillation, ACL, 2023 [Paper] [Code]
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step, ACL, 2023 [Paper]
Distilling Reasoning Capabilities into Smaller Language Models, ACL, 2023 [Paper] [Code]
In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models, arXiv, 2022 [Paper]
Explanations from Large Language Models Make Small Reasoners Better, arXiv, 2022 [Paper]
DISCO: Distilling Counterfactuals with Large Language Models, arXiv, 2022 [Paper] [Code]

Parameter-Sharing

MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT, arXiv, 2024 [Paper]

Efficient Pre-Training

Mixed Precision Acceleration

GACT: Activation Compressed Training for Generic Network Architectures, ICML, 2022 [Paper] [Code]
Mesa: A Memory-saving Training Framework for Transformers, arXiv, 2021 [Paper] [Code]
Bfloat16 Processing for Neural Networks, ARITH, 2019 [Paper]
A Study of BFLOAT16 for Deep Learning Training, arXiv, 2019 [Paper]
Mixed Precision Training, ICLR, 2018 [Paper]

Scaling Models

lemon: lossless model expansion, ICLR, 2024 [Paper]
Preparing Lessons for Progressive Training on Language Models, AAAI, 2024 [Paper]
Learning to Grow Pretrained Models for Efficient Transformer Training, ICLR, 2023 [Paper] [Code]
2x Faster Language Model Pre-training via Masked Structural Growth, arXiv, 2023 [Paper]
Reusing Pretrained Models by Multi-linear Operators for Efficient Training, NeurIPS, 2023 [Paper]
FLM-101B: An Open LLM and How to Train It with $100 K Budget, arXiv, 2023 [Paper] [Code]
Knowledge Inheritance for Pre-trained Language Models, NAACL, 2022 [Paper] [Code]
Staged Training for Transformer Language Models, ICML, 2022 [Paper] [Code]

Initialization Techniques

Deepnet: Scaling transformers to 1,000 layers, arXiv, 2022 [Paper] [Code]
ZerO Initialization: Initializing Neural Networks with only Zeros and Ones, TMLR, 2022 [Paper] [Code]
Rezero is All You Need: Fast Convergence at Large Depth, UAI, 2021 [Paper] [Code]
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks, NeurIPS, 2020 [Paper]
Improving Transformer Optimization Through Better Initialization, ICML, 2020 [Paper] [Code]
Fixup Initialization: Residual Learning without Normalization, ICLR, 2019 [Paper]
On Weight Initialization in Deep Neural Networks, arXiv, 2017 [Paper]

Optimization Strategies

Towards Optimal Learning of Language Models, arXiv, 2024 [Paper] [Code]
Symbolic Discovery of Optimization Algorithms, arXiv, 2023 [Paper]
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training, arXiv, 2023 [Paper] [Code]

Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning

Adapter-based Tuning

OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models, ACL Demo, 2023 [Paper] [Code]
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models, EMNLP, 2023 [Paper] [Code]
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers, NeurIPS, 2023 [Paper] [Code]
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, NeurIPS, 2022 [Paper] [Code]
Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning, AutoML, 2022 [Paper]
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning, EMNLP, 2022 [Paper] [Code]
SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters, EMNLP, 2022 [Paper] [Code]

Low-Rank Adaptation

LoRA Meets Dropout under a Unified Framework, arXiv, 2024 [Paper]
STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models, arXiv, 2024 [Paper]
LoRA+: Efficient Low Rank Adaptation of Large Models, arXiv, 2024 [Paper]
LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning, arXiv, 2023 [Paper]
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition, arXiv, 2023 [Paper] [Code]
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models, arXiv, 2023 [Paper] [Code]
Multi-Head Adapter Routing for Cross-Task Generalization, NeurIPS, 2023 [Paper] [Code]
Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR, 2023 [Paper]
DyLoRA: Parameter-Efficient Tuning of Pretrained Models using Dynamic Search-Free Low Rank Adaptation, EACL, 2023 [Paper] [Code]
Tied-Lora: Enhacing Parameter Efficiency of LoRA with Weight Tying, arXiv, 2023 [Paper]
LoRA: Low-Rank Adaptation of Large Language Models, ICLR, 2022 [Paper] [Code]

Prefix Tuning

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention, arXiv, 2023 [Paper] [Code]
Prefix-Tuning: Optimizing Continuous Prompts for Generation ACL, 2021 [Paper] [Code]

Prompt Tuning

Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt, arXiv, 2023 [Paper]
GPT Understands, Too, AI Open, 2023 [Paper] [Code]
Multi-Task Pre-Training of Modular Prompt for Few-Shot Learning ACL, 2023 [Paper] [Code]
Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning, ICLR, 2023 [Paper]
PPT: Pre-trained Prompt Tuning for Few-shot Learning, ACL, 2022 [Paper] [Code]
Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers, EMNLP-Findings, 2022 [Paper] [Code]
P-Tuning v2: Prompt Tuning Can Be Comparable to Finetuning Universally Across Scales and Tasks，ACL-Short, 2022 [Paper] [Code]
The Power of Scale for Parameter-Efficient Prompt Tuning, EMNLP, 2021 [Paper]

Memory-Efficient Fine-Tuning

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, arXiv, 2024/ins> [Paper]
ReFT: Representation Finetuning for Language Models, arXiv, 2024/ins> [Paper]
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning, arXiv, 2024/ins> [Paper]
BitDelta: Your Fine-Tune May Only Be Worth One Bit, arXiv, 2024/ins> [Paper]
Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model, NeurIPS, 2023 [Paper] [Code]
Memory-Efficient Selective Fine-Tuning, ICML Workshop, 2023 [Paper]
Full Parameter Fine-tuning for Large Language Models with Limited Resources, arXiv, 2023 [Paper] [Code]
Fine-Tuning Language Models with Just Forward Passes, NeurIPS, 2023 [Paper] [Code]
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization, NeurIPS, 2023 [Paper]
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models, arXiv, 2023 [Paper] [Code]
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models, arXiv, 2023 [Paper] [Code]
QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS, 2023 [Paper] [Code1] [Code2]

Efficient Inference

Parallel Decoding

Encode Once and Decode in Parallel: Efficient Transformer Decoding, arXiv, 2024 [Paper]

Speculative Decoding

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding, arXiv, 2024 [Paper]
REST: Retrieval-Based Speculative Decoding, arXiv, 2024 [Paper]
Tandem Transformers for Inference Efficient LLMs, arXiv, 2024 [Paper]
REST: Retrieval-Based Speculative Decoding, 2023 [Paper]
PaSS: Parallel Speculative Sampling, NeurIPS Workshop, 2023 [Paper]
Accelerating Transformer Inference for Translation via Parallel Decoding, ACL, 2023 [Paper] [Code]
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads, Blog, 2023 [Blog] [Code]
Fast Inference from Transformers via Speculative Decoding, ICML, 2023 [Paper]
Accelerating LLM Inference with Staged Speculative Decoding, ICML Workshop, 2023 [Paper]
Accelerating Large Language Model Decoding with Speculative Sampling, arXiv, 2023 [Paper]
Speculative Decoding with Big Little Decoder, NeurIPS, 2023 [Paper] [Code]
SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification, arXiv, 2023 [Paper] [Code]
Inference with Reference: Lossless Acceleration of Large Language Models, arXiv, 2023 [Paper] [Code]

KV-Cache Optimization

SnapKV : LLM Knows What You are Looking for Before Generation, arXiv, 2024 [Paper]
Anchor-based Large Language Models, arXiv, 2024 [Paper]
kvquant: Towards 10 million context length llm inference with kv cache quantization, arXiv, 2024 [Paper]
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM, arXiv, 2024 [Paper]
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference, arXiv, 2024 [Paper]
No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization, arXiv, 2024 [Paper]
Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference, arXiv, 2024 [Paper]
WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More, arXiv, 2024 [Paper]
On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference, arXiv, 2024 [Paper]
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, arXiv, 2024 [Paper] [Code]
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs, ICLR, 2024 [Paper]
SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference, arXiv, 2023 [Paper]
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS, 2023 [Paper]
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time, NeurIPS, 2023 [Paper]
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers, arXiv, 2023 [Paper]

Efficient Architecture

Efficient Attention

Sharing-based Attention

LoMA: Lossless Compressed Memory Attention, arXiv, 2024 [Paper]
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases, arXiv, 2024 [Paper]
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP, 2023 [Paper]
Fast Transformer Decoding: One Write-Head is All You Need, arXiv, 2019 [Paper]

Feature Information Reduction

Nyströmformer: A nyström-based algorithm for approximating self-attention, AAAI, 2021 [Paper] [Code]
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing, NeurIPS, 2020 [Paper] [Code]
Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks, ICML, 2019 [Paper]

Kernelization or Low-Rank

Sumformer: Universal Approximation for Efficient Transformers, ICML Workshop, 2023 [Paper]
FLuRKA: Fast fused Low-Rank & Kernel Attention, arXiv, 2023 [Paper]
Scatterbrain: Unifying Sparse and Low-rank Attention, NeurlPS, 2021 [Paper] [Code]
Rethinking Attention with Performers, ICLR, 2021 [Paper] [Code]
Random Feature Attention, ICLR, 2021 [Paper]
Linformer: Self-Attention with Linear Complexity, arXiv, 2020 [Paper] [Code]
Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer, ICASSP, 2020 [Paper]
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, ICML, 2020 [Paper] [Code]

Fixed Pattern Strategies

Simple linear attention language models balance the recall-throughput tradeoff, arXiv, 2024 [Paper]
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models, arXiv, 2024 [Paper] [Code]
Faster Causal Attention Over Large Sequences Through Sparse Flash Attention, ICML Workshop, 2023 [Paper]
Poolingformer: Long Document Modeling with Pooling Attention, ICML, 2021 [Paper]
Big Bird: Transformers for Longer Sequences, NeurIPS, 2020 [Paper] [Code]
Longformer: The Long-Document Transformer, arXiv, 2020 [Paper] [Code]
Blockwise Self-Attention for Long Document Understanding, EMNLP, 2020 [Paper] [Code]
Generating Long Sequences with Sparse Transformers, arXiv, 2019 [Paper]

Learnable Pattern Strategies

HyperAttention: Long-context Attention in Near-Linear Time, arXiv, 2023 [Paper] [Code]
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer, ACL, 2022 [Paper]
Reformer: The Efficient Transformer, ICLR, 2022 [Paper] [Code]
Sparse Sinkhorn Attention, ICML, 2020 [Paper]
Fast Transformers with Clustered Attention, NeurIPS, 2020 [Paper] [Code]
Efficient Content-Based Sparse Attention with Routing Transformers, TACL, 2020 [Paper] [Code]

Mixture of Experts

MoE-based LLMs

An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing, 2024 [Paper]
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models, 2024 [Paper]
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM, 2024 [Paper]
Mixtral of Experts, arXiv, 2024 [Paper] [Code]
Mistral 7B, arXiv, 2023 [Paper] [Code]
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing, arXiv, 2023 [Paper]
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, JMLR, 2022 [Paper] [Code]
Efficient Large Scale Language Modeling with Mixtures of Experts, EMNLP, 2022 [Paper] [Code]
BASE Layers: Simplifying Training of Large, Sparse Models, ICML, 2021 [Paper] [Code]
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, ICLR, 2021 [Paper]

Algorithm-Level MoE Optimization

SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts, arXiv, 2024/ins> [Paper]
Scaling Laws for Fine-Grained Mixture of Experts, arXiv, 2024/ins> [Paper]
Lifelong Language Pretraining with Distribution-Specialized Experts, ICML, 2023 [Paper]
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models, arXiv, 2023 [Paper]
Mixture-of-Experts with Expert Choice Routing, NeurIPS, 2022 [Paper]
StableMoE: Stable Routing Strategy for Mixture of Experts, ACL, 2022 [Paper] [Code]
On the Representation Collapse of Sparse Mixture of Experts, NeurIPS, 2022 [Paper]

Long Context LLMs

Extrapolation and Interpolation

∞-Bench: Extending Long Context Evaluation Beyond 100K Tokens, arXiv, 2024 [Paper]
Resonance RoPE: Improving Context Length Generalization of Large Language Models, arXiv, 2024 [Paper] [Code]
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens, arXiv, 2024 [Paper]
E^2-LLM:Efficient and Extreme Length Extension of Large Language Models, arXiv, 2024 [Paper]
Scaling Laws of RoPE-based Extrapolation, arXiv, 2023 [Paper]
A Length-Extrapolatable Transformer, ACL, 2023 [Paper] [Code]
Extending Context Window of Large Language Models via Positional Interpolation, arXiv, 2023 [Paper]
NTK Interpolation, Blog, 2023 [Reddit post]
YaRN: Efficient Context Window Extension of Large Language Models, arXiv, 2023 [Paper] [Code]
CLEX: Continuous Length Extrapolation for Large Language Models, arXiv, 2023 [Paper][Code]
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training, arXiv, 2023 [Paper][Code]
Functional Interpolation for Relative Positions Improves Long Context Transformers, arXiv, 2023 [Paper]
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, ICLR, 2022 [Paper] [Code]
Exploring Length Generalization in Large Language Models, NeurIPS, 2022 [Paper]

Recurrent Structure

Retentive Network: A Successor to Transformer for Large Language Models, arXiv, 2023 [Paper] [Code]
Recurrent Memory Transformer, NeurIPS, 2022 [Paper] [Code]
Block-Recurrent Transformers, NeurIPS, 2022 [Paper] [Code]
∞-former: Infinite Memory Transformer, ACL, 2022 [Paper] [Code]
Memformer: A Memory-Augmented Transformer for Sequence Modeling, AACL-Findings, 2020 [Paper] [Code]
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, ACL, 2019 [Paper] [Code]

Segmentation and Sliding Window

Naive Bayes-based Context Extension for Large Language Models, NAACL, 2024 [Paper]
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention, arXiv, 2024 [Paper]
Training LLMs over Neurally Compressed Text, arXiv, 2024 [Paper]
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models, arXiv, 2024 [Paper]
Training-Free Long-Context Scaling of Large Language Models, arXiv, 2024 [Paper] [Code]
Long-Context Language Modeling with Parallel Context Encoding, arXiv, 2024 [Paper] [Code]
Soaring from 4K to 400K: Extending LLM’s Context with Activation Beacon, arXiv, 2024 [Paper] [Code]
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning, arXiv, 2024 [Paper] [Code]
Extending Context Window of Large Language Models via Semantic Compression, arXiv, 2023 [Paper]
Efficient Streaming Language Models with Attention Sinks, arXiv, 2023 [Paper] [Code]
Parallel Context Windows for Large Language Models, ACL, 2023 [Paper] [Code]
LongNet: Scaling Transformers to 1,000,000,000 Tokens, arXiv, 2023 [Paper] [Code]
Efficient Long-Text Understanding with Short-Text Models, TACL, 2023 [Paper] [Code]

Memory-Retrieval Augmentation

InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory, arXiv, 2024 [Paper]
Landmark Attention: Random-Access Infinite Context Length for Transformers, arXiv, 2023 [Paper] [Code]
Augmenting Language Models with Long-Term Memory, NeurIPS, 2023 [Paper]
Unlimiformer: Long-Range Transformers with Unlimited Length Input, NeurIPS, 2023 [Paper] [Code]
Focused Transformer: Contrastive Training for Context Scaling, NeurIPS, 2023 [Paper] [Code]
Retrieval meets Long Context Large Language Models, arXiv, 2023 [Paper]
Memorizing Transformers, ICLR, 2022 [Paper] [Code]

Transformer Alternative Architecture

State Space Models

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts, arXiv, 2024 [Paper]
DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models, arXiv, 2024 [Paper] [Code]
MambaByte: Token-free Selective State Space Model, arXiv, 2024 [Paper]
Sparse Modular Activation for Efficient Sequence Modeling, NeurIPS, 2023 [Paper] [Code]
Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv, 2023 [Paper] [Code]
Hungry Hungry Hippos: Towards Language Modeling with State Space Models, ICLR 2023 [Paper] [Code]
Long Range Language Modeling via Gated State Spaces, ICLR, 2023 [Paper]
Block-State Transformers, NeurIPS, 2023 [Paper]
Efficiently Modeling Long Sequences with Structured State Spaces, ICLR, 2022 [Paper] [Code]
Diagonal State Spaces are as Effective as Structured State Spaces, NeurIPS, 2022 [Paper] [Code]

Other Sequential Models

MEGALODON: Efficient LLM Pretraining and Inference with Unlimited Context Length, arXiv, 2024 [Paper]
DiJiang: Efficient Large Language Models through Compact Kernelization, arXiv, 2024 [Paper]
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, arXiv, 2024 [Paper]
PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensation, arXiv, 2023 [Paper]
RWKV: Reinventing RNNs for the Transformer Era, EMNLP-Findings, 2023 [Paper]
Hyena Hierarchy: Towards Larger Convolutional Language Models, arXiv, 2023 [Paper]
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers, arXiv, 2023 [Paper]

🔢 Data-Centric Methods

Data Selection

Data Selection for Efficient Pre-Training

DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, NeurIPS, 2023 [Paper]
Data Selection for Language Models via Importance Resampling, NeurIPS, 2023 [Paper] [Code]
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework, ICML, 2022 [Paper] [Code]
Span Selection Pre-training for Question Answering, ACL, 2020 [Paper] [Code]

Data Selection for Efficient Fine-Tuning

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models, arXiv, 2024 [Paper]
AutoMathText: Autonomous Data Selection with Language Models for Mathematical Texts, arXiv, 2024 [Paper] [Code]
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning, ICLR, 2024 [Paper] [Code]
How to Train Data-Efficient LLMs, arXiv, 2024[Paper]
LESS: Selecting Influential Data for Targeted Instruction Tuning, arXiv, 2024 [Paper] [Code]
Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning, arXiv, 2024 [Paper] [Code]
One Shot Learning as Instruction Data Prospector for Large Language Models, arXiv, 2023 [Paper]
MoDS: Model-oriented Data Selection for Instruction Tuning, arXiv, 2023 [Paper] [Code]
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning, arXiv, 2023 [Paper] [Code]
Instruction Mining: When Data Mining Meets Large Language Model Finetuning, arXiv, 2023 [Paper]
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors, ACL, 2023 [Paper] [Code]
Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values, ACL SRW, 2023 [Paper] [Code]
Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning, arXiv, 2023 [Paper]
AlpaGasus: Training A Better Alpaca with Fewer Data, arXiv, 2023 [Paper] [Code]
LIMA: Less Is More for Alignment, arXiv, 2023 [Paper]

Prompt Engineering

Few-Shot Prompting

Demonstration Organization

Demonstration Selection

Unified Demonstration Retriever for In-Context Learning, ACL, 2023 [Paper] [Code]
Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning, NeurIPS, 2023 [Paper] [Code]
In-Context Learning with Iterative Demonstration Selection, arXiv, 2022 [Paper]
Dr.ICL: Demonstration-Retrieved In-context Learning, arXiv, 2022 [Paper]
Learning to Retrieve In-Context Examples for Large Language Models, arXiv, 2022 [Paper]
Finding Supporting Examples for In-Context Learning, arXiv, 2022 [Paper]
Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering, ACL, 2023 [Paper] [Code]
Selective Annotation Makes Language Models Better Few-Shot Learners, ICLR, 2023 [Paper] [Code]
What Makes Good In-Context Examples for GPT-3? DeeLIO, 2022 [Paper]
Learning To Retrieve Prompts for In-Context Learning, NAACL-HLT, 2022 [Paper] [Code]
Active Example Selection for In-Context Learning, EMNLP, 2022 [Paper] [Code]
Rethinking the Role of Demonstrations: What makes In-context Learning Work? EMNLP, 2022 [Paper] [Code]

Demonstration Ordering

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity, ACL, 2022 [Paper]

Template Formatting

Instruction Generation

Large Language Models as Optimizers, arXiv, 2023 [Paper]
Instruction Induction: From Few Examples to Natural Language Task Descriptions, ACL, 2023 [Paper] [Code]
Large Language Models Are Human-Level Prompt Engineers, ICLR, 2023 [Paper] [Code]
TeGit: Generating High-Quality Instruction-Tuning Data with Text-Grounded Task Design, arXiv, 2023 [Paper]
Self-Instruct: Aligning Language Model with Self Generated Instructions, ACL, 2023 [Paper] [Code]

Multi-Step Reasoning

Automatic Chain of Thought Prompting in Large Language Models, ICLR, 2023 [Paper] [Code]
Measuring and Narrowing the Compositionality Gap in Language Models, EMNLP, 2023 [Paper] [Code]
ReAct: Synergizing Reasoning and Acting in Language Models, ICLR, 2023 [Paper] [Code]
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR, 2023 [Paper]
Graph of Thoughts: Solving Elaborate Problems with Large Language Models, arXiv, 2023 [Paper] [Code]
Tree of Thoughts: Deliberate Problem Solving with Large Language Models, NeurIPS, 2023 [Paper] [Code]
Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR, 2023 [Paper]
Graph of Thoughts: Solving Elaborate Problems with Large Language Models, arXiv, 2023 [Paper] [Code]
Contrastive Chain-of-Thought Prompting, arXiv, 2023 [Paper] [Code]
Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation, arXiv, 2023 [Paper]
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS, 2022 [Paper]

Parallel Generation

Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding, arXiv, 2023 [Paper] [Code]

Prompt Compression

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression, arXiv, 2024 [Paper]
PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models, arXiv, 2024 [Paper]
Compressed Context Memory For Online Language Model Interaction, ICLR, 2024 [Paper]
Learning to Compress Prompts with Gist Tokens, arXiv, 2023 [Paper]
Adapting Language Models to Compress Contexts, EMNLP, 2023 [Paper] [Code]
In-context Autoencoder for Context Compression in a Large Language Model, arXiv, 2023 [Paper] [Code]
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression, arXiv, 2023 [Paper] [Code]
Discrete Prompt Compression with Reinforcement Learning, arXiv, 2023 [Paper]
Nugget 2D: Dynamic Contextual Compression for Scaling Decoder-only Language Models, arXiv, 2023 [Paper]

Prompt Generation

TempLM: Distilling Language Models into Template-Based Generators, arXiv, 2022 [Paper] [Code]
PromptGen: Automatically Generate Prompts using Generative Models, NAACL Findings, 2022 [Paper]
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts, EMNLP, 2020 [Paper] [Code]

🧑‍💻 System-Level Efficiency Optimization and LLM Frameworks

System-Level Efficiency Optimization

System-Level Pre-Training Efficiency Optimization

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs, arXiv, 2024 [Paper]
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way, EMNLP, 2023 [Paper] [Code]
An Efficient 2D Method for Training Super-Large Deep Learning Models, IPDPS, 2023 [Paper] [Code]
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel, VLDB, 2023 [Paper]
Bamboo: Making Preemptible Instances Resilient for Affordable Training, NSDI, 2023 [Paper] [Code]
Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates, SOSP, 2023 [Paper] [Code]
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models, EuroSys, 2022 [Paper] [Code]
Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization, OSDI, 2022 [Paper] [Code]
Tesseract: Parallelize the Tensor Parallelism Efficiently, ICPP, 2022, [Paper]
Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning, OSDI, 2022, [Paper][Code]
Maximizing Parallelism in Distributed Training for Huge Neural Networks, arXiv, 2021 [Paper]
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, arXiv, 2020 [Paper]
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, SC, 2021 [Paper] [Code]
ZeRO-Infinity: breaking the GPU memory wall for extreme scale deep learning, SC, 2021 [Paper]
ZeRO-Offload: Democratizing Billion-Scale Model Training, USENIX ATC, 2021 [Paper] [Code]
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC, 2020 [Paper] [Code]

System-Level Serving Efficiency Optimization

Serving System Design

TurboTransformers: an efficient GPU serving system for transformer models, PPoPP, 2021 [Paper]
Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI, 2022 [Paper]
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, ICML, 2023 [Paper] [Code]
Efficiently Scaling Transformer Inference, MLSys, 2023 [Paper]
DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale, SC, 2022 [Paper]
Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP, 2023 [Paper] [Code]
S-LoRA: Serving Thousands of Concurrent LoRA Adapters, arXiv, 2023 [Paper] [Code]
Petals: Collaborative Inference and Fine-tuning of Large Models, arXiv, 2023 [Paper]
SpotServe: Serving Generative Large Language Models on Preemptible Instances, arXiv, 2023 [Paper]

Serving Performance Optimization

S3: Increasing GPU Utilization during Generative Inference for Higher Throughput, arXiv, 2023 [Paper]
Fast Distributed Inference Serving for Large Language Models, arXiv, 2023 [Paper]
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline, arXiv, 2023 [Paper]
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills, arXiv, 2023 [Paper]
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, arXiv, 2023 [Paper]
Prompt Cache: Modular Attention Reuse for Low-Latency Inference, arXiv, 2023 [Paper]
Fairness in Serving Large Language Models, arXiv, 2023 [Paper]

Algorithm-Hardware Co-Design

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS, 2022 [Paper] [Code]
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, arXiv, 2023 [Paper] [Code]
Flash-Decoding for Long-Context Inference, Blog, 2023 [Blog]
FlashDecoding++: Faster Large Language Model Inference on GPUs, arXiv, 2023 [Paper]
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU, arXiv, 2023 [Paper] [Code]
LLM in a flash: Efficient Large Language Model Inference with Limited Memory, arXiv, 2023 [Paper]
Chiplet Cloud: Building AI Supercomputers for Serving Large Generative Language Models, arXiv, 2023 [Paper]
EdgeMoE: Fast On-Device Inference of MoE-based Large Language Models, arXiv, 2022 [Paper]

LLM Frameworks

	Efficient Training	Efficient Inference	Efficient Fine-Tuning
DeepSpeed [Code]	✅	✅	✅
Megatron [Code]	✅	✅	✅
Alpa [Code]	✅	✅	✅
ColossalAI [Code]	✅	✅	✅
FairScale [Code]	✅	✅	✅
Pax [Code]	✅	✅	✅
Composer [Code]	✅	✅	✅
vLLM [Code]	❌	✅	❌
TensorRT-LLM [Code]	❌	✅	❌
LightLLM [Code]	❌	✅	❌
OpenLLM [Code]	❌	✅	✅
Ray-LLM [Code]	❌	✅	❌
MLC-LLM [Code]	❌	✅	❌
Sax [Code]	❌	✅	❌
Mosec [Code]	❌	✅	❌
LLM-Foundry [Code]	✅	✅	❌

Name		Name	Last commit message	Last commit date
Latest commit History 551 Commits
img		img
CITATION.cff		CITATION.cff
LICENSE.txt		LICENSE.txt
README.md		README.md

License

AIoT-MLSys-Lab/Efficient-LLMs-Survey

Folders and files

Latest commit

History

Repository files navigation