LLM-engineer-handbook

🔥 Large Language Models(LLM) have taken the NLP community AI community the Whole World by storm. The LLM space is complicated! This repo provides a curated list to help you navigate; it includes a collection of Large Language Model frameworks and tutorials, covering model training, serving, fine-tuning, and building LLM applications.

Table of Content

• Applications
• Pretraining
• Fine-tuning
• Serving
• Datasets
• Benchmarks
• Learn LLM Applications
• Understand LLM
• Social Accounts

Applications

Build & Auto-optimize

• AdalFlow - The library to build & auto-optimize LLM applications, from Chatbot, RAG, to Agent. It is AI-first with PyTorch-like design patterns.

• dspy - DSPy: The framework for programming—not prompting—foundation models.

Build

• LlamaIndex — A Python library for augmenting LLM apps with data.
• LangChain — A popular Python/JavaScript library for chaining sequences of language model prompts.

Prompt Optimization

• AutoPrompt - A framework for prompt tuning using Intent-based Prompt Calibration
• PromptFify - A library for prompt engineering that simplifies NLP tasks (e.g., NER, classification) using LLMs like GPT.

Others

• LiteLLM - Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format.

Pretraining

• PyTorch - PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.
• TensorFlow - TensorFlow is an open source machine learning library developed by Google.
• JAX - Google’s library for high-performance computing and automatic differentiation.
• tinygrad - A minimalistic deep learning library with a focus on simplicity and educational use, created by George Hotz.
• micrograd - A simple, lightweight autograd engine for educational purposes, created by Andrej Karpathy.

Fine-tuning

• Transformers - Hugging Face Transformers is a popular library for Natural Language Processing (NLP) tasks, including fine-tuning large language models.
• Unsloth - Finetune Llama 3.2, Mistral, Phi-3.5 & Gemma 2-5x faster with 80% less memory!

Serving

• TorchServe - An open-source model serving library developed by AWS and Facebook specifically for PyTorch models, enabling scalable deployment, model versioning, and A/B testing.

• TensorFlow Serving - A flexible, high-performance serving system for machine learning models, designed for production environments, and optimized for TensorFlow models but also supports other formats.

• Ray Serve - Part of the Ray ecosystem, Ray Serve is a scalable model-serving library that supports deployment of machine learning models across multiple frameworks, with built-in support for Python-based APIs and model pipelines.

• NVIDIA Triton Inference Server - A high-performance inference server supporting multiple ML/DL frameworks (TensorFlow, PyTorch, ONNX, etc.), optimized for GPU deployments, and ideal for both cloud and on-premises serving.

• vllm - An optimized, high-throughput serving engine for large language models, designed to efficiently handle massive-scale inference with reduced latency.

• sglang - SGLang is a fast serving framework for large language models and vision language models.

Datasets

Use Cases

• Datasets - A vast collection of ready-to-use datasets for machine learning tasks, including NLP, computer vision, and audio, with tools for easy access, filtering, and preprocessing.

Fine-tuning

• LLMDataHub - A quick guide (especially) for trending instruction finetuning datasets
• LLM Datasets - High-quality datasets, tools, and concepts for LLM fine-tuning.

Benchmarks

Agent

• TravelPlanner - paper A Benchmark for Real-World Planning with Language Agents

Understand LLM

Prompt Engineering

• Brown, Tom B. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

Reasoning & Planning

• Kambhampati, Subbarao, et al. "LLMs can't plan, but can help planning in LLM-modulo frameworks." arXiv preprint arXiv:2402.01817 (2024).

Learn LLM Applications

General

• AdalFlow documentation - Includes tutorials from building RAG, Agent, to LLM evaluation and fine-tuning.

• CS224N - Stanford course covering NLP fundamentals, LLMs, and PyTorch-based model building, led by Chris Manning and Shikhar Murty.

• LLM-driven Data Engineering - A playlist of 6 lectures by Zach Wilson on how LLMs will impact data pipeline development

Agent

Lectures

• LLM Agents MOOC - A playlist of 11 lectures by the Berkeley RDI Center on Decentralization & AI, featuring guest speakers like Yuandong Tian, Graham Neubig, Omar Khattab, and others, covering core topics on Large Language Model agents.

Projects

• OpenHands - Open source agents for developers by AllHands.

Social Accounts

• Chip Huyen
• Damien Benveniste, PhD
• Jim Fan
• Li Yin

Contributing

This is an active repository and your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are awesome for LLM frameworks, you could vote for them by adding 👍 to them.

---

If you have any question about this opinionated list, do not hesitate to contact Li Yin