Skip to content
@llm-d

llm-d

llm-d is a Kubernetes-native high-performance distributed LLM inference framework
README.md

Welcome to llm-d: a Kubernetes-native high-performance distributed LLM inference framework

GitHub Org's stars Documentation License

Join Slack X (formerly Twitter) Follow LinkedIn Reddit

llm-d is a well-lit path for serving large language models at scale with the fastest time-to-value and competitive performance per dollar. Built on vLLM, Kubernetes, and Inference Gateway, llm-d provides modular solutions for distributed inference with features like KV-cache aware routing and disaggregated serving.

Key Resources

🤝 How to Contribute

Join the Community

  1. 💬 Slack: Join our development discussions at llm-d.slack.com
  2. 📧 Google Group: Subscribe to llm-d-contributors for architecture docs and meeting invites
  3. 🗓️ Weekly Standup: Wednesdays at 1230 ET - Public Calendar

Contributing Code

  1. Read Guidelines: Review our Code of Conduct and contribution process
  2. Sign Commits: All commits require DCO sign-off (git commit -s)

Ways to Contribute

  • 🐛 Bug fixes and small features - Submit PRs directly to component repos
  • 🚀 New features with APIs - Require project proposals
  • 📚 Documentation - Help improve guides and examples
  • 🧪 Testing & Benchmarking - Contribute to our test coverage
  • 💡 Experimental features - Start in llm-d-incubation org

License: Apache 2.0

Pinned Loading

  1. llm-d llm-d Public

    llm-d is a Kubernetes-native high-performance distributed LLM inference framework

    Makefile 1.2k 86

  2. llm-d-inference-scheduler llm-d-inference-scheduler Public

    Inference scheduler for llm-d

    Go 57 25

  3. llm-d-deployer llm-d-deployer Public

    Helm charts for llm-d

    Shell 42 29

  4. llm-d-kv-cache-manager llm-d-kv-cache-manager Public

    Distributed KV cache coordinator

    Go 35 6

  5. llm-d-model-service llm-d-model-service Public

    Simplified model deployment on llm-d

    Go 24 9

  6. llm-d-benchmark llm-d-benchmark Public

    llm-d benchmark scripts and tooling

    Shell 16 8

Repositories

Showing 10 of 12 repositories
  • llm-d-inference-sim Public

    A light weight vLLM simulator, for mocking out replicas.

    llm-d/llm-d-inference-sim’s past year of commit activity
    Go 24 11 6 1 Updated Jun 19, 2025
  • llm-d-benchmark Public

    llm-d benchmark scripts and tooling

    llm-d/llm-d-benchmark’s past year of commit activity
    Shell 16 Apache-2.0 8 3 0 Updated Jun 19, 2025
  • llm-d Public

    llm-d is a Kubernetes-native high-performance distributed LLM inference framework

    llm-d/llm-d’s past year of commit activity
    Makefile 1,211 Apache-2.0 86 20 13 Updated Jun 18, 2025
  • llm-d-inference-scheduler Public

    Inference scheduler for llm-d

    llm-d/llm-d-inference-scheduler’s past year of commit activity
    Go 57 Apache-2.0 25 52 (2 issues need help) 4 Updated Jun 18, 2025
  • llm-d.github.io Public

    Website for llm-d: This repository builds the website seen at llm-d.ai

    llm-d/llm-d.github.io’s past year of commit activity
    JavaScript 10 14 6 5 Updated Jun 17, 2025
  • llm-d-kv-cache-manager Public

    Distributed KV cache coordinator

    llm-d/llm-d-kv-cache-manager’s past year of commit activity
    Go 35 6 10 (2 issues need help) 3 Updated Jun 17, 2025
  • llm-d-deployer Public

    Helm charts for llm-d

    llm-d/llm-d-deployer’s past year of commit activity
    Shell 42 Apache-2.0 28 24 15 Updated Jun 16, 2025
  • .github Public
    llm-d/.github’s past year of commit activity
    0 0 0 0 Updated Jun 3, 2025
  • llm-d-model-service Public

    Simplified model deployment on llm-d

    llm-d/llm-d-model-service’s past year of commit activity
    Go 24 Apache-2.0 9 29 7 Updated Jun 2, 2025
  • llm-d-pd-utils Public
    llm-d/llm-d-pd-utils’s past year of commit activity
    Makefile 4 3 0 0 Updated May 21, 2025
View all repositories

Most used topics

incubating ai