Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.

Data processing with ML, LLM and Vision LLM

An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention)

Evaluate your LLM's response with Prometheus and GPT4 💯

[EMNLP 2024 Industry Track] This is the official PyTorch implementation of "LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit".

A large-scale simulation framework for LLM inference

The RunPod worker template for serving our large language model endpoints. Powered by vLLM.

Official Implementation of TopicGPT: A Prompt-Based Framework for Topic Modeling (NAACL '24)

Fine-tuning and serving LLMs on any cloud

llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more.

An endpoint server for efficiently serving quantized open-source LLMs for code.

A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.

大模型推理框架加速，让 LLM 飞起来

A REST API for vLLM, production ready

ICE-PIXIU：A Cross-Language Financial Megamodeling Framework

CI scripts designed to build a Pascal-compatible version of vLLM.

vLLM Router

Run code inference-only benchmarks quickly using vLLM

An OpenAI Compatible API which integrates LLM, Embedding and Reranker. 一个集成 LLM、Embedding 和 Reranker 的 OpenAI 兼容 API

An simple implementation of Unet because all the implementations i've seen are wayy tooo complicated.