SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime

LLM model quantization (compression) toolkit with hw acceleration support for Nvidia CUDA, AMD ROCm, Intel XPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

Large Language Models for All, 🦙 Cult and More, Stay in touch !

🦖 X—LLM: Cutting Edge & Easy LLM Finetuning

Run any Large Language Model behind a unified API

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

记录量化LLM中的总结。

A guide about how to use GPTQ models with langchain

An OpenAI Compatible API which integrates LLM, Embedding and Reranker. 一个集成 LLM、Embedding 和 Reranker 的 OpenAI 兼容 API

Run gguf LLM models in Latest Version TextGen-webui and koboldcpp

Private self-improvement coaching with open-source LLMs

ChatSakura：Open-source multilingual conversational model.（开源多语言对话大模型）

This repository is for profiling, extracting, visualizing and reusing generative AI weights to hopefully build more accurate AI models and audit/scan weights at rest to identify knowledge domains for risk(s).

A.L.I.C.E (Artificial Labile Intelligence Cybernated Existence). A REST API of A.I companion for creating more complex system

🎯 Fine-tune large language models and use them for text-related tasks. This repository provides a straightforward approach to fine-tuning models like Gemma, Llama 🦙, and Mistral 🌪️ for various NLP tasks. 🔧 It includes training 📚, fine-tuning 🛠️, and inference pipelines ⚙️. 🚀

Conversation AI model for open domain dialogs

Code for NAACL paper When Quantization Affects Confidence of Large Language Models?

This project will develop a NEPSE chatbot using an open-source LLM, incorporating sentence transformers, vector database and reranking.

Optimized Qwen2.5-3B using GPTQ, reducing size from 5.75GB → 1.93GB and improving inference speed. Ideal for efficient edge AI deployments.

LLM quantization techniques: absmax, zero-point, GPTQ and GGUF