refft.cpp is a building tool to compile LLM/LMs' inference and training on the designated cloud-GPU or edge-NPU backends to a native executable including API, inference serving, training, model, ops, etc
-
Average 20%+ faster inference and training than Python/PyTorch-based inference/training(in the same quantization/precision and use cases)
-
0 running dependencies other than Linux/Android/Mac system and GPU/NPU backends
- Native Compilation -- Compile the whole inference/training of a LLM/LM into the native executable object
- OpenAI-Compatible API -- Seamless integration with existing tools
- Custom Training via Plugins -- Data-loader, Optimizer, Model layers, Loss-function
- Multi-Modal Support -- Text, vision, audio, etc
- Native vRAM mgt -- Native mem mgt instead of GC to lower peak occ-mem and alloc-overhead
- Mixed-precision quantization -- FP16, w4a16, w8a16, etc supported per tensor/channel/block
- NPU dynamics -- enable NPU to support dynamic shape, MoE, control flow, flexible heterogeneous compute
refft.cpp build tools compile a LLM/LM into a single native executable, one per backend: refft-hexagon today, with refft-cuda and refft-mlx coming. They all expose the same commands and the same OpenAI-compatible API, so the Quick Start below uses refft-hexagon as the example.
The previous generation of packages, built around the refft-cli / refft-server binaries, is no longer supported and is archived under Legacy.
| Runtime | Target | Status |
|---|---|---|
refft-hexagon |
Qualcomm Hexagon NPU (Snapdragon) on Android, arm64 Linux and Windows | Available |
refft-cuda |
NVIDIA GPU on x86_64 Linux | Coming |
refft-mlx |
Apple Silicon on macOS arm64 | Coming |
The installer picks the right package automatically. Direct downloads if you prefer:
| Runtime Package | Description |
|---|---|
| refft-hexagon_android-aarch64 | Android arm64, Hexagon HTP runtime + DSP skel |
| refft-hexagon_ubuntu-arm64 | arm64 Linux, Snapdragon dev kits |
| refft-hexagon_windows-arm64 | Windows on Snapdragon |
Releases are tagged vYYYY.MM.DD.NN: year, month, day, and the n-th release of that day. A Hexagon package ships one variant (v73, v81, etc) per SoC generation. Names follow refft-<runtime>_<os>-<arch>[-<variant>], so the upcoming refft-cuda_ubuntu-x86_64 and refft-mlx_macos-arm64 install and run the same way.
Three steps: install, run, chat.
One command installs the newest release for your platform. No version to pick, no path to configure.
Linux, or Android from a host with adb (also works inside Termux):
curl -fsSL https://raw.githubusercontent.com/refinefuture-ai/refft.cpp/main/refft-hexagon/install.sh | shWindows (PowerShell):
irm https://raw.githubusercontent.com/refinefuture-ai/refft.cpp/main/refft-hexagon/install.ps1 | iexIt prints Installation complete and where the runtime landed:
| Platform | Default location |
|---|---|
| Linux | ~/.local/share/refft-hexagon, or /opt/refft-hexagon as root |
Android, pushed over adb |
/data/local/tmp/refft-hexagon |
| Windows | %LOCALAPPDATA%\Programs\refft-hexagon |
Options such as --prefix, --tag and --hexagon-version are documented in refft-hexagon/README.md.
Everything is one binary — refft-hexagon — with three commands: cli, serve, bench.
The examples use refinefuture-ai/SmolVLM2-500M-Video-Instruct-REFFT, a Hexagon serving bundle. Download it first:
hf download refinefuture-ai/SmolVLM2-500M-Video-Instruct-REFFT --local-dir ~/SmolVLM2-500M-Video-Instruct-REFFT--model points at the bundle directory, the one holding serving_manifest.json.
Linux / Termux, from the folder printed by the installer:
cd ~/.local/share/refft-hexagon
export LD_LIBRARY_PATH=$PWD/lib
export ADSP_LIBRARY_PATH=$PWD/lib
export REFFT_HEXAGON_MODULE_PATH=$PWD/lib/librefft_hexagon_v73.so # match the .so in lib/
./bin/refft-hexagon help
./bin/refft-hexagon cli --model ~/SmolVLM2-500M-Video-Instruct-REFFT/SmolVLM2-500M-Video-Instruct-w4.refft --backend hexagon --prompt "Who are you?" --max_new_tokens 128
./bin/refft-hexagon serve --model ~/SmolVLM2-500M-Video-Instruct-REFFT/SmolVLM2-500M-Video-Instruct-w4.refft --backend hexagon --port 8080Android over adb:
adb push ~/SmolVLM2-500M-Video-Instruct-REFFT/SmolVLM2-500M-Video-Instruct-w4.refft /data/local/tmp/
adb shell
cd /data/local/tmp/refft-hexagon
export LD_LIBRARY_PATH=$PWD/lib
export ADSP_LIBRARY_PATH=$PWD/lib
./bin/refft-hexagon cli --model /data/local/tmp/SmolVLM2-500M-Video-Instruct-w4.refft --backend hexagon --prompt "Who are you?" --max_new_tokens 128Windows (PowerShell):
cd $env:LOCALAPPDATA\Programs\refft-hexagon
$env:ADSP_LIBRARY_PATH = "$PWD\lib"
.\bin\refft-hexagon.exe cli --model $HOME\SmolVLM2-500M-Video-Instruct-REFFT\SmolVLM2-500M-Video-Instruct-w4.refft --backend hexagon --prompt "Who are you?" --max_new_tokens 128
.\bin\refft-hexagon.exe serve --model $HOME\SmolVLM2-500M-Video-Instruct-REFFT\SmolVLM2-500M-Video-Instruct-w4.refft --backend hexagon --port 8080| Topic | Options |
|---|---|
| Commands | help, version, cli, serve, bench serve --prompt "Who are you?" |
| Generation | --max_new_tokens <n> (default 1 for cli, 16 for serve), --chat, --stream true, --print_perf_metrics |
| Sampling | pass --sampler chain first, then --do_sample, --temperature, --top_k, --top_p, --min_p, --repetition_penalty, --ignore_eos |
| Model | serving bundle from refinefuture-ai/SmolVLM2-500M-Video-Instruct-REFFT; the binary's built-in builders are listed by cli --list_models |
Once refft-hexagon serve is up:
curl http://127.0.0.1:8080/healthcurl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "SmolVLM2-500M-Video-Instruct",
"messages": [{"role": "user", "content": "Explain KV cache reuse briefly."}],
"max_tokens": 128,
"stream": false
}'The endpoint is OpenAI-compatible, so any OpenAI client works by pointing its base URL at http://127.0.0.1:8080/v1.
Why refft.cpp implements all of modeling, serving and training in C++
It's manly for a better performance and easy-to-use compared to Python/PyTorch-based as well as for scalability on edge-NPU.
Why Triton is not used in refft.cpp
Because the Triton models can get up to 78% of the performance of the CUDA models on the H100 and up to 82% on the A100.
How to support multi-nodes GPU/NPU
Technically refft.cpp supports multi-nodes inference and training, while multi-nodes haven't been tested due to lacking of HW resources. Please contact us if needed.Strictly equivalence of computational precision matters the most in LLM/LM's ops and serving optimization
https://epoch.ai/gradient-updates/why-benchmarking-is-hardhttps://blog.vllm.ai/2025/10/28/Kimi-K2-Accuracy.html
The previous generation, the old QNN / CUDA / MLX packages built around the refft-cli / refft-server binaries, the training flow and the build tools showcase, is no longer part of the supported surface and has been archived:
- docs/legacy/ -- retired packages and their usage
- docs/legacy/install_and_usage.md -- the old install and usage guide
- docs/legacy/training.md -- training of LLM/LM: datasets,
refft train, sample output - docs/legacy/build_tools.md -- the
refft.cppbuild tools showcase
CUDA and Apple Silicon are not going away: they return as the single-binary refft-cuda and refft-mlx runtimes. For anything current, use refft-hexagon as described in Quick Start.
Please contact us via haiteng@refinefuture.ai for commercial uses, technical consulting, sponsorship/partnership opportunities, etc.
refft.cpp was inspired by Andrej Karpathy' llm.c, and also referred to HuggingFace, PyTorch, vLLM, SGLang, FlashAttention, FlashInfer.