Repository navigation
Stable Diffusion image generation on the AMD Ryzen AI NPU #3913
meghsat
started this conversation in
Request for Comment (RFC)
Replies: 1 comment 1 reply
|
There are more models that are supported that are missing from the list: Each model support text to image and image to image, image generation. |
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Proposal size
Minor feature (a single PR)
Updates
Have a working branch fully tested on STX: https://github.com/lemonade-sdk/lemonade/tree/meghsat/ryzen-sd-server
User story
Lemonade can already generate images on CPU, Vulkan, CUDA, ROCm and Metal through
sd-cpp. On a Ryzen AI laptop none of those paths use the NPU.This RFC adds a
ryzenai-sdrecipe that runs pre-converted ONNX Stable Diffusion models on the XDNA2 NPU, exposed through the same OpenAI-compatible Images API Lemonade already serves.High-level design
It follows the existing subprocess backend pattern (
docs/dev/adding-a-backend.md).RyzenAISDServerinheritsWrappedServerandIImageServer, spawnsryzenai-sd-serveras a child process, and proxies requests to it over loopback HTTP.The subprocess binary is not built by Lemonade. It is downloaded from a pinned GitHub release of
ryzenai-sd-server((https://github.com/lemonade-sdk/ryzenai-sd-server/releases/tag/), assetryzenai-sd-server.zip) on first model load, checksum-verified, and extracted into the Lemonade binary cache. The archive is self-contained, executable, ONNX Runtime, the VitisAI EP DLLs and the GenAI-SD library tree, so no RyzenAI SDK installation is required on the host. Only the AMD NPU kernel driver is needed[https://ryzenai.docs.amd.com/en/latest/inst.html#install-npu-drivers].Model handling
Models are ordinary Hugging Face repos of pre-converted ONNX files. Unlike GGUF backends, the server needs the snapshot root directory (containing
text_encoder/,unet/,vae_decoder/, …) rather than a single file, so the backend overridesresolve_checkpoint_path()to return the active snapshot directory.Registered models (
recipe: "ryzenai-sd"): [These are currently set to private]SD-Turbo-NPUamd/sd-turbo-amdnpu-onnxSDXL-Turbo-NPUamd/sdxl-turbo-amdnpu-onnxSDXL-Base-NPUamd/sdxl-base-amdnpu-onnxFLUX.1-Schnell-NPUamd/FLUX.1-schnell-amdnpu-onnxFLUX.2-Klein-NPUamd/FLUX.2-klein-amdnpu-onnxBreaking changes
None
Maintenance plan
backend_versions.jsonasryzenai-sd-serverreleases.xdna2runner alongside the other inference backends.lemonade-sdk/ryzenai-sd-server; integration issues here.Risks
No response
All reactions