Video generation via stable-diffusion.cpp

[Bekhouche]

Proposal size: Major feature (spans multiple well-scoped PRs)

Note: The first phase is already written and open as PR #3506. I am writing this RFC now to follow the new policy.

User story

As someone running Lemonade locally, I want to create short videos from a text prompt, so I can use a local video model in the same way I already use local models for images and speech.

Today this is not possible, even though the engine is already installed. Lemonade includes stable-diffusion.cpp, and this backend already supports video models such as Wan and LTX. But Lemonade does not make these models available to users. Cloud model discovery also skips video models. So right now there is no way to generate a video in Lemonade.

This was requested in discussion #2981 in August, but there was no answer for about four weeks. The request had two main parts: an API endpoint for video generation, and a way to view the video in the desktop app.

High-level design

Add a new video endpoint that takes a text prompt and returns a short video. It sits next to the existing image, speech, and 3D endpoints.

For the user it works like image generation: send a prompt, get a result.

• A model says it is a video model by using a label. The server then sends requests to the right backend, the same way it does for image and speech models.
• Video models have their own model slot. Loading a video model does not unload the chat model the user is talking to.
• One video model is included in the catalog. The feature works after a fresh install, and the user does not have to add a model first.
• The user sends one request and waits for the video. The backend works in the background, and Lemonade waits for it. The user does not need to check for the result.
• A video model says which text encoder it needs. Lemonade sends it to the right place instead of guessing from the file name. The wrong place gives bad output and no clear error.

This first phase covers text to video only.

stable-diffusion.cpp also supports other video modes: image to video, first-and-last-frame to video, and video to video using control frames. Wan also has a VACE mode for video editing. These are not part of phase 1, but the endpoint should be able to grow into them later.

There is one open question here. Text to video only needs a JSON request. The other modes need an input image or video, so they may need a different request format. Lemonade already does this for images: /images/generations takes JSON, and /images/edits takes multipart form data. I would like feedback on whether to plan for this now, or to add a separate endpoint later.

Breaking changes

None. This only adds a new endpoint, a label, and a catalog entry. Existing APIs and behaviour do not change.

Maintenance plan

I will maintain the video part of the stable-diffusion.cpp backend. I have worked with generative AI and diffusion models for the last two to three years, so this is the area I can support best.

I am also happy to help with image generation issues when needed, but video is the part I commit to maintain.

Automated testing will cover less than usual, and I want to be clear about this.

CI can test that:

• The endpoint exists.
• Bad requests are rejected.
• The video capability is connected correctly.

These tests do not download models, so they can run on normal CI runners.

CI cannot test video generation itself. The test needs about 8.5 GB of model weights and a GPU, because CPU inference is too slow. One short video takes about three minutes on a discrete GPU.

There is also a limit on what a test can check. A test can check that the file is a real video, that it has the right number of frames, and that the frames are not all the same. So a single image repeated many times would fail. What a test cannot check is quality. It cannot tell if the video matches the prompt or looks right. A person has to watch it for that.

So the generation test is optional and needs a running server with a GPU. For most changes the automated checks are enough. When a change can affect the output itself, or when a new model is added, someone should also watch a video. I have a discrete GPU and I can do this.

Ongoing maintenance

• The video route in the backend is different from the OpenAI route. Changes from upstream have to be handled here.
• The text encoder family for each model has to be tracked as stable-diffusion.cpp adds new families.
• Adding a model that works like the current one only needs a catalog entry, with no code changes.
• Model families that need extra components will need code changes.

Risks

• One version (quantization) of the encoder can crash the backend when it loads. The user sees a restart, not a clear error. The catalog uses a version that is known to work.
• Video generation keeps the HTTP connection open for several minutes and shows no progress.
• There is no way to cancel a running generation. Unload can report success while the work continues in the background.
• The video comes back inside the response. Responses can be several megabytes, and longer videos make them bigger.
• Video models are large. They keep their slot while they are loaded.

The second and third points are the main things I would fix next, in the progress and cancellation phase below.

Issue #3121 already asks for progress reporting in the stable-diffusion backend. That phase would also answer it for video.

Phases

Phase 1 is already open as PR #3506. After that:

• Progress reporting and cancellation, so a long generation can be watched and stopped.
• Support for more model families.
• More input modes: image to video, first-and-last-frame to video, and video to video.
• Showing a video in the desktop app. This is the second half of Any plans for video generation support? #2981, which is still open.
• Adding it as an MCP tool, which RFC RFC: Make Lemonade capabilities server-owned and reusable through MCP #3426 says capabilities should be.
• Cloud video models. Today cloud discovery only accepts chat models, so a video model from a provider is skipped. Opening this would let a user mix local and cloud video, in the same way Lemonade already mixes local and cloud text.
• Routing between video models. Lemonade can already choose between text models with a router policy. The same idea fits video well: a small fast model for simple prompts, a bigger one when the user needs more quality, or a local model first and a cloud model only when needed.

The last two are a direction, not a promise for the next PR. I mention them because they show the endpoint is not a dead end, and because video has the same small model or big model choice that text already has.

One more thing, better to say now than to find later: the desktop app follows the list of labels in the docs, and it checks how many labels there are. Adding a video label needs a change on the app side at the same time, or the app build breaks after the next merge.

[Bekhouche]

@ramkrishna2910 @jeremyfowers @bitgamma this is the RFC for #3506, following the new policy.
@pwilkin @fl0rianr tagging you too since you are already on the PR.

[jeremyfowers]

Thanks for opening this!

[pwilkin]

@Bekhouche can you describe the quantization crash in more detail (or point me to an upstream issue)? I have a feeling I fixed that one already in one of my branches, so might as well push a fix upstream and get that one out of the way.

[Bekhouche]

@pwilkin Sure. It is not a load crash, I had that wrong in the RFC.

sd-cpp master-827-97d2990, Vulkan, RX 7900 XTX 24 GB, Windows. Wan2.1-T2V-1.3B fp16 + umt5-xxl-encoder-Q5_K_M + wan_2.1_vae, with --vae-tiling --diffusion-fa.

The model loads fine (10083 MB total, text encoder 7132 MB). It dies on the first generation, in T5 conditioning:

ggml_vulkan: Device memory allocation of size 4200595456 failed.
ggml_vulkan: Requested buffer size exceeds device buffer size limit: ErrorOutOfDeviceMemory
[ERROR] model_manager.cpp:738 - model manager alloc params backend buffer failed, size = 4006.00MB
[ERROR] ggml_extend.hpp:1989 - t5 prepare graph weights failed
conditioner.hpp:1514: GGML_ASSERT(!chunk_hidden_states.empty()) failed
The process aborts rather than returning an error, so from Lemonade it looks like the backend just went away.

The odd part: Q8_0 works on the same GPU with the same flags, and it is the bigger file (5.7 GB vs 3.9 GB). So it does not look like a VRAM shortage, more like the k-quant path needing one large contiguous buffer.

No upstream issue for this exact case. leejet/stable-diffusion.cpp#1718 looks like the same mechanism with a different conditioner, and #942 is Wan on Vulkan.

[ramkrishna2910]

Thanks for opening this @Bekhouche.
This would be a good addition. In regards to CI testing we can provide support with stx halo hardware for this feature. Given how long the test takes we will have to decide how frequently we want this to run.

[bitgamma]

@Bekhouche please detail the new proposed API with all fields. Ideally, we want to be at compatible with https://developers.openai.com/api/docs/guides/video-generation at least for the base use-cases

[Bekhouche]

@bitgamma Here are all the fields, split by phase, so it is clear what #3506 ships now and what the API needs to grow into later.

Phase 0: what already exists

stable-diffusion.cpp is already inside Lemonade, and it already supports video. Nothing exposes it, so the feature is there but users cannot reach it. #3506 does not add an engine. It opens the one we already install.

Phase 1: POST /v1/videos/generations, open now as #3506

Text to video only. JSON body, registered under the usual four prefixes.

Field	Required	Type	Default
prompt	yes	string	none
model	yes	string	none
negative_prompt	no	string	none
video_frames	no	int	recipe option, then 33
fps	no	int	recipe option, then 24
width / height	no	int	recipe option, then backend default
steps	no	int	recipe option
cfg_scale	no	number	recipe option
sampling_method	no	string	recipe option
flow_shift	no	number	recipe option
seed	no	int	random
output_format	no	string	webm (also webp, avi)

The response is one JSON object with b64_json, mime_type, output_format, frame_count and fps.

Phase 2: progress and cancellation

This adds no new generation fields. It changes the response shape. The server returns a job instead of a finished video.

This is a smaller change than it looks. /sdcpp/v1/vid_gen is already asynchronous. It answers with 202 and a poll URL. Today I poll it in a loop and hide this from the caller. So showing the job means removing that loop, not writing new code. Lemonade also already has a job engine with pause, interrupt and resume. This phase fixes three risks I listed above: no progress, no way to cancel, and very large responses. It also answers #3121.

Phase 3: more input modes

These are the fields that Phase 1 does not have. The backend already supports all of them. Each one needs a file as input, so they change the request shape.

Mode	Backend input	New fields
Image to video	--init-img	one reference image
First and last frame	--init-img + --end-img	a second reference image
Video to video / control	--control-video, --control-image, --control-strength	a reference video, plus strength
Video editing (Wan VACE)	--vace-strength	a source video, plus strength
Reference guided	-r/--ref-image (can repeat), --ip-adapter-image	one or more reference images
Ref2VA (MiniMax-H3)	--ref-video, --ref-audio, --ref-video-audio	reference video and audio

How this fits the OpenAI guide

The OpenAI shape covers Phase 1 and part of Phase 3. It has text to video, and image to video through input_reference. Remix is close to video editing. But end frame, control video, many reference images and reference audio have no OpenAI field. Those would stay as Lemonade extras. So we can be compatible for the common cases, but not for everything the backend can do.

There are four places where Phase 1 does not match today, and I think you are right that it should:

1. Sync or async. OpenAI returns a job and the client checks it. Mine keeps the HTTP connection open for several minutes. This is Phase 2.
2. Route. OpenAI uses /v1/videos. I used /v1/videos/generations to match our image routes.
3. Length and size. OpenAI uses seconds and size. I use video_frames with fps, and width with height, because that is what the model takes.
4. How the file is returned. OpenAI serves the bytes from a content endpoint. I return base64 inside the response, like our image endpoints.

Point 4 becomes more important in Phase 3. Reference images and videos do not fit well in a JSON body, so the create call needs multipart, like OpenAI does for input_reference. We already do this with images: /images/generations is JSON and /images/edits is multipart.

What I propose

Move the Phase 2 shape into #3506 now: POST /v1/videos that accepts JSON or multipart and returns id, status, progress, created_at and model, then GET /v1/videos/{id} to check it, and GET /v1/videos/{id}/content to get the file. Phase 1 would still ship text to video only, but the shape would then have room for Phase 3 without a breaking change later.

Two questions:

• Names. Should seconds and size be the documented names, and we convert to frames using fps, keeping video_frames, width and height as extras? Or should we keep the model names as the main ones?
• Scope. Should the Phase 2 shape go into feat(backends): video generation via stable-diffusion.cpp #3506, or should feat(backends): video generation via stable-diffusion.cpp #3506 stay as it is and Phase 2 come next? I prefer to do it now, so we do not ship an API and then change it.

Thanks also @ramkrishna2910 for the Strix Halo CI offer. That covers the exact gap I wrote about in the maintenance plan, and I will take it up.

[fl0rianr]

I like the approach but would like to address one thought on the API shape:S
Since OpenAI's Videos API is gone without a replacement (recently, a week ago), I wouldn't treat strict OpenAI compatibility as a hard requirement anymore.

I would still keep familiar OpenAI-style naming where it makes sense, but define Lemonade's video API around generic async job semantics from day one. Video generations can take minutes, sd.cpp is already async internally, and progress/cancellation are much easier to support cleanly that way than by starting with a synchronous API and changing it later.

Hugging Face's Inference Providers may also be worth looking at as another reference: they already abstract text-to-video across multiple providers, but use their own task API rather than extending OpenAI compatibility to video:
https://huggingface.co/docs/inference-providers/tasks/text-to-video

For the public API I'd also keep the common path backend-neutral and user-facing, e.g. duration and size, while exposing things like frames, fps, width/height and sd.cpp-specific controls as optional advanced parameters rather than making backend details part of the core API.

This also fits #3426 nicely: the server should own the generation lifecycle and backend-specific orchestration, while clients/MCP interact with one stable capability instead of knowing how a particular backend handles jobs internally.

[pwilkin]

Do we already have async image generation with progress polling?

[fl0rianr]

No, not currently. Image generation is synchronous at the Lemonade API level.

That might actually be an argument for making the async job abstraction generic enough to cover image generation too, rather than introducing something video-specific. Video is a perfect example where async makes sense.

[Bekhouche]

@fl0rianr @pwilkin Thank you both.

I agree the abstraction should be generic. I would separate two things that are easy to merge together:

1. making the job resource generic, so nothing in the lifecycle or the status vocabulary assumes video;
2. making it cover image generation in the same change.

The first one is a small change and I am happy to do it in #3506. The second one is a different kind of change, since /v1/images/generations is already public, documented and synchronous. Moving it to a job response breaks existing clients, so it needs a deprecation path rather than a redesign inside this PR.

This is also why video is a good place to define the shape: it has no clients yet, so the async form is still free to choose. Image generation does not have that property any more.

So I propose to define the job resource generically in #3506, ship it on video first, and let image generation adopt it later in its own PR. If that is agreed, I will avoid any video-specific naming so the same resource can carry image, audio and 3D later. I can also take the image part myself once the shape is settled, so the two stay consistent.

One clarification on my previous comment. When I mentioned the existing job engine I meant it only as a precedent. I am not proposing to run the generation through it: its ops are system_info, system_stats, models, sleep, load, unload and chat, and chat is exclusive and holds a Router slot. It is an orchestration harness for server-side sequences, so driving a generation through it would give clients a different shape from asking for a video and getting a handle. I think a capability-level job resource fits better, but I am open to the other reading.

One correction first. The API that shut down on 24 September is OpenAI direct. Azure OpenAI still documents Sora 2 video generation, in preview, and it is job based: POST /openai/v1/video/generations/jobs returns an id, the job is polled for succeeded, failed or cancelled, and the video is fetched from a separate content route. So the OpenAI shape is not gone from the ecosystem, and @bitgamma's request is still reasonable.

References. I checked five:

1. Azure OpenAI Sora 2, above. Body is prompt, width, height, n_seconds, n_variants and model.
2. Replicate predictions: id, status with starting, processing, succeeded, failed and canceled, plus output, error, logs, created_at and completed_at. Cancel is POST /v1/predictions/{id}/cancel, and there is no delete.
3. fal queue: request_id, status with IN_QUEUE, IN_PROGRESS and COMPLETED, plus queue_position, logs and metrics. Cancel is a PUT on the request id.
4. Google Veo in the Gemini API. Also asynchronous, but it uses a long-running operation with a done flag rather than a status string.
5. Hugging Face Inference Providers, the one you suggested. It is a task API rather than a job API: inputs with parameters holding num_frames, num_inference_steps, guidance_scale, negative_prompt and seed, and the documented response is raw video bytes.

All five are asynchronous. Three of them also use a status string and a separate route for the bytes, which is the part I would copy:

POST   /v1/videos               -> id, status, created_at, model
GET    /v1/videos/{id}          -> status, progress, error
GET    /v1/videos/{id}/content  -> the encoded video
POST   /v1/videos/{id}/cancel   -> stops the run, keeps the record

status: queued | running | completed | failed | canceled

Cancel is a POST rather than a DELETE, because DELETE would read as removing the record instead of stopping the run, and Replicate uses a POST sub-route for the same reason. The status vocabulary maps directly to what we already have, as sd-cpp reports queued, generating and running on its own poll URL today. Webhooks exist in Replicate and in the Gemini API, but I would leave them out of phase 1.

On duration and size. I checked this against the references and I would revise what I said earlier. Azure OpenAI uses width and height as separate integers with n_seconds, not a combined size string, and Hugging Face has no size field at all. So separate width and height is already the common form, and it is what we have today. I would keep them.

Duration is the part worth adding. Both n_seconds and the retired seconds show that seconds is the public unit. The difficulty is that the model takes a frame count, so duration is frames divided by fps. We would have to pick an fps to convert, and models accept only certain frame counts, so the returned clip would not be exactly the requested length. I suggest accepting duration and documenting it as approximate, with video_frames and fps kept as the exact controls.

@bitgamma, does this work for you: the job shape above, width and height as Azure has them, and duration in seconds as the neutral field?

[fl0rianr]

With current main, I think there is an important connection here: Lemonade already has the generic /v1/jobs system, and its design explicitly says it is modality-agnostic and that image generation/TTS/etc. should be additive ops.

So we could avoid creating a second independent job implementation specifically for /v1/videos. This can still be the clean public video API, but internally it could use/bridge the existing JobManager with a generate_video op. The same mechanism could then be reused for images later.

One caveat though: the current JobManager does not yet seem to provide fine-grained progress inside a running op. pause is step-boundary based, and sd.cpp's native async job API currently exposes queued/generating/completed + cancellation, but not a documented percentage. So I would keep progress optional until we have a real backend signal rather than inventing a percentage.

That also seems like a good answer to the image question: image generation is synchronous in Lemonade today, but sd.cpp already exposes both image and video through the same native async job API, so this could be the point where we add the generic bridge once and use it for both.

[Bekhouche]

You are right, I was wrong about the job engine. docs/dev/job-system.md says it is modality-agnostic and names SdServer with generate_image and generate_3d as examples, so a generate_video op is the intended way rather than a second job implementation. I will take that route, with /v1/videos still the public API.

I agree on progress. I checked the sd-server binary: it exposes /sdcpp/v1/img_gen, /sdcpp/v1/vid_gen and /sdcpp/v1/jobs/, and the job states are queued, generating, completed, failed and cancelled, with cancellation but no progress field. So progress stays optional until the backend reports one.

Two small things from the same check:

1. our poll loop also accepts running, which sd-server never emits, so I will remove it;
2. sd-cpp spells it cancelled while Replicate uses canceled, so we need to pick one and map.

Thank you for the correction.