fix(capi): vllm_server_main broke every VLLM_CPP_SERVER=OFF build - #202
Merged
Conversation
#189 added the vllm_server_main wrapper to src/capi UNCONDITIONALLY, but the translation unit it calls (src/vllm/entrypoints/openai/server_main.cpp) is compiled only under `if(VLLM_CPP_SERVER)` because it pulls in the vendored httplib transport. Any build with the server disabled therefore failed to LINK: Undefined symbols for architecture arm64: "vllm::entrypoints::openai::VllmServerMain(int, char**)", referenced from: _vllm_server_main in libvllm.a[255](vllm_c.cpp.o) Found downstream, again: LocalAI links libvllm with the server OFF (it wants the library, not the HTTP endpoint), and mudler/LocalAI#11424's darwin lane went red on the dylib link the moment it took a pin carrying #189. The SYMBOL must exist in every build — it is part of ABI v17, and a consumer that dlopen's the library and resolves entry points has to FIND it rather than fail to load. So the no-server arm reports the missing capability through SetError and returns non-zero, instead of the symbol going missing. Verified BOTH configurations this time, which is the actual lesson: my #189 verification ran only VLLM_CPP_SERVER=ON (the default), so the broken arm was never built. Now: * SERVER=ON — builds, `./server --help` runs through the ABI (#189) * SERVER=OFF — configure+link clean, libvllm.so produced, and `nm -D` confirms vllm_server_main is still exported alongside vllm_video_generate and vllm_transcribe FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
mudler
added a commit
to mudler/LocalAI
that referenced
this pull request
Aug 9, 2026
The previous pin carried vllm.cpp's ABI v17 (vllm_server_main) but not the guard that makes it link when the server is compiled out. This backend builds libvllm with VLLM_CPP_SERVER off, so the darwin lane failed at the dylib link with vllm::entrypoints::openai::VllmServerMain undefined. Fixed upstream in mudler/vllm.cpp#202: the C entry point is now guarded, so the symbol is still exported (ABI v17 stays resolvable for dlopen) while the no-server arm reports the missing capability instead of dragging in a translation unit that was never compiled. Verified upstream in BOTH arms before re-pinning: SERVER=ON builds and runs, and SERVER=OFF configures, links, produces libvllm.so, and `nm -D` shows vllm_server_main exported next to vllm_video_generate and vllm_transcribe. Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
mudler
added a commit
to mudler/LocalAI
that referenced
this pull request
Aug 9, 2026
* feat(vllm-cpp): serve MiniMax-H3 video+audio generation
vllm.cpp's C ABI grew a video slice (ABI v12): a second engine handle
loaded from the MiniMax-H3 checkpoint SET, one blocking generate, and a
composed ffmpeg argv the caller execs. This wires that into LocalAI's
existing /video endpoint, so `vllm-cpp` now serves both text and video
and a clip comes back as an MP4 with a real audio track rather than a
silent render.
The video engine is a separate handle rather than a mode of the text
one because H3 is not a model directory: the DiT, the text encoder and
two VAEs are separate artifacts, and vllm.cpp has the two loaders refuse
each other's checkpoints. `Load` takes the video branch when the config
declares any of the video options; `parameters.model` is the DiT and the
rest of the set is named in `options:`.
Three details are worth calling out because getting them wrong is
expensive:
- The partition is DECLARED, not detected. The community quantisations
strip the release metadata and the FL2VA and Ref2VA DiTs are
byte-structurally identical, so the engine refuses to generate until
it is told which it has. Worse, a mismatch does not fail cleanly: a
reference passed to an FL2VA DiT renders for hours and returns a
coloured lattice over the frame. The backend refuses that combination
up front instead.
- ffmpeg comes from the host. libvllm writes frames plus a WAV and
composes the mux argv, then spawns nothing - that process boundary is
upstream's decision. The backend execs it, the same arrangement
vibevoice-cpp uses for transcoding, and ffmpeg also converts a
start_image upload into the binary PPM at the exact output canvas the
engine requires.
- It is slow. Roughly 176 s per denoise step at the default 1344x768
canvas on a 20-SM device, so the 50-step default is a multi-hour job.
Nothing on this path imposes a deadline.
The /video endpoint no longer forces 512x512 when the request omits the
geometry. Every video backend already supplies its own default for a
zero (512x512 for stablediffusion-ggml, 1280x720 for diffusers, 832x480
for longcat-video, 1344x768 for H3), so the hardcoded value only ever
overrode the model's trained canvas with one three of the four were
never trained at.
Moving the engine pin from ABI v10 to v16 also grows the text
vllm_model_params mirror by the v14 device field and the v16 KV-sizing
knobs. LocalAI sets none of them - 0 is the pre-v14 engine byte for byte
- but the struct SIZE is part of the layout contract, so leaving them
out would have vllm_engine_load read past the allocation.
Gallery: `minimax-h3-fl2va-q4` installs the Q4_K_M FL2VA set (~40 GB
across five weight files plus the two VAE configs that carry the latent
statistics).
Assisted-by: Claude:claude-opus-5 golangci-lint yamllint go-vet
* fix(vllm-cpp): unbreak the Darwin build at the new engine pin
src/capi/vllm_c.cpp opens one `extern "C" {` for the whole ABI surface,
so file-local helpers declared inside it inherit C linkage. The video
slice added one that returns std::string, which Apple Clang reports as
-Wreturn-type-c-linkage and vllm.cpp's target-local -Werror turns into a
build failure. GCC and upstream Clang do not diagnose it, so only the
metal-darwin-arm64 job saw it.
Suppress it the same way this Makefile already suppresses Apple Clang's
-Wgnu-folding-constant on the Metal build. The helper is never called
across the boundary so the warning describes no hazard here, but it is a
real upstream wart: the fix belongs in vllm.cpp, hoisting the helper
above the extern "C" block, and this flag should go when a pin carrying
that fix lands.
Assisted-by: Claude:claude-opus-5
* fix(vllm-cpp): patch the engine clone instead of the warning flag
The -Wno-return-type-c-linkage added in the previous commit does nothing.
vllm_cpp_set_warnings adds `-Wall -Wextra -Werror` as PRIVATE target
options, so they land after anything CMAKE_CXX_FLAGS contributes, and
-Wall re-enables the -Wreturn-type group that -Wreturn-type-c-linkage
belongs to. The darwin job failed again on the same line, which is the
evidence: a consumer cannot wave this off from outside the engine.
Position is the only fix, so carry it as a patch against the pinned SHA,
the way longcat-video patches its own upstream. It hoists the helper
above the `extern "C" {` that gives it C linkage; it is file-local and
never called across the boundary, so nothing else moves.
`git apply` is unguarded on purpose: a patch that stops applying must
fail the clone loudly, because the alternative is a pin that silently
ships without a fix it is documented to carry. The patch header names
what retires it - a pin carrying the fix upstream, where it belongs.
Verified by applying the patch with `git apply` to the exact blob at the
pinned SHA and diffing the result against the intended file.
Assisted-by: Claude:claude-opus-5
* chore(vllm-cpp): bump the engine pin to ABI v17 and drop the vendored OrEmpty patch
The OrEmpty linkage fix this backend carried as patches/0001-* landed upstream
(mudler/vllm.cpp#195, 7534da65), so the patch has done its job. It is deleted
rather than left in place: the Makefile applies patches/*.patch unguarded and
documents that "a patch that no longer applies must FAIL the clone", so keeping
it against fixed source would break the build the moment the pin moved. Bumping
the pin and deleting the patch therefore have to be the SAME change.
Pin f921062b -> 776c56f1 (current vllm.cpp main).
That range also carries the engine's ABI v17 (vllm_server_main: the OpenAI server
published on the public surface). registerLib compares the library's
vllm_abi_version against `abiVersion` for EXACT equality, so the constant moves
16 -> 17 in the same commit or every load fails with an ABI mismatch.
The bump is safe for the layout assertions in video_test.go: diffing include/vllm.h
across the two pins shows zero struct-field changes -- v17 adds one function
declaration, the version macro and a doc comment, nothing else -- so every
unsafe.Offsetof in the video params test still holds.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* chore(vllm-cpp): re-pin to pick up the VLLM_CPP_SERVER=OFF link fix
The previous pin carried vllm.cpp's ABI v17 (vllm_server_main) but not the guard
that makes it link when the server is compiled out. This backend builds libvllm
with VLLM_CPP_SERVER off, so the darwin lane failed at the dylib link with
vllm::entrypoints::openai::VllmServerMain undefined.
Fixed upstream in mudler/vllm.cpp#202: the C entry point is now guarded, so the
symbol is still exported (ABI v17 stays resolvable for dlopen) while the
no-server arm reports the missing capability instead of dragging in a translation
unit that was never compiled.
Verified upstream in BOTH arms before re-pinning: SERVER=ON builds and runs, and
SERVER=OFF configures, links, produces libvllm.so, and `nm -D` shows
vllm_server_main exported next to vllm_video_generate and vllm_transcribe.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The break
#189 added the
vllm_server_mainwrapper tosrc/capiunconditionally, but the translation unit it calls (src/vllm/entrypoints/openai/server_main.cpp) compiles only underif(VLLM_CPP_SERVER)— it pulls in the vendored httplib transport. Any build with the server disabled failed to link:Found downstream, again: LocalAI links
libvllmwith the server off (it wants the library, not the HTTP endpoint), and mudler/LocalAI#11424's darwin lane went red on the dylib link the moment it took a pin carrying #189.The fix
The symbol must exist in every build — it's part of ABI v17, and a consumer that
dlopens the library and resolves entry points has to find it rather than fail to load. So the no-server arm reports the missing capability viaSetErrorand returns non-zero, instead of the symbol going missing.Verified both configurations
That's the actual lesson here. My #189 verification ran only
VLLM_CPP_SERVER=ON(the default), so the broken arm was never built at all.SERVER=ON./server --helpruns through the ABI (#189)SERVER=OFFlibvllm.soproducedAnd
nm -D --defined-onlyon the SERVER=OFF library confirmsvllm_server_mainis still exported, alongsidevllm_video_generateandvllm_transcribe— so ABI v17 stays honest in both arms.