Description
After the refactor from YAML inference specs to Python plugins, the --jinja/--no-jinja flag logic was lost from the llama-server command builder. This causes:
- Text models (granite, phi4): missing
--jinja + --chat-template-file, leading to template issues
- Multimodal models (smolvlm): receiving neither
--no-jinja nor the mmproj-specific exclusion of --chat-template-file, causing tokenization errors
Steps to Reproduce
ramalama serve --port 8080 --name multi-modal -d smolvlm
- Open the camera demo page
- Submit an image for inference
- Get error:
Server error: 400 - Failed to tokenize prompt with number of bitmaps (1) does not match number of markers (0)
Root Cause
In ramalama/plugins/runtimes/inference/llama_cpp_commands.py, the _cmd_run() method lost the jinja logic that was previously defined in the YAML spec (inference-spec/engines/llama.cpp.yaml). The old YAML spec had:
- Text models:
--jinja + --chat-template-file {path}
- Multimodal models (with mmproj):
--no-jinja and NO --chat-template-file
Additionally, _get_mmproj_path() returns an empty string "" in dryrun mode (which is falsy in Python), so a double-check is needed: bool(mmproj_path) or bool(model._get_mmproj_path(False, False, False)).
Expected Behavior
The command builder should:
- Detect if the model has a mmproj file (multimodal)
- If multimodal: add
--mmproj path + --no-jinja
- If text-only: add
--chat-template-file (if available) + --jinja
Description
After the refactor from YAML inference specs to Python plugins, the
--jinja/--no-jinjaflag logic was lost from the llama-server command builder. This causes:--jinja+--chat-template-file, leading to template issues--no-jinjanor the mmproj-specific exclusion of--chat-template-file, causing tokenization errorsSteps to Reproduce
ramalama serve --port 8080 --name multi-modal -d smolvlmServer error: 400 - Failed to tokenize promptwithnumber of bitmaps (1) does not match number of markers (0)Root Cause
In
ramalama/plugins/runtimes/inference/llama_cpp_commands.py, the_cmd_run()method lost the jinja logic that was previously defined in the YAML spec (inference-spec/engines/llama.cpp.yaml). The old YAML spec had:--jinja+--chat-template-file {path}--no-jinjaand NO--chat-template-fileAdditionally,
_get_mmproj_path()returns an empty string""in dryrun mode (which is falsy in Python), so a double-check is needed:bool(mmproj_path) or bool(model._get_mmproj_path(False, False, False)).Expected Behavior
The command builder should:
--mmprojpath +--no-jinja--chat-template-file(if available) +--jinja