Skip to content

Missing --jinja/--no-jinja flags in llama.cpp plugin after refactor #2669

Description

@AlexonOliveiraRH

Description

After the refactor from YAML inference specs to Python plugins, the --jinja/--no-jinja flag logic was lost from the llama-server command builder. This causes:

  • Text models (granite, phi4): missing --jinja + --chat-template-file, leading to template issues
  • Multimodal models (smolvlm): receiving neither --no-jinja nor the mmproj-specific exclusion of --chat-template-file, causing tokenization errors

Steps to Reproduce

  1. ramalama serve --port 8080 --name multi-modal -d smolvlm
  2. Open the camera demo page
  3. Submit an image for inference
  4. Get error: Server error: 400 - Failed to tokenize prompt with number of bitmaps (1) does not match number of markers (0)

Root Cause

In ramalama/plugins/runtimes/inference/llama_cpp_commands.py, the _cmd_run() method lost the jinja logic that was previously defined in the YAML spec (inference-spec/engines/llama.cpp.yaml). The old YAML spec had:

  • Text models: --jinja + --chat-template-file {path}
  • Multimodal models (with mmproj): --no-jinja and NO --chat-template-file

Additionally, _get_mmproj_path() returns an empty string "" in dryrun mode (which is falsy in Python), so a double-check is needed: bool(mmproj_path) or bool(model._get_mmproj_path(False, False, False)).

Expected Behavior

The command builder should:

  1. Detect if the model has a mmproj file (multimodal)
  2. If multimodal: add --mmproj path + --no-jinja
  3. If text-only: add --chat-template-file (if available) + --jinja

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions