feat(mtmd): add SMT backend multimodal inference support for llama-server and llama-mtmd-cli

[co-seven]

Summary

This PR adds multimodal large-model inference feature support on the Spacemit SMT backend, targeting llama-server and llama-mtmd-cli in the spacemit-mtmd branch.

The main goal is to enable SMT-based multimodal inference workflows on Spacemit platforms, including server-side and CLI-side integration, runtime backend selection, model/config compatibility updates, and related usability improvements for
image/audio pipelines.

What is included

• Add SMT backend multimodal integration for llama-server
• Add SMT backend multimodal integration for llama-mtmd-cli
• Support runtime backend selection and related config handling
• Add SMT image/audio processing path and warmup support
• Add profiling support for vision/audio encoder stages
• Add timing summary for vision/audio encode stages
• Add SMT-only multimodal chat controls such as vision_history
• Improve backend loading and thread/config handling
• Add compatibility updates for related multimodal model paths and configs
• Add support updates for:
  • Qwen3.5-related model arch handling
  • Qwen3-ASR SMT audio path
  • PaddleOCR-VL SMT vision pipeline
• Add tokenizer/config fallback and encoder input-size compatibility fixes
• Update build and usage documentation for SMT / multimodal workflows

User-facing impact

This PR enables multimodal inference feature support on the SMT backend for Spacemit targets, including:

• image understanding through SMT vision integration
• audio-capable multimodal workflows on supported models
• SMT-backed server inference through llama-server
• SMT-backed CLI inference through llama-mtmd-cli
• improved deployment/configuration usability for multimodal inference on Spacemit platforms

Notes

• This PR focuses on SMT backend multimodal feature enablement and related integration work.
• The submitted changes are the selected feature/fix commits prepared for upstreaming into spacemit-mtmd.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds SMT (SpacemiT) multimodal inference support across llama-server and llama-mtmd-cli, including runtime backend selection and profiling hooks.

Changes:

• Introduces SMT vision/audio wrappers and server-side SMT media encode/decode integration.
• Adds CLI SMT backend entrypoint and CMake wiring for SpacemiT ORT bundle builds.
• Adds ggml trace/profiling events and exposes backend metadata via server routes.

Reviewed changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tools/server/server-smt-vision.h	Declares SMT media encoding/decoding API and fallback stubs when disabled.
tools/server/server-smt-vision.cpp	Implements SMT vision/audio encode + LLM embedding decode path and token boundary detection.
tools/server/server-context.h	Adds media_backend to server metadata.
tools/server/server-context.cpp	Adds backend selection (mtmd vs smt), SMT prompt processing, encoder timing stats, and exposes backend in responses.
tools/server/server-common.h	Extends server_tokens to carry SMT media chunks and adds SMT prompt API surface.
tools/server/CMakeLists.txt	Adds LLAMA_SERVER_SMT_VISION build option wiring and links ORT + SpacemiT sample sources.
tools/mtmd/smt-vision-wrapper.h / .cpp	Adds SMT vision engine wrapper (config parsing, warmup, encode).
tools/mtmd/smt-vision-preprocess.h / .cpp	Adds SMT image preprocessing from common image formats to float32 NCHW tensor bytes.
tools/mtmd/smt-audio-wrapper.h / .cpp	Adds SMT audio split-encoder wrapper with ONNXRuntime + optional SpacemiT EP init.
tools/mtmd/mtmd-cli.cpp	Routes to SMT CLI backend based on params.
tools/mtmd/mtmd-cli-smt.cpp	New SMT-backed multimodal CLI implementation.
tools/mtmd/mtmd-audio.h / .cpp	Exposes a reusable log-mel spectrogram helper for SMT audio path.
tools/mtmd/CMakeLists.txt	Adds SMT-related sources/includes/libs to mtmd CLI build when enabled.
src/llama-context.cpp	Adds trace begin/end events around llama_decode.
ggml/src/ggml-profile.c	Adds JSON trace emitter implementation behind GGML_BUILD_PROFILE.
ggml/include/ggml-profile.h	Adds profiling/trace API surface (enabled via GGML_BUILD_PROFILE).
docs/build-riscv64-spacemit.md	Updates build/run docs for SMT enablement and usage.
convert_hf_to_gguf.py	Adds tokenizer loading fallback logic and Qwen3-ASR text arch handling.
common/common.h	Adds SMT backend selection params under LLAMA_SERVER_SMT_VISION.
common/arg.cpp	Adds --media-backend/--vision-backend and --smt-config-dir CLI options.
CMakeLists.txt	Adds LLAMA_SERVER_SMT_VISION option and compile definition.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 27 changed files in this pull request and generated 10 comments.