Skip to content

Add is_model_splitted interface#71

Merged
zhaixuejun1993 merged 5 commits intoravi9:dev_backend_openvinofrom
zhaixuejun1993:xuejun/ov-bk-add-func-is-splited-model
Mar 19, 2026
Merged

Add is_model_splitted interface#71
zhaixuejun1993 merged 5 commits intoravi9:dev_backend_openvinofrom
zhaixuejun1993:xuejun/ov-bk-add-func-is-splited-model

Conversation

@zhaixuejun1993
Copy link
Collaborator

This pull request adds support for detecting and handling split-model computation graphs ("splitted" models) in the OpenVINO GGML decoder integration. The changes introduce a heuristic function to determine if a computation graph is a split subgraph and propagate this information through the decoder's API and logic. This enables more robust and accurate model handling, especially for cases where models are partitioned or composed of multiple subgraphs.

Key changes include:

Split-model detection and handling:

  • Introduced the is_model_splitted function in utils.cpp and its declaration in utils.h, which heuristically checks if a ggml_cgraph is a split-model fragment. This function is now used in the dynamic compute path to determine whether to use naive computation or not. [1] [2] [3]

  • Updated the GgmlOvDecoder class and its constructor to accept and store a model_is_splitted flag, and added the is_splited_model() virtual method to the decoder interface. This allows downstream components to query if the current model is split. [1] [2] [3] [4] [5]

Integration with compute logic:

  • Modified the dynamic and static compute routines in utils.cpp to utilize the new split-model detection logic and to correctly construct GgmlOvDecoder instances with the appropriate model_is_splitted flag. [1] [2] [3]

@zhaixuejun1993 zhaixuejun1993 merged commit be67f32 into ravi9:dev_backend_openvino Mar 19, 2026
94 of 117 checks passed
@wine99
Copy link
Collaborator

wine99 commented Mar 19, 2026

@zhaixuejun1993 The split check seems to be expensive when running with quantized models

llama-simple -m llama-3.2-1B-Q4_0:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants