Skip to content

Enable OpenVINO backend fallback to CPU backend#130

Open
zhaixuejun1993 wants to merge 1 commit intoravi9:dev_backend_openvinofrom
zhaixuejun1993:xuejun/openvino-fallback-cpu-v1
Open

Enable OpenVINO backend fallback to CPU backend#130
zhaixuejun1993 wants to merge 1 commit intoravi9:dev_backend_openvinofrom
zhaixuejun1993:xuejun/openvino-fallback-cpu-v1

Conversation

@zhaixuejun1993
Copy link
Copy Markdown
Collaborator

This pull request introduces enhancements to tensor tracking and backend scheduling in the GGML library, primarily to improve handling of in-place operations and debugging. The changes add a new field to the tensor structure for tracking original source tensors, propagate this information through backend scheduling and OpenVINO decoder logic, and improve debugging by modifying tensor naming.

Tensor structure and tracking improvements:

  • Added an org_src pointer to the ggml_tensor struct to track the original source tensor in in-place operations, and increased the padding size to accommodate this change. (ggml/include/ggml.h, ggml/src/ggml.c) [1] [2]
  • When copying tensors during backend graph splitting, the org_src field is set to the source tensor, preserving provenance information. (ggml/src/ggml-backend.cpp)

Backend scheduling and debugging:

  • Improved backend assignment for view operations by ensuring backend IDs are propagated from view sources when supported, and by updating view nodes' backend IDs when necessary. (ggml/src/ggml-backend.cpp) [1] [2]
  • Enhanced debugging by appending node IDs to tensor names during backend scheduling, making it easier to trace tensors in logs or debugging output. (ggml/src/ggml-backend.cpp)

OpenVINO decoder logic:

  • Updated the OpenVINO decoder to utilize the new org_src field when determining dynamic dimensions and input/output relationships, ensuring correct handling of tensors originating from in-place operations. (ggml/src/ggml-openvino/ggml-decoder.cpp, ggml/src/ggml-openvino/ggml-decoder.h) [1] [2] [3]## Overview

Additional information

Requirements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant