-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactoring: add helper class to bind qnn tensor -> ggml tensor #2
base: qualcomm_qnn_backend_for_ggml
Are you sure you want to change the base?
Refactoring: add helper class to bind qnn tensor -> ggml tensor #2
Conversation
ggml-qnn.cpp
Outdated
QNN_LOG_WARN("alloc rpcmem failure, %s\n", strerror(errno)); | ||
QNN_LOG_DEBUG("tensor%p name %s", _qnn_tensor, QNN_TENSOR_GET_NAME(*_qnn_tensor)); | ||
_context = nullptr; | ||
// TODO: should we free the tensor here? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we free the _qnn_tensor
here create by tensorCreateGraphTensor
(line 1979)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
不需要。高通QNN SDK貌似没有提供类似的函数。看高通的文档,貌似SDK内部会管理这些内部资源。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
您在这个PR里提到的问题我此前已经注意到了,暂时没有理解为啥会这样。高通QNN SDK的技术资料比较少,目前只有哪个SDK reference manual.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
个人判断应该是漏了些同步操作,不过确实没啥信息
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
不太清楚,我做过各种实验。没有查到公开的有价值的参考资料。
从已有的公开资料来看,国内目前已经实现了高通NPU加速的公司有几家,其中发布了Open MiniCPM-V的面壁智能是其中一家。如果您是商业公司雇员,可以联系对方。如果是我这样的独立开发者,在没有与QTI签定NDA以及得到高通技术支持的情况下,试图完全做出来,难度可能不小:比如哪些出错信息代码,根本不知道具体是啥意思。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
之前用过Qualcomm的GPU profiler,也是bug一堆,这种问题感觉得等他们自己修了,我们要workaround的话,会花掉很多无谓的时间
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
赞同您的观点。最好得到高通的技术支持。
thanks for your PR. we can discuss this problem in my personal learning&study project:https://github.com/zhouwg/kantv/tree/ggml-qnn-refine/core/ggml/llamacpp/tests/ggml-qnn. 不了解您是啥背景?类似我这样闲着没事干的独立个人开发者?还是AI相关公司雇员?如果您是公司雇员,可以联系面壁智能(他们应该已经做到了高通NPU的加速). 没想到您还对这个问题感兴趣。我现在对上游项目不感兴趣了,如您愿意,可以在我的学习&研究项目里研究ggml-qnn相关问题,可以用中文,这样更方便,最后将研究结果贡献给社区。不用费劲提交给上游了(事实上,自从哪个ggml-rpc.cpp相关的多个pr被合并到master分支后------很失望)。 |
我个人对这个问题还是比较乐观,毕竟rpc那系列pr我也看过,并且日常我也用,和qnn这个,我认为也不冲突。 我做这个还是出于个人爱好,主要可以学习交流新东西,兴趣使然。所以也没打算和厂商联系,基于公开资料做做这样。 |
感谢老哥回复,你用业余时间做到这样,已经不错了,加油! |
4d70039
to
65a14d9
Compare
@zhouwg 老哥,有空麻烦看看PR,如果没啥问题能不能帮忙merge到你的分支去。 |
7a77028
to
dfe159f
Compare
Thank you for the development. Device:Snapdragon8 Gen3 16GB llama-server -m models/Kitsunebi-v1-Gemma2-8k-9B.Q4_K_M.gguf -ngl 40
...
[ggml_qnn_graph, 27]: graph name MUL_MAT_3584x2048
x1x1_3584x2x1x1_2048x2x1x1
[ggml_qnn_graph, 75]: can't create qnn graph handl
e with graph name MUL_MAT_3584x2048x1x1_3584x2x1x1
_2048x2x1x1, error = 6003 diff --git a/ggml/src/ggml-backend.c b/ggml/src/gg
ml-backend.c
index a8eafac4..e2b421e2 100644
--- a/ggml/src/ggml-backend.c
+++ b/ggml/src/ggml-backend.c
@@ -287,6 +287,7 @@ bool ggml_backend_supports_op(
ggml_backend_t backend, const struct ggml_tensor *
}
bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft) {
+ if (NULL == backend->iface.supports_buft) return true;
┆ ┆return backend->iface.supports_buft(backend, buft);
} |
Hi @myan-o , thanks your for the feedback, as i said before, in ggml, the input tensor of matmul operator need to be transposed, to achieve that i've a lot more refactoring work to do, so now the mulmat operator is still under constructiion, for more information, could have a look here: chraac@63dc587 |
Thank you for your answer. So does that mean that matmul operations are not implemented yet? I have also made a pull request for some minor fixes, so please take a look. |
The termux development environment lacks the c++ std library and the build fails. |
As I said in your upstream PR, better to have a function for wrapping
ggml_tensor
intoQnn_Tensor_t
.So here i create a PR for it.
Run test on cpu backend, works well
![5338c775ff17bc845aca02c6380446e](https://private-user-images.githubusercontent.com/1937740/340175572-c3c38fda-964e-4c75-bfb4-e1878cef25e7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNzQ1OTEsIm5iZiI6MTcyNDI3NDI5MSwicGF0aCI6Ii8xOTM3NzQwLzM0MDE3NTU3Mi1jM2MzOGZkYS05NjRlLTRjNzUtYmZiNC1lMTg3OGNlZjI1ZTcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjFUMjEwNDUxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9M2U2MTI1MDJjNWUwNWM3MjJkZGY3MzRkY2U0NDcwNWZiNTMyYTcxOGY5Mzg5Y2NhMjc2ZDdlN2VjMmRhMTBjZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.ne-L0CVDlph9cX-n8RxKWCtKtvCv9MCLaliTUql5skI)
Run on npu backend, also works well:
![1648ce8fa1b30cb385978edd4840dbd](https://private-user-images.githubusercontent.com/1937740/340175832-65363226-953f-4d50-a8a7-b639d33baff7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNzQ1OTEsIm5iZiI6MTcyNDI3NDI5MSwicGF0aCI6Ii8xOTM3NzQwLzM0MDE3NTgzMi02NTM2MzIyNi05NTNmLTRkNTAtYThhNy1iNjM5ZDMzYmFmZjcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjFUMjEwNDUxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OTZhMTc1MWVkYzAwNGRiNTYxYzRhNjBmNjEwZDYxMDM5NzczYmFiZjA1ZThmYmRlMGM1NTliYzA5ZDFjYTFhNSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.jZZWAQLMggrO9mzQ70VfDO-_KIgVYCdrbd7Bfi5-xfo)