support (src_f32,wei_fp16,dst_f32) in ip - #246
Conversation
| CPU_INSTANCE_AVX512(brgemm_inner_product_fwd_t, avx512_core) | ||
| CPU_INSTANCE_AVX2(brgemm_inner_product_fwd_t, avx2) | ||
| nullptr, | ||
| }}, |
There was a problem hiding this comment.
Could you, please, provide a use case for such configuration?
There was a problem hiding this comment.
Hey @dzarukin. This is needed for LLMs with fp16 compressed weights (lower speedup than for u8/u4 compressed weights, but no accuracy loss comparing with fp32).
There was a problem hiding this comment.
@dmitry-gorokhov thanks for explaining. Should oneDNN follow the same path and add such configurations support to the library?
There was a problem hiding this comment.
@dzarukin I would highly recommend to do that. Pretty cheap optimization in terms of implementation effort, but very benefitial for LLMs at the same time. Should be useful for majority of DL frameworks focusing on edge inference, since weights conversion to FP16 doesn't require any fine-tuning/complex compression techniques,
577bbce to
9cc2b8c
Compare
### Details: - *Add support for (src_f32, weight_f16) case in FC* oneDNN-fork PR: openvinotoolkit/oneDNN#246 ### Tickets: - *CVS-133453* - *CVS-139274*
3 participants
Description
Add support for (src_f32, weight_f16, dst_f32) in inner product
Fixes CVS-133453
Checklist
General
make testandmake test_benchdnn_*) pass locally for each commit?Performance improvements
New features
Bug fixes
RFC PR