Add SVE2 optimizations of class SynetQuantizedInnerProductGemmV0 - #860
Merged
Conversation
Port the AVX512BW/NEON GemmV0 quantized inner-product kernel to ARM64 SVE/SVE2 (USDOT, 12x2 micro-kernel, scalable F), including Init dispatch, tests, VS2022 Sve2 project files, and release notes. Co-authored-by: igor.ermolaev <igor.ermolaev@altezza.org>
SVE2 USDOT uses the full int8 range, which can differ from the float32 quantized control by 3 on large GEMMs. Co-authored-by: igor.ermolaev <igor.ermolaev@altezza.org>
ermig1979
marked this pull request as ready for review
August 14, 2026 09:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds ARM/ARM64 SVE/SVE2 SIMD optimizations for
SynetQuantizedInnerProductGemmV0, following the AVX512BW 12-row GemmV0 layout and existing SVE2 quantized GEMM kernels (svusdot/Madd4<false>, scalableF = svcntw()).Changes
Sve2::SynetQuantizedInnerProductGemmV0inSimdSve2SynetQuantizedInnerProductGemmV0.cppSve2::SynetQuantizedInnerProductInitdispatch inSimdSve2SynetQuantizedInnerProduct.cppSIMD_SVE2_FUNCinSimdLib.cppTest::SynetQuantizedInnerProductForwardAutoTestcovers SVE2 (full int8 range / USDOT), extra M/N/K tail shapes, andcontrolDiffMax=3when overflow is disabledSve2.vcxproj/.filtersupdateddocs/2026.html(7.2.165)SVE-BF16 is part of the existing SVE2 CPU feature gate (
HWCAP2_SVEBF16); this kernel is int8×int8 via SVE2 USDOT rather than BF16 MMA.Testing
Cross-compiled for aarch64 (
aarch64-linux-gnu-g++,-DSIMD_SVE2=ON) and ran under qemu-user:qemu-aarch64 -cpu max(SVE VL=512): ALL TESTS ARE FINISHED SUCCESSFULLYqemu-aarch64 -cpu neoverse-n2(SVE VL=128): ALL TESTS ARE FINISHED SUCCESSFULLYBase vs SVE2 is an exact uint8 match with overflow-safe weights. SVE2 vs the float32 quantized control stays within
controlDiffMax(2 with overflow, 3 for full int8 USDOT).Qemu-on-x86 timings are not representative of native ARM. On VL=512 large GEMMs, SVE2 is about 2.3–4.4× vs Base and about 1.1× vs Neon.