std::arch::x86_64::_mm_mulhi_epu16(vector, _mm_set1_epi16(scalar)) generates slow emulated path instead of pmulhuw (regressed in rustc 1.75)

[gergo-salyi]

Yes, this apparently regressed more than 2 years ago. The following example on godbolt: https://godbolt.org/z/4bY5eM9az

Code

use std::arch::x86_64::*;

#[no_mangle]
#[target_feature(enable = "sse2")]
pub unsafe fn f(buf: &mut [__m128i], factor: i16) {
    let factor = _mm_set1_epi16(factor);
    for i in 0..buf.len() {
        buf[i] = _mm_mulhi_epu16(buf[i], factor);
    }
}

Version it worked on: rustc 1.74

Broadcasts factor in xmm0 before the loop, then uses the desired pmulhuw for mulhi

f:
        test    rsi, rsi
        je      .LBB0_6
        movd    xmm0, edx
        pshuflw xmm0, xmm0, 0
        pshufd  xmm0, xmm0, 0
        mov     eax, esi
        and     eax, 3
        cmp     rsi, 4
        jae     .LBB0_7
        xor     ecx, ecx
        jmp     .LBB0_3
.LBB0_7:
        and     rsi, -4
        lea     rdx, [rdi + 48]
        xor     ecx, ecx
.LBB0_8:
        movdqa  xmm1, xmmword ptr [rdx - 48]
        pmulhuw xmm1, xmm0
        movdqa  xmmword ptr [rdx - 48], xmm1
        movdqa  xmm1, xmmword ptr [rdx - 32]
        pmulhuw xmm1, xmm0
        movdqa  xmmword ptr [rdx - 32], xmm1
        movdqa  xmm1, xmmword ptr [rdx - 16]
        pmulhuw xmm1, xmm0
        movdqa  xmmword ptr [rdx - 16], xmm1
        add     rcx, 4
        movdqa  xmm1, xmmword ptr [rdx]
        pmulhuw xmm1, xmm0
        movdqa  xmmword ptr [rdx], xmm1
        add     rdx, 64
        cmp     rsi, rcx
        jne     .LBB0_8
.LBB0_3:
        test    rax, rax
        je      .LBB0_6
        shl     rcx, 4
        add     rdi, rcx
        shl     rax, 4
        xor     ecx, ecx
.LBB0_5:
        movdqa  xmm1, xmmword ptr [rdi + rcx]
        pmulhuw xmm1, xmm0
        movdqa  xmmword ptr [rdi + rcx], xmm1
        add     rcx, 16
        cmp     rax, rcx
        jne     .LBB0_5
.LBB0_6:
        ret

Version with regression: rustc 1.75

Slow code that emulates pmulhuw through the definition: ((vector_u16 as u32) * (factor_u16 as u32)) >> 16

.LCPI0_0:
        .short  65535
        .short  0
        .short  65535
        .short  0
        .short  65535
        .short  0
        .short  65535
        .short  0
f:
        test    rsi, rsi
        je      .LBB0_3
        movd    xmm0, edx
        pshuflw xmm0, xmm0, 0
        pshufd  xmm1, xmm0, 0
        pxor    xmm2, xmm2
        punpcklwd       xmm0, xmm2
        pand    xmm1, xmmword ptr [rip + .LCPI0_0]
        pshufd  xmm3, xmm1, 245
        pshufd  xmm4, xmm0, 245
.LBB0_2:
        movdqa  xmm5, xmmword ptr [rdi]
        movdqa  xmm6, xmm5
        punpckhwd       xmm6, xmm2
        punpcklwd       xmm5, xmm2
        pshufd  xmm7, xmm5, 245
        pmuludq xmm5, xmm1
        pshufd  xmm5, xmm5, 232
        pmuludq xmm7, xmm3
        pshufd  xmm7, xmm7, 232
        punpckldq       xmm5, xmm7
        pshufd  xmm7, xmm6, 245
        pmuludq xmm6, xmm0
        pshufd  xmm6, xmm6, 232
        pmuludq xmm7, xmm4
        pshufd  xmm7, xmm7, 232
        punpckldq       xmm6, xmm7
        psrad   xmm6, 16
        psrad   xmm5, 16
        packssdw        xmm5, xmm6
        movdqa  xmmword ptr [rdi], xmm5
        add     rdi, 16
        dec     rsi
        jne     .LBB0_2
.LBB0_3:
        ret

Related issues

#124216
#130782
#138725

All of these seem to have been concluded with a resolved LLVM issue already. In the same order:

llvm/llvm-project#132166
llvm/llvm-project#109790
llvm/llvm-project#132166 (duplicate)

Causes

I did not bisect.

One note: stdarch commit 7abc64d5 rust-lang/stdarch@7abc64d changed the implementation of _mm_mulhi_epu16 from #[link_name = "llvm.x86.sse2.pmulhu.w"] to using Rust portable SIMD.

Manually reverting that by using a _mm_mulhi_epu16 backed by #[link_name = "llvm.x86.sse2.pmulhu.w"] fixes this issue at least on current nightly. Godbolt: https://godbolt.org/z/zqc1fMvjM

Moreover using a runtime variable scalar factor for one of the operand of mulhi_epu16 seems to be necessary to trigger this bad codegen for mulhi_epu16. Doing mulhi_epu16 in a loop on 2 &[__m128i] slices works fine.

Workaround

Use inline assembly for the problematic instructions...

Use case

Fixed point image signal processing which generated optimal code up to rustc 1.72, see godbolt: https://godbolt.org/z/bh6zTqWa3 (This example is further complicated by _mm_unpack*_epi8 breaking in rustc 1.73 due to a different LLVM regression which was fixed first in the LLVM of rustc 1.95)

@rustbot modify labels: +regression-from-stable-to-stable -regression-untriaged

EDIT: fix a copy-paste mistake

[rustbot]

Assigning P-medium (discussion on Zulip).