[X86] Poor code generation for boolean vectors on AVX2

[jhuber6]

Boolean vectors are often lowered to integral operations. This code generates the expected bitwise operations only when avx512f is available even though it does not require use of mask registers https://clang.godbolt.org/z/vrdbo9YKa.

#include <cstdint>

typedef __attribute__((ext_vector_type(16))) int16_t i16x16;
typedef __attribute__((ext_vector_type(16))) bool i1x16;

int A(i16x16);

void test(i16x16 v, i1x16 a, i1x16 b) {
    i1x16 m = (a & (~b)) != 0;
    if (__builtin_reduce_or(m))
        A(v);
}

This is introduced at the first instruction selection phase, so I am guessing that lowering these is trying to use the mask registers which then get optimized away.

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: Joseph Huber (jhuber6)

Boolean vectors are often lowered to integral operations. This code generates the expected bitwise operations only when `avx512f` is available even though it does not require use of mask registers https://clang.godbolt.org/z/vrdbo9YKa.

#include <cstdint>

typedef __attribute__((ext_vector_type(16))) int16_t i16x16;
typedef __attribute__((ext_vector_type(16))) bool i1x16;

int A(i16x16);

void test(i16x16 v, i1x16 a, i1x16 b) {
    i1x16 m = (a & (~b)) != 0;
    if (__builtin_reduce_or(m))
        A(v);
}

This is introduced at the first instruction selection phase, so I am guessing that lowering these is trying to use the mask registers which then get optimized away.

[RKSimon]

We might be able to get vectorcombine to handle this - its what VectorCombine::foldBitOpOfCastops is there to do

[RKSimon]

CC @XChy who worked on foldBitOpOfCastops/foldBitOpOfCastConstant recently

[XChy]

Looks like the constraint on destination type can be relaxed to both vector and scalar.