Branchless wide-integer comparisons and sign extension

[raimannma]

Makes wide::integer comparisons and sign-extending conversions branchless, so the columnar kernels autovectorize. Combines two changes, as suggested in review (#109475 (review)): the comparison work from this PR and the conversion work merged from #109475.

Comparisons (<, >, <=, >=). wide::integer compares limb-by-limb with an early exit, plus a sign pre-check for signed types. These data-dependent branches make the columnar comparison kernels in FunctionsComparison.h scalar and heavily mispredicting on real data. operator_less and operator_greater now compare the most significant limb as signed for signed types (which subsumes the sign check) and fold in the lower limbs with carry-free boolean logic and no early-out. This autovectorizes for all widths: the x86-64-v3 kernels use AVX2 vpcmpgtq, the x86-64-v4 kernels keep the same AVX-512 vpcmpltuq/vpcmpgtq/vpcmpeqq shape as before, and the 256-bit kernels become branchless too instead of being limited to the 128-bit case.

An earlier revision of this PR used a std::bit_cast to native __int128 for the 128-bit case. That fixed the x86-64-v3 baseline but de-vectorized the x86-64-v4 clones of the kernels (clang cannot re-lift a 128-bit scalar compare into AVX-512 mask compares), which showed up in the CI perf report as i1 < i2 +15% and i1 >= i2 +17% on x86. The generic branchless form replaces it and vectorizes on both v3 and v4.

Columnar less-than kernel (clang 21, x86-64-v3): up to ~7x faster on branch-hostile data (Int128 with random signs 8.0 -> 51.8 GB/s, UInt128 with 50% shared high limbs 8.4 -> 57.0 GB/s), with no regression on any tested shape, including the old code's best case (predictable early exit) and scalar std::sort comparators. Results are bit-for-bit identical to the previous implementation, verified over millions of randomized and boundary pairs for all six comparison operators on all four wide types, plus mixed operand types and constexpr evaluation, on clang 20 and 21.

Conversions to wider signed types. wide_integer_from_builtin and wide_integer_from_wide_integer sign-extend through an if (rhs < 0) branch that selects between two separate fill loops. In columnar conversion loops this data-dependent branch mispredicts on mixed-sign data, and the two-loop shape prevents vectorization even when the branch predicts well. The extension value (0 or -1) is now selected branchlessly and a single fill loop is used.

Kernel benchmark of the columnar conversion loop (clang 21, x86-64-v3, 64k-element blocks): Int64 to Int256 with random signs 10.3 -> 73.9 GB/s (~7x), positive-only 31.3 -> 73.3 GB/s, Int128 to Int256 with random signs 12.2 -> 68.0 GB/s; unsigned conversions unchanged. Results are bit-for-bit identical, verified over tens of millions of checks covering all builtin integral source types with boundary values, 128-to-256-bit widening for all signedness combinations, narrowing, and constexpr evaluation. The CI perf run on #109475 confirmed the effect at the query level: the new wide_integer_conversion test is 2-3x faster on both architectures and decimal_casts #14 (toInt256(Decimal)) improved by 17%.

Performance tests with branch-hostile data shapes are included for both comparisons and conversions.

Related: #109475

Changelog category (leave one):

• Performance Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Improve performance of comparisons (<, >, <=, >=) of wide integer types (Int128, UInt128, Int256, UInt256 and types based on them, such as Decimal128) by up to 7 times, and of converting signed integers to Int256 (up to 7 times on mixed-sign data), by making the comparison and sign-extension code branchless.

[clickhouse-gh]

Workflow [PR], commit [8d36d93]

Summary: ✅

• Performance Comparison: Performance dashboard

---

AI Review

Summary

This PR makes wide::integer comparisons and sign-extending widening conversions branchless, adds focused boundary tests for the new comparison logic, and adds performance cases for the comparison and conversion workloads. After reviewing the current diff, the resolved discussion threads, and the existing surrounding coverage, I did not find an unresolved correctness, compatibility, or evidence issue that still blocks merge.

Final Verdict

Status: ✅ Approve

[raimannma]

@Algunenano I suppose those are flaky tests 04001_join_reorder_through_expression

[Algunenano]

@Algunenano I suppose those are flaky tests 04001_join_reorder_through_expression

Yes, it's being fixed. I'll handle it, don't worry about it

[Algunenano]

The perf tests show regressions on some cases but a big speedup on the sort case (ORDER BY i1 -60%). An odd case is SELECT count() FROM cmp128 WHERE i1 < i2 where the CI sees a regression on x86, and a speed up on ARM.

PS: 04001_join_reorder_through_expression is fixed. Merging master or just pushing a new commit will fix it.

[Algunenano]

The perf tests show regressions on the vector/vector kernels (i1 < i2 +15%, i1 >= i2 +17%) but a big speedup on the sort case (ORDER BY i1 -60%). Looks like might have regressed the v4 code path, which was vectorized before.

Analysis (it's LLM-generated, but I'm guiding the analysis and building locally):

This is the answer, and it's more serious than noise. The vectorVector I first showed was the v3 baseline; the code that actually runs on AVX-512 hardware (this machine, and the CI perf machines) is the _x86_64_v4 clone. Comparing both clones before/after:

┌────────────┬──────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────┐
│            │                   BEFORE (master)                    │                          AFTER (PR)                           │
├────────────┼──────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ v3         │ branchy scalar (js/jne per elem)                     │ branchless scalar (cmp/sbb/setl) ✅                           │
│ baseline   │                                                      │                                                               │
├────────────┼──────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ v4         │ true SIMD: vpermt2q split limbs, vpcmpltuq/vpcmpnltq │ loads zmm then vpextrq/vmovq extracts every limb to GPR and   │
│ (AVX-512)  │  mask compare, 8 elems/iter                          │ does scalar cmp/sbb/setl per element ❌                       │
└────────────┴──────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────┘

Why is it scalar: std::bit_cast<__int128> + < hands clang a 128-bit scalar compare it can't re-lift into cross-lane AVX-512 mask compares (x86 has no 128-bit-wide integer compare instruction). So on the v4 path the PR de-vectorizes the kernel: master compared 8 Int128s per iteration with vpcmpltuq; the PR extracts each limb back to a GPR and compares scalar. The v3 win and the v4 loss are the same if constexpr acting differently at the two targets.

The LLM initially preferred this other approach just for the 128 case:

#ifdef __SIZEOF_INT128__
            if constexpr (Bits == 128 && std::is_same_v<T, integer<Bits, Signed>> && std::endian::native == std::endian::
little)
            {
                using Hi = std::conditional_t<std::is_same_v<Signed, signed>, int64_t, uint64_t>;
                struct Limbs { uint64_t lo; Hi hi; };
                auto la = std::bit_cast<Limbs>(lhs);
                auto lb = std::bit_cast<Limbs>(rhs);
                return (la.hi > lb.hi) | ((la.hi == lb.hi) & (la.lo > lb.lo));
            }
#endif

But I prefer generic code as much as possible, because usually it's also simpler and more optimizable (Sorry for the bad variable names, please change them if you use it):

diff --git a/base/base/wide_integer_impl.h b/base/base/wide_integer_impl.h
index 27297dcfb7f..798e1477ad9 100644
--- a/base/base/wide_integer_impl.h
+++ b/base/base/wide_integer_impl.h
@@ -933,19 +933,27 @@ public:
     {
         if constexpr (should_keep_size<T>())
         {
-            if (std::numeric_limits<T>::is_signed && (is_negative(lhs) != is_negative(rhs)))
-                return is_negative(rhs);
-
             integer<Bits, Signed> t = rhs;
-            for (unsigned i = 0; i < item_count; ++i)
-            {
-                base_type rhs_item = get_item(t, big(i));
 
-                if (lhs.items[big(i)] != rhs_item)
-                    return lhs.items[big(i)] > rhs_item;
+            /// Branchless lexicographic comparison from the most significant limb: no early-out, so the
+            /// columnar comparison kernels stay branchless and vectorize (e.g. to AVX-512 vpcmpq). The top
+            /// limb is compared as signed for signed types, which subsumes the sign check; lower limbs unsigned.
+            base_type l0 = lhs.items[big(0)];
+            base_type r0 = get_item(t, big(0));
+            bool gt;
+            if constexpr (std::is_same_v<Signed, signed>)
+                gt = static_cast<signed_base_type>(l0) > static_cast<signed_base_type>(r0);
+            else
+                gt = l0 > r0;
+            bool eq = l0 == r0;
+            for (unsigned i = 1; i < item_count; ++i)
+            {
+                base_type l = lhs.items[big(i)];
+                base_type r = get_item(t, big(i));
+                gt = gt | (eq & (l > r));
+                eq = eq & (l == r);
             }
-
-            return false;
+            return gt;
         }
         else
         {
@@ -959,19 +967,25 @@ public:
     {
         if constexpr (should_keep_size<T>())
         {
-            if (std::numeric_limits<T>::is_signed && (is_negative(lhs) != is_negative(rhs)))
-                return is_negative(lhs);
-
             integer<Bits, Signed> t = rhs;
-            for (unsigned i = 0; i < item_count; ++i)
-            {
-                base_type rhs_item = get_item(t, big(i));
 
-                if (lhs.items[big(i)] != rhs_item)
-                    return lhs.items[big(i)] < rhs_item;
+            /// See the rationale in operator_greater.
+            base_type l0 = lhs.items[big(0)];
+            base_type r0 = get_item(t, big(0));
+            bool lt;
+            if constexpr (std::is_same_v<Signed, signed>)
+                lt = static_cast<signed_base_type>(l0) < static_cast<signed_base_type>(r0);
+            else
+                lt = l0 < r0;
+            bool eq = l0 == r0;
+            for (unsigned i = 1; i < item_count; ++i)
+            {
+                base_type l = lhs.items[big(i)];
+                base_type r = get_item(t, big(i));
+                lt = lt | (eq & (l < r));
+                eq = eq & (l == r);
             }
-
-            return false;
+            return lt;
         }
         else
         {

Results look good:

Summary

Your approach — delete the #ifdef block, make the generic limb loop branchless — is the right fix. Verified on real object files (same flags, real template instantiations):

┌────────────────────────┬─────────────────────────────────┬────────────────────────────────────────┬──────────────────────────────┐
│                        │           v3 baseline           │              v4 (AVX-512)              │           256-bit            │
├────────────────────────┼─────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────┤
│ master                 │ branchy scalar (js/jne per      │ SIMD vpcmpltuq                         │ SIMD                         │
│                        │ elem)                           │                                        │                              │
├────────────────────────┼─────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────┤
│ PR                     │ branchless scalar ✅            │ scalarized (vpextrq+sbb, ~33% slower)  │ unchanged (ifdef only hit    │
│ (bit_cast<__int128>)   │                                 │ ❌                                     │ 128)                         │
├────────────────────────┼─────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────┤
│ generic branchless     │ AVX2-vectorized (vpcmpgtq ymm)  │ SIMD vpcmpltuq ✅                      │ SIMD + branchless ✅         │
│ loop                   │ ✅                              │                                        │                              │
└────────────────────────┴─────────────────────────────────┴────────────────────────────────────────┴──────────────────────────────┘

PS: 04001_join_reorder_through_expression is fixed. Merging master or just pushing a new commit will fix it.

[alexey-milovidov]

Updating the branch to fix the 04001_join_reorder_through_expression test, which was broken today in master.

[clickhouse-gh]

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	85.60%	85.60%	+0.00%
Functions	92.70%	92.70%	+0.00%
Branches	77.80%	77.80%	+0.00%

Changed lines: Changed C/C++ lines covered: 142/142 (100.00%) · Uncovered code

Full report · Diff report

[Algunenano]

Current perf comparison:

The regression isn't noise, but it's acceptable:

Not noise, and not topK — it's a real, reproducible regression with a clean microarchitectural cause. perf stat nails it:

┌─────────────────┬────────┬────────┬────────────────────────────────┐
│     metric      │ master │   pr   │               Δ                │
├─────────────────┼────────┼────────┼────────────────────────────────┤
│ instructions    │ 42.98B │ 42.99B │ ~0                             │
├─────────────────┼────────┼────────┼────────────────────────────────┤
│ branches        │ 7.46B  │ 4.34B  │ −42% (branchless, as designed) │
├─────────────────┼────────┼────────┼────────────────────────────────┤
│ branch-misses   │ 86.1M  │ 86.9M  │ ~0 (identical)                 │
├─────────────────┼────────┼────────┼────────────────────────────────┤
│ L1-dcache-loads │ 10.1B  │ 13.3B  │ +32%                           │
└─────────────────┴────────┴────────┴────────────────────────────────┘

The +32% more loads matches the +33% runtime almost exactly. Here's the mechanism:

- The hotspot is miniselect::floyd_rivest_select_loop<…, ColumnVector<UInt128>::greater&> (66–68% self-time) — the ORDER BY … LIMIT 10 selection, and it's comparison-bound in-cache.
- u1's high 64 bits are cityHash64(n,1) → essentially always distinct between rows. So the old early-exit comparator (cmp hi; jne; …) decides on the high limb alone ~100% of the time, never reads the low limb, and — critically — that branch is perfectly predicted (branch-misses are identical between the two builds). It gets a free skip of half the work.
- The branchless combine (seta hi; seta lo; and; or) has no branch to mispredict, but it always reads and compares both limbs — hence +32% L1 loads for zero benefit on this data.

So this is the exact flip side of the signed sort (#7, −55%): there the removed branch was the sign check on random-sign data that mispredicted ~50%, so branchless won big. Here the removed branch was the high-limb-inequality check that predicts ~100% and let you skip the low-limb load, so branchless is pure overhead.

It's the inherent branchless tradeoff, not a bug: you win when the branch is unpredictable (signed sort, columnar SIMD kernels — the PR's main goal) and lose a little when it's perfectly predictable and work-saving (high-cardinality unsigned top-N). Absolute cost is small (~3ms on a 9ms query), and it only hits the scalar comparator path — the columnar WHERE kernels still vectorize. Worth a line in the PR, but the net across the changed paths is still clearly positive.

We can accept a slightly worse single case (when the branch was always predicted correctly and it was a shortcut) but win in all the other ones (unpredictable branches). Also, it only regresses that bit on x86, and ARM sees benefits all around (I'm guessing it's previous code was less simd friendly)

Good job, thanks!

[PedroTadim]

@raimannma should performance tests for the <= and >= operators be added?

[raimannma]

#110369