Implement vectorized min_ / max_element for ints #2447

AlexGuteniev · 2021-12-26T15:49:21Z

📝 Summary

SSE4.1 implementation of min_element / max_element / minmax_element for signed and unsigned integers of sizes 1,2,4,8.

Resolves #2438

The algorithm is more complex than existing vector algorithms, not sure if this level of complexity is fine.

🧭Further directions (suggested next PRs)

Add AVX2. Twice faster, twice bloater
ranges::min, ranges::max, ranges::minmax -- as don't need iterators, this will be simpler faster algorithm, will consist only of vertical max and one reduction.
Add the same for floating point types

🏁 Perf benchmark

Benchmark

#include <algorithm>
#include <cstdint>
#include <chrono>
#include <iostream>
#include <ranges>
#include <random>
#include <intrin.h>

enum class Kind {
    Min,
    Max,
    Minmax,
};

void* volatile discard1;
void* volatile discard2;

template<typename T>
void benchmark_find(T* a, std::size_t max, Kind kind, size_t rep) {
    auto t1 = std::chrono::steady_clock::now();

    switch (kind)
    {
    case Kind::Min:
        for (std::size_t s = 0; s < rep; s++) {
            discard1 = std::min_element(a, a + max);
        }
        break;
    case Kind::Max:
        for (std::size_t s = 0; s < rep; s++) {
            discard2 = std::max_element(a, a + max);
        }
        break;
    case Kind::Minmax:
        for (std::size_t s = 0; s < rep; s++) {
            std::tie(discard1, discard2) = std::minmax_element(a, a + max);
        }
        break;
    }

    auto t2 = std::chrono::steady_clock::now();

    const char* op_str = nullptr;
    switch (kind)
    {
    case Kind::Min:
        op_str = "min";
        break;
    case Kind::Max:
        op_str = "max";
        break;
    case Kind::Minmax:
        op_str = "minmax";
        break;
    }
    std::cout << std::setw(10) << std::chrono::duration_cast<std::chrono::duration<double>>(t2 - t1).count() << "s  -- "
        << "Op " << op_str << " Size " << sizeof(T) << " byte elements, array size " << max << "; " << rep << " repetitions \n";
}


constexpr std::size_t Nmax = 8192 - 1;

alignas(64) std::uint8_t    a8[Nmax];
alignas(64) std::uint16_t   a16[Nmax];
alignas(64) std::uint32_t   a32[Nmax];
alignas(64) std::uint64_t   a64[Nmax];


extern "C" long __isa_enabled;

int main()
{
    std::mt19937 gen(84710);
    std::uniform_int_distribution<int> dis(1, 20);
    std::generate(std::begin(a8), std::begin(a8), [&] { return dis(gen); });
    std::generate(std::begin(a16), std::begin(a16), [&] { return dis(gen); });
    std::generate(std::begin(a32), std::begin(a32), [&] { return dis(gen); });
    std::generate(std::begin(a64), std::begin(a64), [&] { return dis(gen); });

    std::cout << "Vector alg used: " << _USE_STD_VECTOR_ALGORITHMS << "\n";

    benchmark_find(a8, Nmax, Kind::Minmax, 100000);
    benchmark_find(a16, Nmax, Kind::Minmax, 100000);
    benchmark_find(a32, Nmax, Kind::Minmax, 100000);
    benchmark_find(a64, Nmax, Kind::Minmax, 100000);

    std::cout << '\n';

    benchmark_find(a8, Nmax, Kind::Min, 100000);
    benchmark_find(a16, Nmax, Kind::Min, 100000);
    benchmark_find(a32, Nmax, Kind::Min, 100000);
    benchmark_find(a64, Nmax, Kind::Min, 100000);

    std::cout << '\n';

    benchmark_find(a8, Nmax, Kind::Max, 100000);
    benchmark_find(a16, Nmax, Kind::Max, 100000);
    benchmark_find(a32, Nmax, Kind::Max, 100000);
    benchmark_find(a64, Nmax, Kind::Max, 100000);

    std::cout << "Done\n";

    return 0;
}

Current benchmark results

**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.1.0-pre.1.1
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

C:\Program Files\Microsoft Visual Studio\2022\Preview>cd/d C:\Project\vector_find_benchmark

C:\Project\vector_find_benchmark>set INCLUDE=C:\Project\STL\out\build\x64\out\inc;%INCLUDE%

C:\Project\vector_find_benchmark>set LIB=C:\Project\STL\out\build\x64\out\lib\amd64;%LIB%

C:\Project\vector_find_benchmark>set PATH=C:\Project\STL\out\build\x64\out\bin\amd64;%PATH%

C:\Project\vector_find_benchmark>cl /O2 /std:c++latest /EHsc /D_USE_STD_VECTOR_ALGORITHMS=0 /nologo vector_find_benchmark.cpp
vector_find_benchmark.cpp

C:\Project\vector_find_benchmark>vector_find_benchmark.exe
Vector alg used: 0
  0.740433s  -- Op minmax Size 1 byte elements, array size 8191; 100000 repetitions
    0.7414s  -- Op minmax Size 2 byte elements, array size 8191; 100000 repetitions
  0.739575s  -- Op minmax Size 4 byte elements, array size 8191; 100000 repetitions
   0.74237s  -- Op minmax Size 8 byte elements, array size 8191; 100000 repetitions

   1.47435s  -- Op min Size 1 byte elements, array size 8191; 100000 repetitions
   1.47671s  -- Op min Size 2 byte elements, array size 8191; 100000 repetitions
   1.47945s  -- Op min Size 4 byte elements, array size 8191; 100000 repetitions
   1.47856s  -- Op min Size 8 byte elements, array size 8191; 100000 repetitions

    1.4785s  -- Op max Size 1 byte elements, array size 8191; 100000 repetitions
   1.47736s  -- Op max Size 2 byte elements, array size 8191; 100000 repetitions
    1.4765s  -- Op max Size 4 byte elements, array size 8191; 100000 repetitions
   1.47861s  -- Op max Size 8 byte elements, array size 8191; 100000 repetitions
Done

C:\Project\vector_find_benchmark>cl /O2 /std:c++latest /EHsc /D_USE_STD_VECTOR_ALGORITHMS=1 /nologo vector_find_benchmark.cpp
vector_find_benchmark.cpp

C:\Project\vector_find_benchmark>vector_find_benchmark.exe
Vector alg used: 1
  0.071214s  -- Op minmax Size 1 byte elements, array size 8191; 100000 repetitions
  0.134923s  -- Op minmax Size 2 byte elements, array size 8191; 100000 repetitions
  0.217484s  -- Op minmax Size 4 byte elements, array size 8191; 100000 repetitions
  0.513987s  -- Op minmax Size 8 byte elements, array size 8191; 100000 repetitions

 0.0567489s  -- Op min Size 1 byte elements, array size 8191; 100000 repetitions
   0.13397s  -- Op min Size 2 byte elements, array size 8191; 100000 repetitions
  0.160763s  -- Op min Size 4 byte elements, array size 8191; 100000 repetitions
  0.480221s  -- Op min Size 8 byte elements, array size 8191; 100000 repetitions

 0.0559508s  -- Op max Size 1 byte elements, array size 8191; 100000 repetitions
  0.134017s  -- Op max Size 2 byte elements, array size 8191; 100000 repetitions
  0.161021s  -- Op max Size 4 byte elements, array size 8191; 100000 repetitions
  0.491206s  -- Op max Size 8 byte elements, array size 8191; 100000 repetitions
Done

Results table

size	before	after
minmax 1 byte	0.74361s	0.071214s
minmax 2 bytes	0.746617s	0.134923s
minmax 4 bytes	0.742254s	0.217484s
minmax 8 bytes	0.750876s	0.513987s
min 1 byte	1.47848s	0.0567489s
min 2 bytes	1.4882s	0.13397s
min 4 bytes	1.47821s	0.160763s
min 8 bytes	1.47858s	0.480221s
max 1 byte	1.48783s	0.0559508s
max 2 bytes	1.48865s	0.134017s
max 4 bytes	1.48718s	0.161021s
max 8 bytes	1.48245s	0.491206s

⚖️ Size impact

The change adds more code.
DLLs and PDBs for them are not affected. Static libraries are affected.
The impact is negligible for static libs, but noticeable for import libs.

Table

File name	Size before	Size after
libcpmt.lib	35,023,488	35,271,438
libcpmt1.lib	35,802,670	36,050,304
libcpmtd.lib	36,120,578	36,357,874
libcpmtd0.lib	34,999,810	35,236,962
libcpmtd1.lib	35,874,386	36,111,490
msvcprt.lib	1,141,058	1,388,864
msvcprtd.lib	1,138,636	1,375,852
stl_asan.lib	3,014	3,014

✔️ Test coverage

Expand VSO_0000000_vector_algorithms to test newly vectorized cases with various sizes

Resolves microsoft#2438 TODO: * Test coverage * Attach minmax_element * Add AVX2 version of the same ---- <detail> <summary><b>Benchmark</b></summary> ```C++ #include <algorithm> #include <cstdint> #include <chrono> #include <iostream> #include <ranges> #include <intrin.h> enum class Kind { Min, Max, }; template<typename T> void benchmark_find(T* a, std::size_t max, size_t start, size_t pos, Kind kind, size_t rep) { std::fill_n(a, max, '0'); if (pos < max && pos >= start) { if (kind == Kind::Min) { a[pos] = '*'; } else { a[pos] = '1'; } } auto t1 = std::chrono::steady_clock::now(); switch (kind) { case Kind::Min: for (std::size_t s = 0; s < rep; s++) { if (std::min_element(a + start, a + max) != a + pos) { abort(); } } break; case Kind::Max: for (std::size_t s = 0; s < rep; s++) { if (std::min_element(a + start, a + max) != a + pos) { abort(); } } break; } auto t2 = std::chrono::steady_clock::now(); const char* op_str = nullptr; switch (kind) { case Kind::Min: op_str = "min"; break; case Kind::Max: op_str = "max"; break; } std::cout << std::setw(10) << std::chrono::duration_cast<std::chrono::duration<double>>(t2 - t1).count() << "s -- " << "Op " << op_str << " Size " << sizeof(T) << " byte elements, array size " << max << " starting at " << start << " found at " << pos << "; " << rep << " repetitions \n"; } constexpr std::size_t Nmax = 8192; alignas(64) std::uint8_t a8[Nmax]; alignas(64) std::uint16_t a16[Nmax]; alignas(64) std::uint32_t a32[Nmax]; alignas(64) std::uint64_t a64[Nmax]; extern "C" long __isa_enabled; int main() { std::cout << "Vector alg used: " << _USE_STD_VECTOR_ALGORITHMS << "\n"; benchmark_find(a8, Nmax, 0, 3459, Kind::Min, 100000); benchmark_find(a16, Nmax, 0, 3459, Kind::Min, 100000); benchmark_find(a32, Nmax, 0, 3459, Kind::Min, 100000); benchmark_find(a64, Nmax, 0, 3459, Kind::Min, 100000); benchmark_find(a8, Nmax, 0, 3459, Kind::Max, 100000); benchmark_find(a16, Nmax, 0, 3459, Kind::Max, 100000); benchmark_find(a32, Nmax, 0, 3459, Kind::Max, 100000); benchmark_find(a64, Nmax, 0, 3459, Kind::Max, 100000); std::cout << "Done\n"; return 0; } ``` <detail> <summary><b>Current benchmark results</b></summary> ``` ********************************************************************** ** Visual Studio 2022 Developer Command Prompt v17.1.0-pre.1.1 ** Copyright (c) 2021 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x64' C:\Program Files\Microsoft Visual Studio\2022\Preview>cd/d C:\Project\vector_find_benchmark C:\Project\vector_find_benchmark>set INCLUDE=C:\Project\STL\out\build\x64\out\inc;%INCLUDE% C:\Project\vector_find_benchmark>set LIB=C:\Project\STL\out\build\x64\out\lib\amd64;%LIB% C:\Project\vector_find_benchmark>set PATH=C:\Project\STL\out\build\x64\out\bin\amd64;%PATH% C:\Project\vector_find_benchmark>cl /O2 /std:c++latest /EHsc /D_USE_STD_VECTOR_ALGORITHMS=0 /nologo vector_find_benchmark.cpp vector_find_benchmark.cpp vector_find_benchmark.cpp(1): warning C4005: '_USE_STD_VECTOR_ALGORITHMS': macro redefinition vector_find_benchmark.cpp: note: see previous definition of '_USE_STD_VECTOR_ALGORITHMS' C:\Project\vector_find_benchmark>cl /O2 /std:c++latest /EHsc /D_USE_STD_VECTOR_ALGORITHMS=0 /nologo vector_find_benchmark.cpp vector_find_benchmark.cpp C:\Project\vector_find_benchmark>vector_find_benchmark.exe Vector alg used: 0 1.48497s -- Op min Size 1 byte elements, array size 8192 starting at 0 found at 3459; 100000 repetitions 1.48125s -- Op min Size 2 byte elements, array size 8192 starting at 0 found at 3459; 100000 repetitions 1.47988s -- Op min Size 4 byte elements, array size 8192 starting at 0 found at 3459; 100000 repetitions 1.48431s -- Op min Size 8 byte elements, array size 8192 starting at 0 found at 3459; 100000 repetitions C:\Project\vector_find_benchmark>cl /O2 /std:c++latest /EHsc /D_USE_STD_VECTOR_ALGORITHMS=1 /nologo vector_find_benchmark.cpp vector_find_benchmark.cpp C:\Project\vector_find_benchmark>vector_find_benchmark.exe Vector alg used: 1 0.0559598s -- Op min Size 1 byte elements, array size 8192 starting at 0 found at 3459; 100000 repetitions 0.0681002s -- Op min Size 2 byte elements, array size 8192 starting at 0 found at 3459; 100000 repetitions 0.159074s -- Op min Size 4 byte elements, array size 8192 starting at 0 found at 3459; 100000 repetitions 0.597614s -- Op min Size 8 byte elements, array size 8192 starting at 0 found at 3459; 100000 repetitions ``` </detail>

stl/inc/algorithm

stl/src/vector_algorithms.cpp

Need to guard in C++20 to call _STD is_constant_evaluated()

stl/src/vector_algorithms.cpp

tests/std/tests/VSO_0000000_vector_algorithms/test.cpp

stl/src/vector_algorithms.cpp

StephanTLavavej · 2022-06-18T09:00:42Z

@AlexGuteniev Thanks, this looks good - another amazing speedup! I pushed changes for the issues I noticed (FYI @barcharcraz in case you want to double-check). Edit: Also renamed "cor" to "correction".

~~I would like to know what the "TRANSITION, 17.3 Preview 2" comments are referring to, but that doesn't have to block this PR from merging.~~ Edit: Found it, pushing changes.

~~I am also somewhat curious whether CUDA 11.6 has opened up new possibilities, but the PR should be fine as-is.~~ Edit: Removed CUDA workarounds.

We believe that CUDA 11.6 supports `__builtin_is_constant_evaluated`.

StephanTLavavej · 2022-06-19T02:21:35Z

I'm mirroring this to the MSVC-internal repo - please notify me if any further changes are pushed.

StephanTLavavej · 2022-06-19T02:59:58Z

We need to restore the warning workarounds because the internal build is still using a slightly older compiler without the fix. I've opted to make them uncommented perma-workarounds.

StephanTLavavej · 2022-06-19T23:38:26Z

Thanks for minimizing the time and maximizing the speed of these algorithms! 🚀 🎉 😻

Co-authored-by: Stephan T. Lavavej <stl@nuwen.net>

AlexGuteniev added 2 commits December 26, 2021 17:47

fix copypasta

19ba1d0

miscco reviewed Dec 26, 2021

View reviewed changes

AlexGuteniev added 22 commits December 26, 2021 18:24

Guard C++ features

44b19c9

typo

7c233f4

@miscco review

e245018

contexpr in C++14 / C++17 :-(

3bf81a7

Need to guard in C++20 to call _STD is_constant_evaluated()

Fix algorithm

f781a70

x86 build

34b935d

minmax_element

de0397a

fix corner case; +comments

f118d4c

misleading comments

bc6a055

enable x86 function fusion optimziation

6a7708e

Comment on __stdcall

64b9e80

-stray def param

8e2bf94

literate

300b3b4

Simplify _Base

7597d6f

strange, tests locally pass for me

0519a3e

try to undo __std_minmax_element

4246874

ptrptr

121ea2a

minmax element wrapper

2a7b50b

minmax

6664519

fix minmax_element for larger value type

d0ccf20

some test coverage

24dbd57

Fix x86 64-bit handling

3ed6d99

AlexGuteniev marked this pull request as ready for review December 27, 2021 12:03

AlexGuteniev requested a review from a team as a code owner December 27, 2021 12:03

AlexGuteniev added 3 commits December 27, 2021 14:44

minor cleanup

2b43c89

more text cleanup

137f413

-stdlib dependency

f01c3c6

barcharcraz added this to @barcharcraz in Maintainer Priorities Jun 14, 2022

barcharcraz approved these changes Jun 15, 2022

View reviewed changes

barcharcraz moved this from Initial Review to Final Review in Code Reviews Jun 15, 2022

Merge branch 'main' into vector_max

5c42c76

StephanTLavavej reviewed Jun 18, 2022

View reviewed changes

Code review feedback.

56799e8

StephanTLavavej mentioned this pull request Jun 18, 2022

More minmax vectorization #2803

Closed

3 tasks

StephanTLavavej approved these changes Jun 18, 2022

View reviewed changes

Remove workarounds now that we have 17.3 Preview 2.

f538bbc

StephanTLavavej approved these changes Jun 18, 2022

View reviewed changes

StephanTLavavej removed their assignment Jun 18, 2022

StephanTLavavej moved this from Final Review to Ready To Merge in Code Reviews Jun 18, 2022

StephanTLavavej added 2 commits June 18, 2022 16:03

Rename "cor" to "correction".

1172483

Remove CUDA guards.

f566426

We believe that CUDA 11.6 supports `__builtin_is_constant_evaluated`.

StephanTLavavej approved these changes Jun 18, 2022

View reviewed changes

StephanTLavavej self-assigned this Jun 19, 2022

Restore perma-workarounds.

d178303

StephanTLavavej approved these changes Jun 19, 2022

View reviewed changes

StephanTLavavej merged commit 78cd436 into microsoft:main Jun 19, 2022

Code Reviews automation moved this from Ready To Merge to Done Jun 19, 2022

Maintainer Priorities automation moved this from @barcharcraz to Done Jun 19, 2022

AlexGuteniev deleted the vector_max branch June 20, 2022 05:34

This was referenced Jun 23, 2022

iterator_traits<vehicle_part_iterator<T>> is missing reference CleverRaven/Cataclysm-DDA#58643

Closed

Fix vectorized min/max/minmax_element for 64-bit types on x86 #2821

Merged

fsb4000 pushed a commit to fsb4000/STL that referenced this pull request Aug 13, 2022

Implement vectorized min_ / max_element for ints (microsoft#2447)

0986811

Co-authored-by: Stephan T. Lavavej <stl@nuwen.net>

StephanTLavavej mentioned this pull request Apr 3, 2023

<algorithm>: Silent bad codegen for vectorized meow_element() above 4 GB #3617

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement vectorized min_ / max_element for ints #2447

Implement vectorized min_ / max_element for ints #2447

AlexGuteniev commented Dec 26, 2021 •

edited

Loading

StephanTLavavej commented Jun 18, 2022 •

edited

Loading

StephanTLavavej commented Jun 19, 2022

StephanTLavavej commented Jun 19, 2022

StephanTLavavej commented Jun 19, 2022

Implement vectorized min_ / max_element for ints #2447

Implement vectorized min_ / max_element for ints #2447

Conversation

AlexGuteniev commented Dec 26, 2021 • edited Loading

📝 Summary

🧭Further directions (suggested next PRs)

🏁 Perf benchmark

⚖️ Size impact

✔️ Test coverage

StephanTLavavej commented Jun 18, 2022 • edited Loading

StephanTLavavej commented Jun 19, 2022

StephanTLavavej commented Jun 19, 2022

StephanTLavavej commented Jun 19, 2022

AlexGuteniev commented Dec 26, 2021 •

edited

Loading

StephanTLavavej commented Jun 18, 2022 •

edited

Loading