Propose optional CPUARCH target AlderLakeAVX512, as aliase of SapphireRapids

[FCLC]

Context

It's been reasonably well documented that on alder lake, if, and only if, the GraceMount Ecore's are disabled, it becomes possible to enable all of the available AVX512 instruction available on the Golden Cove Pcore's the same core's used in SapphireRapids.

For relevant workloads, many of which OpenBLAS has AVX512 accelerated code paths, this can lead to a significant performance uplift.

As of release 0.3.19, even if ecores are disabled and AVX512 is available, the build system will not make use of them automatically.

Current workaround

The user has the option of either passing:

CFLAGS='-O3 -march=sapphirerapids -mno-amx-tile -mno-amx-int8 -mno-amx-bf16'
CXXFLAGS='-O3 -march=sapphirerapids -mno-amx-tile -mno-amx-int8 -mno-amx-bf16'
FFLAGS='-O3 -march=sapphirerapids -mno-amx-tile -mno-amx-int8 -mno-amx-bf16'

or alternatively passing

CFLAGS='-O3 -march=native -mavx512f -mavx512dq -mavx512ifma -mavx512cd -mavx512bw -mavx512vl -mavx512bf16 -mavx512vbmi -mavx512vbmi2 -mavx512vnni -mavx512bitalg -mavx512vpopcntdq -mavx512vp2intersect'
CXXFLAGS='-O3 -march=native -mavx512f -mavx512dq -mavx512ifma -mavx512cd -mavx512bw -mavx512vl -mavx512bf16 -mavx512vbmi -mavx512vbmi2 -mavx512vnni -mavx512bitalg -mavx512vpopcntdq -mavx512vp2intersect'
FFLAGS='-O3 -march=native -mavx512f -mavx512dq -mavx512ifma -mavx512cd -mavx512bw -mavx512vl -mavx512bf16 -mavx512vbmi -mavx512vbmi2 -mavx512vnni -mavx512bitalg -mavx512vpopcntdq -mavx512vp2intersect'

NOTE: It is preferred to use the -march=sapphirerapids option as GCC, Clang, LLVM based ICX and ICC will then preserve the relevant AVX512 cost functions for auto-vectorization vs other instruction pathways, instead of generic costs meant as a catch all across all supported architectures.
NOTE: We must disable the AMX tile and AMX instructions as the additional hardware was not built into Alder Lake.

Proposed solution/Request

I'd like to propose that, if the user supplies an architecture flag AlderLakeAVX512, it be aliased to be enable all features of Sapphire Rapids not explicitly requiring the AMX Tile.

Sources on AVX512 support on alder lake:

Anandtech article

Phoronix Article

OpenBenchmarking Results of disabling 8 ecores to enable AVX512 on 8 pcores

Testing of AVX512 per instruction cost and pipeline on Alderlake

discussion of AVX512 performance support and performance

[martin-frbg]

You should be able to achieve just that by building with TARGET=SAPPHIRERAPIDS already. The autodetection code will default to Haswell as it does not "know" if the Ecores are disabled (and going to stay that way)

[FCLC]

TARGET=SAPPHIRERAPIDS

Unless I'm mistaken, this would mean that AMX instructions would also be compiled into source, which are not supported on alder lake and would cause a crash if/when any routines call on them.

Specifically any calls that can be accelerated by int8 or bf-16 are expected to be offloaded to the AMX tile instead of being done in AVX 512 because of the much higher throughput of the AMX matrix instructions vs the AVX vector instructions.

see this slide from intel for example https://pbs.twimg.com/media/E9LHMgwWQAUlrdD?format=jpg&name=large

or alternatively this preproduction intel example video (length of video ~1:05 minutes total) https://www.youtube.com/watch?v=sFzg4zmIlc8

[martin-frbg]

Ah sorry, I replied to fast - should have written "TARGET=SKYLAKEX" (though an OpenBLAS built for SapphireRapids would probably work just as well, provided one does not actually throw any bfloat16 data at it)
EDITED to add: ... but I do not have access to Alder Lake hardware so no idea how much performance would suffer from having the compiler use cost functions for the older SkylakeX architecture

[FCLC]

"TARGET=SKYLAKEX"

Yeah, Id thought of doing this as a work around, but this would also produce a sub-optimal build unfortunately for 2 reasons:

1. it would not take advantage of the additional AVX512 instructiotions added over Skylake
2. It would use the wrong cost functions (saw your edit, including this for the sake of documenting for others in this thread, but there's a little bit more to it, see below)

2a. This is also the reason we don't want to use march=native or march=alderlake +mavx[list of instructions to include], because the compiler will then select cost functions to balance between both the Golden Cove Pcores && the gracemount ecores, leaving performance on the table due to things like different ALU's, cache size's, having SMT etc.
2b. We'd also end up not using all available AVX instructions, since SKYLAKEX supports significantly less.

SkylakeX:

Intel Skylake Server CPU with 64-bit extensions, MOVBE, MMX,
SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, PKU,
AVX, AVX2, AES, PCLMUL, FSGSBASE, RDRND, FMA, BMI,
BMI2, F16C, RDSEED, ADCX, PREFETCHW, CLFLUSHOPT,
XSAVEC, XSAVES, AVX512F, CLWB, AVX512VL, AVX512BW,
AVX512DQ and AVX512CD instruction set support.

SapphireRapids:

Intel sapphirerapids CPU with 64-bit extensions, MOVBE, MMX,
SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, PKU,
AVX, AVX2, AES, PCLMUL, FSGSBASE, RDRND, FMA, BMI,
BMI2, F16C, RDSEED, ADCX, PREFETCHW, CLFLUSHOPT,
XSAVEC, XSAVES, AVX512F, CLWB, AVX512VL, AVX512BW,
AVX512DQ, AVX512CD, AVX512VNNI, AVX512BF16,
MOVDIRI, MOVDIR64B, AVX512VP2INTERSECT, EN-
QCMD, CLDEMOTE, PTWRITE, WAITPKG, SERIALIZE,
TSXLDTRK, UINTR, AMX-BF16, AMX-TILE, AMX-INT8 and
AVX-VNNI instruction set support.

of which the new instructions relevant for alder lake are:

AVX512VNNI, AVX512BF16,
MOVDIRI, MOVDIR64B, AVX512VP2INTERSECT, EN-
QCMD, CLDEMOTE, PTWRITE, WAITPKG, SERIALIZE,
TSXLDTRK, UINTR, and
AVX-VNNI 

Which means we do have BF16 support in AVX, the same way as Sapphire rapids, but don't have int8 at all.

Edit: source for above: GCC-11.2 manual https://gcc.gnu.org/onlinedocs/gcc-11.2.0/gcc.pdf

Edit2: tangentially related, as part of testing and profiling different suites I created a script to detect and generate compiler flags for any supported AVX512 instructions detected on a host system, it can be pipped into other tools to force the use of instructions, but doesn't include the cost file as observed above. It's a workaround, but it does help. It can be found in my repository here: https://github.com/FCLC/Choosing-a-compiler-performance-testing-GCC_ICC_ICPX_NVCC_CLANG_HIP/blob/main/Binary_tree/detect_avx.sh

[martin-frbg]

Hmmm. As you appear to have the hardware, can you confirm that switching AVX512 on/off by whatever means gets accurately reflected in the capabilities it reports e.g. through /proc/cpuinfo (or more precisely through the cpuid bits that OpenBLAS already knows to check via the support_avx512() and support_avx512_bf16() helper functions in cpuid_x86.c (build-time) and driver/others/dynamic.c (run-time in DYNAMIC_ARCH builds) ? And is it safe to assume by now that the temporarily orphaned
AVX512 hardware is producing correct results ?

[brada4]

seems to be big-small CPUs where half of cores have no avx-512.

[martin-frbg]

seems to be big-small CPUs where half of cores have no avx-512.

Err, yes, but the point is that AVX512 appears to be completely disabled by default, until one disables the little guys in the BIOS - at which point it transforms into something in between a SkylakeX and a SapphireRapids cpu.
If this is accurately reflected in the cpuid bits it would simplify checking for this particular (non-default) operating mode, and probably warrant a dedicated TARGET.

[brada4]

We would need at least /proc/cpuinfo from both modes? If any of stepping codes change, it is already instrumented, and if CPUID bits change, then too, the only danger is asymmetric CPUID-s on same system?

[FCLC]

Hmmm. As you appear to have the hardware, can you confirm that switching AVX512 on/off by whatever means gets accurately reflected in the capabilities it reports e.g. through /proc/cpuinfo (or more precisely through the cpuid bits that OpenBLAS already knows to check via the support_avx512() and support_avx512_bf16() helper functions in cpuid_x86.c (build-time) and driver/others/dynamic.c (run-time in DYNAMIC_ARCH builds) ? And is it safe to assume by now that the temporarily orphaned
AVX512 hardware is producing correct results ?

Yes, I can confirm that performance is correct (tested against known good results on each core over the course of multiple days with no results that deviated) and that, after a reboot and enabling or disabling instructions in bios instructions are reported via cpuid as having or not having the relevant instructions. (Depending on configuration)

The sources in the original post have some screen shots of the CPUID bits being changed for AVX512 enablement, but for convenience here's a system output:

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 151
model name	: 12th Gen Intel(R) Core(TM) i7-12700K
stepping	: 2
microcode	: 0x15
cpu MHz		: 3600.000
cache size	: 25600 KB
physical id	: 0
siblings	: 16
core id		: 28
cpu cores	: 8
apicid		: 57
initial apicid	: 57
fpu		: yes
fpu_exception	: yes
cpuid level	: 32
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 invpcid_single cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni avx512_bf16 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear serialize pconfig arch_lbr avx512_fp16 flush_l1d arch_capabilities
vmx flags	: vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid ple shadow_vmcs pml ept_mode_based_exec tsc_scaling usr_wait_pause
bugs		: spectre_v1 spectre_v2 spec_store_bypass swapgs
bogomips	: 7219.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

The procedure for enabling AVX512 support varies slightly based on mainboard vendor, but Asus MSI Gigabyte and AsRock have all committed (and delivered) support in bios for enabling AVX512 once the ecores are all disabled. This is exclusive to the z690 chipset as of now.

I'll have to pull the latest source for a clean build environment, but once completed I'll verify that OpenBLASs can properly detect the intructions for both build and runtime (how this then effects build flags and compiler costs would then need to be profiled I think?)

seems to be big-small CPUs where half of cores have no avx-512.

Correct, it's a hybrid architecture x86 CPU with different ISA support between the 2 cores.
Still awaiting confirmation from intel, but it also seems that only the K class chips will have the ability to enable AVX-512.

functionally that means that we have 3 chips available:
8 core i9 12900k with 30MB of L3
8 core i7 12700k with 25MB of L3
6 core i5 12600k with 20MB of L3

The difference in cache is that the L3 is shared and is still enabled even when the ecores are disabled.

between a SkylakeX and a SapphireRapids cpu.

This is correct, though the per core performance would actually partially out pace that of the Sapphire rapids version due to higher available power budget per core (and therefore frequency) and slightly increased L3 per core due to the disabled e-cores still contributing public L3

Edit:

We would need at least /proc/cpuinfo from both modes? If any of stepping codes change, it is already instrumented, and if CPUID bits change, then too, the only danger is asymmetric CPUID-s on same system?

Would you like me to provide both?

[FCLC]

find attached both cpuinfo outputs as results of cat /proc/cpuinfo >> [file].txt
cpu_info_ecores_enabled.txt
cpu_info_avx512.txt

[martin-frbg]

very quick and dirty patch (Sapphire Rapids is currently treated like Cooper Lake and I just noticed autodetection does not yet have it as a separate type):

--- cpuid_x86.c 2021-12-19 14:07:28.243783197 +0100
+++ cpuid_x86.cnew      2021-12-20 22:50:44.629040294 +0100
@@ -1495,6 +1495,10 @@
         switch (model) {
         case 7: // Alder Lake desktop
         case 10: // Alder Lake mobile
+         if(support_avx512_bf16())
+            return CPUTYPE_COOPERLAKE;
+          if(support_avx512())
+            return CPUTYPE_SKYLAKEX;
           if(support_avx2())
             return CPUTYPE_HASWELL;
           if(support_avx())

[wjc404]

How about the peak throughput of AVX512 instructions comparing to their AVX2 counterparts on Alder Lake? On some architectures, the dedicated avx512 unit on port 5 is disabled, leaving the theoretical GFLOPS of AVX512 the same as that of AVX2.

[brada4]

Also it is documented that AVX-512 anywhere switches off turbo boost, so 1-core performance also should be compared, going back even to sandybridge (AVX1 only)

[FCLC]

How about the peak throughput of AVX512 instructions comparing to their AVX2 counterparts on Alder Lake? On some architectures, the dedicated avx512 unit on port 5 is disabled, leaving the theoretical GFLOPS of AVX512 the same as that of AVX2.

I've gotten as high as 5x throughput as compared to AVX2 on alder lake for code that is very AVX512 optimized such as particle simulations.
In part this is due to how when the ecore's are disabled (a requirement to use avx512) the ring bus is then relative to the p core frequency.
Typically ring frequency = (e core multiplier -3) * base clock (aka bclk)
for the i9 12900k:
Ecore on
Max ring bus frequency of 3.6GHz, which also drives L3 cache for all cores, memory access etc.
Ecore off
Max ring bus frequency of 4.9 GHz, which also drives L3 cache for all cores, memory access etc. (you also get the keep the L3 of the disabled ecores, meaning your per core L3 cache size goes up)

The per instruction throughput per clock with including avx512 has been formally measured and can be found here:
https://github.com/InstLatx64/InstLatx64/blob/master/GenuineIntel/GenuineIntel0090672_AlderLake_BC_AVX512_InstLatX64.txt
As compared to the standard configuration without avx512 and the 8 ecores enabled
https://github.com/InstLatx64/InstLatx64/blob/master/GenuineIntel/GenuineIntel0090672_AlderLake_BC_InstLatX64.txt

Also it is documented that AVX-512 anywhere switches off turbo boost, so 1-core performance also should be compared, going back even to sandybridge (AVX1 only)

This isn't quite right, it's more that it makes turbo boost less effective.

This is dependent on generation, but I think you're referring to AVX offset. Typically a core that is executing AVX code and having to use XMM or ZMM registers generates more heat/draws more power. As a way of balancing power budget what Intel did is apply a net offset to the current turbo boost frequency. This applies for Turbo boost 2.0 and 3.0

i.e.
If a chip has a base of 3.5, and is within temperature and power limits, and a load is applied, it will boost to say (base multiplier +11)* baseclock =4.6GHz.

TurboBoost 2.0 behavior (before Alder lake)
As it executes, let's say the program then calls AVX2 code using the XMM registers (XMM is 128 bit, YMM is 256). These will typically offset current clock by -1 on multiplier, or (3.5 +11(turbo boost)-1(avx2 off set))*base clock= 4.5GHz

let's say that same program then requests AVX512 code, these use the ZMM registers. typically AVX512 is either a -2 or -3 multiplier offset. as above,
we end up at say 4.3Ghz.

Now, if we then exceed/approach power/temperature limits, turbo boost will lessen until it stabilizes. In workstations with proper cooling this will be near maximum frequency -1-2, so we end up with a sustained AVX512 frequency of 4.1GHz

Alder Lake is a little more interesting, but suffice to say, in my testing so far, I've noticed that avx2 has no offset from base standard multiplier, and avx512 does not directly apply an offset, rather it informs the internal system frequency governor to be more "careful" with maxing out clocks. This means that, with a proper cooling set up, I can happily run 8 cores of AVX512 code on the i7 12700k at 5GHz sustained for a tested 24+hrs

Note: AVX2 and AVX1 are pretty much the same in terms of actual usage/performance, you could easily refer to avx2 as avx1.1 from how small the differences were.

Note: AVX512 on the Other hand is a slight pain to deal with from a low level/asm point of view. AVX 512 is not a complete instruction set, but rather a familly of instruction sets that only guarantees a single "common" core of instructions, and vendors can choose to implement other parts of the AVX512 family of instructions as they see fit.

The AVX512 Wikipedia page has a quite good chart showing what family of instructions are supported on what hardware

https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512