Compiler Flags for Different Architectures

Last updated: 3/29/19

This page lists flags that are used to compile Spatter and STREAM comparisons on different architectures.

STREAM

Some general notes for STREAM can be found at this blog post:

Additionally:

• ICC generally will generate the best quality code for STREAM and Spatter on Intel architectures.
• Streaming loads/stores may be needed to increase performance to "peak" performance.

STREAM flags per architecture

Common flags for Intel compilers with OpenMP backend: -0fast -qopenmp -qopenmp-link=static -fargument-noalias

TBD - when do we use -ffreestanding?

Note that in many cases, you can check for vectorized instructions by generating the assembly with the -S flag or by using objdump -d <compiled_app> to look at the assembly code. As mentioned in this StackOverflow post, you want to look for instructions with names like vgatherpf0qpd.

Architecture	Short Name	Compiler	Flags	Notes
Sandy Bridge	SNB	icc	-march=sandybridge
Broadwell	BDW	icc	-march=broadwell
Skylake	SKL	icc	-march=skylake
Skylake with AVX512	SKL	icc	-march=skylake-avx512
Knight's Landing with AVX512 and MCDRAM	KNL	icpc	icpc -xCOMMON-AVX512	Compilation notes
Power9	PWR9	codexl		Use xlc_r to create thread-safe version of Spatter
			-qtune=pwr9	Tune for Power9 arch (auto tunes for arch where compiled)
			-qsimd=auto	Implied for -O3 or high opt level
			-qenablevmx	Enable vector generation

Intel-specific flags

To use HBM on KNL:

#Check mem settings
numactl -H 
#Run on NUMA mem region 1 (HBM)
numactl --membind 1 ./run-app

How do we know if code has been vectorized with a specific compiler?

ICC

Returns info on which loops were vectorized and why: -qopt-report=1 -qopt-report-phase=vec

Returns info on loops that were not vectorized and why: -qopt-report-phase=vec,loop -qopt-report=2