Add icelake as CPU architecture

[masterleinad]

So far I didn't touch the Makefile because I don't quite understand what kind of variables I should set for icelake. In CMake we just try -march=icelake-server -mtune=icelake-server and don't set anything else.

[dalg24]

Please confirm that KOKKOS_ENABLE_TM does not need to be set

[masterleinad]

Please confirm that KOKKOS_ENABLE_TM does not need to be set

Wikipedia says (https://en.wikipedia.org/wiki/Transactional_Synchronization_Extensions):

[...] According to Intel 64 and IA-32 Architectures Software Developer's Manual from May 2020, Volume 1, Chapter 2.5 Intel Instruction Set Architecture And Features Removed,[18] HLE has been removed from Intel products released in 2019 and later. RTM is not documented as removed. However, Intel 10th generation Comet Lake and Ice Lake CPUs, which were released in 2020, do not support TSX/TSX-NI,[28][29][30][31][32] including both HLE and RTM. Engineering versions of Comet Lake processors were still retaining TSX/TSX-NI support.

Hence, it seems that at least some Ice Lake CPUs don't support transactional memory.

[crtrott]

Why the change the AVX512? Note that there is unfortunately a difference between server grade SKX (which our setting uses) and non-server. The same for icelake. We probably need to distinguish.

[crtrott]

why this change?

[dalg24]

Would you prefer

KOKKOS_INTERNAL_USE_ARCH_AVX512XEON := $(shell expr $(KOKKOS_INTERNAL_USE_ARCH_SKX) + $(KOKKOS_INTERNAL_USE_ARCH_ICX))

and then later

ifeq ($(KOKKOS_INTERNAL_USE_ARCH_AVX512XEON), 1)
  tmp := $(call kokkos_append_header,"$H""define KOKKOS_ARCH_AVX512XEON")
endif

?

[crtrott]

Wondering about making SKX the SKX512 or SKX_SERVER
and ICX ICX512 or ICX_SERVER
The difference of the server variant is that it got AVX512
the icelake-server gcc option probably reflects that.
Then we could make SKX and ICX the none-server variant, which is I think how spack handles it?

[masterleinad]

In Makefile.kokkos KOKKOS_INTERNAL_USE_ARCH_AVX512XEON is only used to set flags for Skylake, though,

  KOKKOS_CXXFLAGS += -march=skylake-avx512 -mtune=skylake-avx512 -mrtm

Enabling it for Ice Lake wouldn't allow us to set the desired architecture flags. Hence, I decided to get rid of KOKKOS_INTERNAL_USE_ARCH_AVX512XEON altogether and just set the preprocessor variable KOKKOS_ARCH_AVX512XEON directly according to the architecture.

The alternative to this is what Damien is proposing.

[masterleinad]

Why the change the AVX512? Note that there is unfortunately a difference between server grade SKX (which our setting uses) and non-server. The same for icelake. We probably need to distinguish.

The instruction set supported for Ice Lake is more similar to Skylake than to Knights Landing so it seemed appropriate to use KOKKOS_ARCH_AVX512XEON for Ice Lake as well (noting that it's not used anywhere in Kokkos).

The flags we are using for Skylake are the server ones (march=skylake-avx512 vs. -march=skylake) AFAICT so I thought that we also want to use the server ones for Ice Lake.

[masterleinad]

Wondering about making SKX the SKX512 or SKX_SERVER
and ICX ICX512 or ICX_SERVER
The difference of the server variant is that it got AVX512
the icelake-server gcc option probably reflects that.
Then we could make SKX and ICX the none-server variant, which is I think how spack handles it?

https://spack.readthedocs.io/en/latest/basic_usage.html#support-for-specific-microarchitectures shows that spack has

GenuineIntel - x86_64
    nocona  nehalem   sandybridge  haswell    skylake  cannonlake      cascadelake
    core2   westmere  ivybridge    broadwell  mic_knl  skylake_avx512  icelake

as microarchitecture. Also, I don't know if we need to be concerned with backward compatibility. According to https://en.wikichip.org/wiki/intel/microarchitectures/ice_lake_(client) and https://en.wikichip.org/wiki/intel/microarchitectures/ice_lake_(server) both Ice Lake variants support AVX512.

[nmhamster]

Folks - SKX means Sky Lake Xeon (which is the server), SKL (Sky Lake) is the client part. It might be good to try to use this naming so we have similar terms to those that Intel uses. ICX and ICL follow similar patterns.

[crtrott]

OK I like Si's comment here.

[masterleinad]

OK I like Si's comment here.

@crtrott I think in that case this pull request does the right thing (adding IceLake server as ICX to both the CMake build system and Makefile.kokkos; SKX continues to mean Skylake server).

[crtrott]

@masterleinad can you resolve the conflict? We also should probably add SKL and ICL as Si mentioned but could do that in a different one.

[masterleinad]

@masterleinad can you resolve the conflict? We also should probably add SKL and ICL as Si mentioned but could do that in a different one.

Done. I'll add SKL and ICL in a separate pull request.

[dalg24]

Retest this please