Compiling with AMD AOCL?

[btglau]

Hello devs,

Can pyscf be compiled with the AMD equivalent of MKL, AOCL?

1. Are the AOCL libraries a drop in replacement for the functional that pyscf needs from other BLAS? I assume that to be the case, although the closed source bits of AOCL, such as LibM and amd_memcpy, look like they require changes to code.

2. Ideally, there would be a substitution for:

cmake -DBLA_VENDOR=Intel10_64lp_seq ..

however, CMake FINDBLAS does not seem to support AOCL yet, as I found a code in molcas' CMakeLists that addresses this issue:
https://github.com/Molcas/OpenMolcas/blob/master/CMakeLists.txt#L1271

[jamesETsmith]

Hey @btglau, you can try setting the path to your AOCL BLAS by setting some cmake variables in the CMakeLists.txt file. Check out these lines:

pyscf/pyscf/lib/CMakeLists.txt

Lines 73 to 85 in 9291e74

	# if unable to find mkl library, just create BLAS_LIBRARIES here, e.g.
	# set(BLAS_LIBRARIES "-L/path/to/mkl/lib -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lmkl_avx -lm")
	# or
	# set(BLAS_LIBRARIES " /path/to/mkl/lib/intel64/libmkl_intel_lp64.so")
	# set(BLAS_LIBRARIES "${BLAS_LIBRARIES};/path/to/mkl/lib/intel64/libmkl_sequential.so")
	# set(BLAS_LIBRARIES "${BLAS_LIBRARIES};/path/to/mkl/lib/intel64/libmkl_core.so")
	# set(BLAS_LIBRARIES "${BLAS_LIBRARIES};/path/to/mkl/lib/intel64/libmkl_avx.so")
	# set(BLAS_LIBRARIES "${BLAS_LIBRARIES};/path/to/mkl/lib/intel64/libmkl_def.so")
	# set(BLAS_LIBRARIES "-Wl,-rpath=${MKLROOT}/lib/intel64/ ${BLAS_LIBRARIES}")
	#
	# These settings can be written in the cmake.arch.inc file. This config file
	# will automatically load all settings specified in cmake.arch.inc
	#

[btglau]

My cluster department responded with:

"Considerable testing showed that AOCC/AOCL, at the current time, is not particularly performant compared to other options. The LAPACK implementation is incomplete, and still quite complicated to incorporate into your code. There is some drop-in support, but it's not always straightforward."

With the conclusion that compiling with MKL is the best practice as of now. I will update in the future if there is any progress with AMD's compilers.

[sunqm]

It's probably better to use MKL and the "famous" setting MKL_DEBUG_CPU_TYPE=5 for AMD cpu

https://www.pugetsystems.com/labs/hpc/How-To-Use-MKL-with-AMD-Ryzen-and-Threadripper-CPU-s-Effectively-for-Python-Numpy-And-Other-Applications-1637/

[jamesETsmith]

Just be aware that support for that env variable (MKL_DEBUG_CPU_TYPE) is being pulled for more recent versions of MKL (2020.1 and onward) see here for more details. I've had pretty good luck with BLIS as PySCF's BLAS backend on AMD machines and it's a breeze to compile so I'd recommend it. Since it's only tangentially related to this issue, we can talk more about it offline if you're interested.