Compilation error with Makefile.kokkos on MacOS

[dwatersg]

I have a Monte Carlo example that uses Kokkos for on-node parallelism, and wanted to see if it still worked correctly with the newest release (3.7.00). It compiled and ran successfully on Summit, Crusher, and Perlmutter, even when targeted for the appropriate GPUs on those machines. I then tried on my MacBook (just targeting the host) but got the linker error shown below. Another member of my team has said he can't compile that example with Kokkos 3.6.00 either (using Apple clang version 14.0.0).

danielwaters@Daniels-MacBook-Air:~/upcxx-extras/examples/kokkos_montecarlo[Wed Sep 28|02:21:40]$ make -j
make[1]: Entering directory '/Users/danielwaters/upcxx-extras/examples/kokkos_montecarlo'
make[1]: Nothing to be done for 'KokkosCore_config.h'. # Kokkos headers were built by previous call to make -j
make[1]: Leaving directory '/Users/danielwaters/upcxx-extras/examples/kokkos_montecarlo'
make[1]: Entering directory '/Users/danielwaters/upcxx-extras/examples/kokkos_montecarlo'
/usr/local/upcxx/bin/upcxx -L/Users/danielwaters/upcxx-extras/examples/kokkos_montecarlo MC_DartSampler.o -lkokkos -ldl -latomic -lpthread -o MC_DartSampler
ld: library not found for -latomic
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:77: MC_DartSampler] Error 1
make[1]: Leaving directory '/Users/danielwaters/upcxx-extras/examples/kokkos_montecarlo'
make: *** [Makefile:74: all] Error 2

Please also include the following items to support reproducing the bug

1. Compiler: Apple clang version 11.0.0 (clang-1100.0.33.17)
2. Kokkos release: 3.7.00
3. Platform and Operating System: macOS Catalina (10.15.7)
4. CMake configure line: Used a Makefile found in the example's directory below
5. CMake configure output: N/A
6. Application code used with Kokkos: https://bitbucket.org/berkeleylab/upcxx-extras/src/master/examples/kokkos_montecarlo/
7. Build/compile command: UPCXX_CODEMODE=debug make -j (The example's Makefile sets KOKKOS_DEVICES/ARCH)
8. KokkosCore_config.h header file (generated during the build)

/* ---------------------------------------------
Makefile constructed configuration:
----------------------------------------------*/
#if !defined(KOKKOS_MACROS_HPP) || defined(KOKKOS_CORE_CONFIG_H)
#error "Do not include KokkosCore_config.h directly; include Kokkos_Macros.hpp instead."
#else
#define KOKKOS_CORE_CONFIG_H
#endif

#define KOKKOS_VERSION 30700

/* Execution Spaces /
#define KOKKOS_ENABLE_THREADS
/ General Settings /
#define KOKKOS_ENABLE_DEPRECATED_CODE_3
#define KOKKOS_ENABLE_CXX14
#define KOKKOS_ENABLE_COMPLEX_ALIGN
#define KOKKOS_ENABLE_LIBDL
/ Optimization Settings /
/ Cuda Settings */
#define KOKKOS_ENABLE_IMPL_DESUL_ATOMICS

Any additional info
The error appears to come from this code in Makefile.Kokkos

KOKKOS_INTERNAL_COMPILER_APPLE_CLANG := $(call kokkos_has_string,$(KOKKOS_CXX_VERSION),Apple clang)
KOKKOS_INTERNAL_COMPILER_HCC := $(call kokkos_has_string,$(KOKKOS_CXX_VERSION),HCC)
KOKKOS_INTERNAL_COMPILER_GCC := $(call kokkos_has_string,$(KOKKOS_CXX_VERSION),GCC)
...
#ifeq ($(KOKKOS_INTERNAL_COMPILER_GCC), 1)
KOKKOS_LIBS += -latomic
#endif

[JBludau]

Hi @dwatersg. So if you open a terminal on your mac an type gcc --version it will print something containing Apple clang version ... apple sets up your macbook to have the command gcc (or g++) to point to clang. Now gnu Make comes along and detects g++ (I can not blame gnu for preferring their own compiler) thus setting CXX=g++ internally to be the compiler for the Makefile.kokkos. Down the line this results in the -latomic being not found.
I guess this is not a problem on Kokkos side but a problem that results from how apple-clang sets itself up and how the defaults in gnu Make deal with the result.
What can you do:

• install gcc and have it appear first on your PATH (that should convince Make to use the real gcc)
• Tell make to use another compiler for CXX. Either set it in the Makefile or in the commandline make CXX=/path-to-your-compiler/g++
• Tell make to use the apple-clang (probably best by using the line above adapted for clang)

[crtrott]

This whole libatomic business is really nasty. If someone has a decent solution to it I am all ears. Basically, its semi random whether you need the library or not (I mean its not really random, but it depends on so many factors).