Removed the preprocessor branch for KOKKOS_ENABLE_PROFILING

[DavidPoliakoff]

You know, the one that leads to eight follow-up pull requests every time I touch Kokkos?

Importantly, anywhere we were guarding dlfcn.h with KOKKOS_ENABLE_PROFILING, I changed that to KOKKOS_ENABLE_LIBDL. Once this passes or fails CI I'll give it another look, but it would be nice if afterwards another careful developer would give it a once-over, make sure there's nothing subtle I missed

[DavidPoliakoff]

Addresses #3095

[masterleinad]

Would you mind also replacing Kokkos_ENABLE_PROFILING with Kokkos_ENABLE_LIBDL in .jenkins?

I am currently getting

/tmp/kokkos_new/core/src/impl/Kokkos_Profiling.cpp: In function ‘void Kokkos::Tools::initialize()’:
/tmp/kokkos_new/core/src/impl/Kokkos_Profiling.cpp:314:54: error: ‘RTLD_NOW’ was not declared in this scope
     firstProfileLibrary = dlopen(profileLibraryName, RTLD_NOW | RTLD_GLOBAL);
                                                      ^~~~~~~~
/tmp/kokkos_new/core/src/impl/Kokkos_Profiling.cpp:314:65: error: ‘RTLD_GLOBAL’ was not declared in this scope
     firstProfileLibrary = dlopen(profileLibraryName, RTLD_NOW | RTLD_GLOBAL);
                                                                 ^~~~~~~~~~~
/tmp/kokkos_new/core/src/impl/Kokkos_Profiling.cpp:314:27: error: ‘dlopen’ was not declared in this scope
     firstProfileLibrary = dlopen(profileLibraryName, RTLD_NOW | RTLD_GLOBAL);
                           ^~~~~~
/tmp/kokkos_new/core/src/impl/Kokkos_Profiling.cpp:314:27: note: suggested alternative: ‘popen’
     firstProfileLibrary = dlopen(profileLibraryName, RTLD_NOW | RTLD_GLOBAL);
                           ^~~~~~
                           popen
/tmp/kokkos_new/core/src/impl/Kokkos_Profiling.cpp:320:64: error: ‘dlerror’ was not declared in this scope
                 << ", RTLD_NOW | RTLD_GLOBAL) failed with " << dlerror()
                                                                ^~~~~~~
/tmp/kokkos_new/core/src/impl/Kokkos_Profiling.cpp:320:64: note: suggested alternative: ‘perror’
                 << ", RTLD_NOW | RTLD_GLOBAL) failed with " << dlerror()
                                                                ^~~~~~~
                                                                perror
/tmp/kokkos_new/core/src/impl/Kokkos_Profiling.cpp:331:17: error: ‘dlsym’ was not declared in this scope
       auto p1 = dlsym(firstProfileLibrary, "kokkosp_begin_parallel_for");
                 ^~~~~

if I try to disable LIBDL. There seem to be a few guards missing in core/src/impl/Kokkos_Profiling.cpp. Otherwise, this looks good.

Of course, you also need to rebase.

[masterleinad]

We didn't run clang-format on this file before?

[crtrott]

yeah the clang-format script does only apply to source files.

[DavidPoliakoff]

@masterleinad : I think I've addressed your comments. I'll wait until it passes CI, but if it does this is ready for another review

[masterleinad]

You should move free(envProfileCopy); in line 492 to line 423.

[DavidPoliakoff]

Good further catches, Daniel, thanks for the second pair of eyes on this

[masterleinad]

LGTM

[DavidPoliakoff]

In my mind this is ready to merge, let me know if there are any additional comments. Looking forward to it, this is a huge maintenance debt gone

[dalg24]

Why were we initializing here in the first place?

[DavidPoliakoff]

That was added by @crtrott here: fae0d00 , apparently it addressed this issue #192 , where Trilinos used to initialize execution spaces individually rather than initializing Kokkos. If we're going to change that, I'd prefer it be in another PR