Add ability to assign kokkos profile function from executable

[bathmatt]

We have our own profiler which inserts things into the stacked timer. We use this with the standard interface. However, we often build with static for migration to the big systems. Therefore we'd like this

kokkosp_set_parallel_for(beginFunction* begin,  endFunction* end);

[bathmatt]

let me know if you need a hand w/ this @dalg24

[DavidPoliakoff]

@bathmatt , basically I plan to have three methods to allow profiling (and tuning).

1. Current method, KOKKOS_PROFILE_LIBRARY
2. (What you're requesting) per Kokkos callback, have a function setCallback that takes in a function and assigns that callback to the provided function.
3. KOKKOS_PROFILE_LIBRARY=builtin, which looks in the currently running executable for the symbols, rather than dlopen'ing an external library. This one is mainly to help @vbrunini , who is linking a tool into his application but still has to set KOKKOS_PROFILE_LIBRARY to point to that library, which is a clunky implementation.

Because I can't help myself, I'm having some more grandiose implementation ideas (a "suspendTools" call that saves aside all the callbacks and clears them, suspending tools, and a "resumeTools" call that restores the tool state to where they were before being suspended), but those are a start

[bathmatt]

OK, if you want to dribble these in that would be great, we have our profiler and want an input deck option like Profile: Memory or Full or Parallel Loops or Regions which turn on and off different profiling.

[DavidPoliakoff]

Once I get #2929 merged it's this issue is next on my list. Also, cool idea on the profiler working that way, I've found that to be a really nice model for codes

[DavidPoliakoff]

Done in #2982