Kokkos Tool integration hooks

[DavidPoliakoff]

This adds Kokkos Tool tracking capabilities to the existing pretty print capability.

If you load the included tool, adds a printViewMetadata command that will show where a View was allocated, as well as a history of where it was deep_copied to and from.

Mostly meant as a proof of concept, I think this method in which Kokkos exports events, which a Tool tracks, and which GDB can look up, will be a really powerful one. Follow-up should include looking at what all debugging cases people would like supported

[csiefer2]

So, the procedure is:

1. Build the tool
2. Set Kokkos profiling environment variable to use the tool.
3. Run code with GDB
4. Use printViewMetadata at a breakpoint to see the goodies

Is that right?

[DavidPoliakoff]

Close. You also have to build the Python support in GDBKokkos itself, as described here

https://github.com/Char-Aznable/GDBKokkos/blob/master/README.md

You'll need a GDB that supports Python 3 (kokkos-dev-2 didn't, not sure about other platforms). Then just pip install GDBKokkos, and you should be able to have the process you outlined work

[csiefer2]

Oh. You do realize that the second you said "just pip install" the likelihood of me getting this working dropped radically? The pain threshold of getting a functional pip environment on one of our machines is really, really, high.

I'll still try it though...

[DavidPoliakoff]

@csiefer2: if I can get you to say "this is really useful, it's just a real PITA to install," that would be progress. What we really need is to have installs of this in places. If only I had somebody who worked on a really important Sandia code who could yell at me about needing regular installs of this...

[csiefer2]

Wait. I need this version of GDBKokkos, don't I?

Because that needs ctypes. Pip has immense trouble finding ctypes on RHEL7 machines. So it won't install. Also, this needs not just python3, but python 3.8.10 or greater, which we have nowhere (except for the one system I installed it on myself).

@DavidPoliakoff I need to get it working somewhere once before I can decide it is actually useful. As it is there's nowhere I can even evaluate this.

[Char-Aznable]

@csiefer2 For this PR, you can checkout the DavidPoliakoff:feature/tool-integration branch and do a local pip install -e .. For the python version requirement, I highly doubt there is hard requirement on the python subversion but the real requirement is on the numpy and pandas versions so I bet it would work for python 3.8 or maybe 3.7. The thing with 3.8.10 is that most of the conda packages required for running the unit test has been tagged with 3.8.10 and I haven't spent time to make it right.

[csiefer2]

I have access to Python 3.7+ on exactly two machines. One of them has the pip/ctypes/RHEL problem, so no dice. I did finally drop the required version to 3.7.0 on the other machine (Ubuntu) and it installed.

Unfortunately that machine doesn't have a new enough version of cmake to build kokkos. So now I have to build that. Or I would be able to compile that if the machine had gmake. Or ninja.

sigh

I give up.

[Char-Aznable]

@DavidPoliakoff For the ToolExtensions/kp_gdb_extension.cpp, are the C-link functions basically callback functions from the push region call as you showed me on slack:

  Kokkos::Tools::pushRegion("dogs");
  Kokkos::View<float***> left("left", 10, 10, 10);
  Kokkos::Tools::pushRegion("other_dogs");
  Kokkos::View<float***> right("right", 10, 10, 10);
  Kokkos::deep_copy(left,right);
  Kokkos::Tools::popRegion();
  Kokkos::Tools::popRegion();

i.e., Kokkos::Tools has some mechanism to call kokkosp_push_profile_region() and kokkosp_begin_deep_copy() and then these routines in ToolExtensions/kp_gdb_extension.cpp set up data structure to hold the history of the view? I guess I don't know the Kokkos::Tools enough to understand what's actually happening.

If my guess is correct, it seems it would be possible to do the region_stack entirely in GDB via python, does that sound right? Basically, set up a break point at pushRegion() and popRegion() and register them in python and move on, etc. The motivation is that it might be easier to integrate this with other parts of the tool or future extension done in python in the same language and it maybe easier to distribute the package (pure python vs python + c++).

[Char-Aznable]

@csiefer2 I had to deal with similar situation on old machine. My ugly solution was to use linuxbrew, which had its own compiler tool chain so I don't have to rely on the OS ancient glibc but i don't know if that's the problem you are running into

[DavidPoliakoff]

@DavidPoliakoff For the ToolExtensions/kp_gdb_extension.cpp, are the C-link functions basically callback functions from the push region call as you showed me on slack:

  Kokkos::Tools::pushRegion("dogs");
  Kokkos::View<float***> left("left", 10, 10, 10);
  Kokkos::Tools::pushRegion("other_dogs");
  Kokkos::View<float***> right("right", 10, 10, 10);
  Kokkos::deep_copy(left,right);
  Kokkos::Tools::popRegion();
  Kokkos::Tools::popRegion();

i.e., Kokkos::Tools has some mechanism to call kokkosp_push_profile_region() and kokkosp_begin_deep_copy() and then these routines in ToolExtensions/kp_gdb_extension.cpp set up data structure to hold the history of the view? I guess I don't know the Kokkos::Tools enough to understand what's actually happening.

If my guess is correct, it seems it would be possible to do the region_stack entirely in GDB via python, does that sound right? Basically, set up a break point at pushRegion() and popRegion() and register them in python and move on, etc. The motivation is that it might be easier to integrate this with other parts of the tool or future extension done in python in the same language and it maybe easier to distribute the package (pure python vs python + c++).

@Char-Aznable : so this is likely to be technically feasible, and you have our system figured out right (let me know if you ever want more details on Tools, I can talk you through this). The problem would be that I'm a lot more familiar with the methodology I've implemented here. If we say it happens in Python, that's a huge amount of debugging scripting to figure out. My recommendation would be to merge PRs like this (using a tool library), and if in the future we have problems with usability we can look at integrating functionality into the Python scripts. Just a recommendation, though

[csiefer2]

I tried again using a new python (sems-beta 3.9) and now I can actually build the software. But GDB refuses to py import GDBKokkos, probably because GDB insists on using an older python.

[Char-Aznable]

@csiefer2 I believe GDB is hardcoded to invoke the python version which it was built against so you would need to use that particular python version's pip install to install the package.

@DavidPoliakoff Sorry for a long delay on this due to my getting overwhelmed by a research project. Probably won't have time to come back to work on this until after the holiday...

[csiefer2]

@Char-Aznable Yeah, we already established that won't work. I've filed a ticket to get the folks who built python to build GDB too.