Manually join rocprof runs

[coleramos425]

In an attempt to address kernel merging issues with default rocprofiler implementation (i.e. $ROCPROF_DIR/libexec/rocprofiler/tblextr.py), we've created join_prof().

• pmc_perf_split() separates pmc_perf.txt into 17 input files (one for each line)

• 17 separate output files (pmc_perf_*.csv) are generated by rocprof and fed into join_prof()

• join_prof() merges all 17 files into pmc_perf.csv by combining counters across kernels and averaging timestamps

[coleramos425]

Everything is looking good in my tests. Only issue I've noticed is that gpu-id, which is used in analyze mode (i.e. for gpu-id based filtering), is not accounted for at the moment
https://github.com/AMDResearch/omniperf/blob/8c173446d2909db48c6cfe024ab54205fbfef1e8/src/utils/perfagg.py#L107-L132

If gpu-id will always be consistent across all runs then it's simple to merge. However, I'm thinking there might be some edge cases where this isn't true... (i.e. MPI runs?) which may introduce some complications. Any thoughts @arghdos ?

[skyreflectedinmirrors]

However, I'm thinking there might be some edge cases where this isn't true... (i.e. MPI runs?) which may introduce some complications. Any thoughts @arghdos ?

You're probably right that there could be some cases where different MPI runs target different GPUs per process. We could probably code up some strategies the user can select from for joing such an app, but in the short term I'm less concerned about that.

Edit: this is assuming that e.g., if I do hipSetDevice(0) on two subsequent MPI runs, I get the same "gpu-id" in rocprof? Which, I think I should

[coleramos425]

I've redefined "boring duplicates" such that gpu-id required in filtering, along with wgr, vgpr, sgpr, lds, scr, grd required in wavefront launch are no longer ignored.
https://github.com/AMDResearch/omniperf/blob/444102d92726baf286d1cb321fc2eb3f1bc30ee3/src/utils/perfagg.py#L119-L144

[coleramos425]

Note we also have a --join-type option
https://github.com/AMDResearch/omniperf/blob/444102d92726baf286d1cb321fc2eb3f1bc30ee3/src/parser.py#L185-L192

https://github.com/AMDResearch/omniperf/blob/444102d92726baf286d1cb321fc2eb3f1bc30ee3/src/utils/perfagg.py#L101-L108

[skyreflectedinmirrors]

This looks good to me, minor comment about "perhaps we wanna save the individual files" for logging purposes.

Thanks @coleramos425 !

[coleramos425]

Thanks for the review. Looks good to me. Merging this into dev for others to try out.