HIP program state re-initialization logic

[whchung]

This PR tries to re-initialize HIP runtime data structures in program_state.cpp. In applications such as TensorFlow it was evident that HIP kernels within shared libraries may not be identified if they are loaded later after initialization by dlopen(). Instead of raising an exception immediately we re-initialize data structures within program_state.cpp with a rebuild flag.

[whchung]

/cc @adityaaatluri @sabreshao for awareness

The patch to HIP for program state re-initialization has been updated so it could possibly be merged with the tip of HIP. Shall this PR be merged projects such as RCCL & PaddlePaddle should run properly.

[whchung]

Build issue in CI #5 should have nothing to do with the PR as the PR has had successfully CI build #4 before.

Meanwhile please hold this PR as we need to make sure end applications do function properly with this PR.

[mangupta]

@whchung Is this PR fine to merge now? Or do we still need to verify some end applications?

[whchung]

TensorFlow is fine but PaddlePaddle is not. Dynamic shared library loading behavior of PaddlePaddle is more complicated and we’ll hit ROCR runtime limitations. The PR needs to be improved to only construct HSA executables for newly identified code objects.

[whchung]

Working on reducing data structures need to be reinitialized when we hit a missing kernel function address. Pushed one change to minimize number of HSA executables to be created. Saw perf improvement on ROCm 1.7.2 but the gain was reduced on ROCm 1.8.1. Still has room for improvement.

[whchung]

@mangupta
TensorFlow nightly builds have been adopting the updated PR for 3 days and everything looks fine. Now we just need confirmation from PaddlePaddle team so we can merge this in.

[whchung]

@mangupta Earlier today PaddlePaddle team reported an error and I've fixed it in commit# 32789a8 . Now both PaddlePaddle and TensorFlow could properly load dynamic libs without bumping into ROCR runtime limits. Please help review this PR once again and consider merge it. Thanks.

Also it would be preferable if this PR could be made into roc-1.8.x release branch for upcoming 1.8.2 release.

[jichangjichang]

With below 3 commits, PaddlePaddle can dynamic load rccl library and run rccl kernels normally.

• HIP program state re-initialization logic
• Improve performance of re-initialization logic
• Keep the map which tracks GPU kernel symbols to grow monotonically

[whchung]

@mangupta thanks!

@parallelo now HIP PR 457 has been merged I’ll submit a PR to change Dockerfile for tensorflow build