Fix hardcoded TF_ROCM_AMDGPU_TARGETS

[mmakevic-amd]

Motivation

In CI, multiple mGPU tests are failing with:

Could not load RepeatBufferKernel: INTERNAL: Failed call to hipGetFuncBySymbol: hipError_t(98)

The problem is that we run them locally on gfx950 and in our Bazel command, we hardcoded:

--repo_env=TF_ROCM_AMDGPU_TARGETS=gfx90a,gfx942

Technical Details

This change will make the testing script more flexible, since it will automatically detect the present target and only build for that target

Test Plan

Needs to be tested in CI

Test Result

TBA

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[leo-automation]

Does RBE use TF_ROCM_AMDGPU_TARGETS? If yes I believe that way we'll send gfx950 when running this script on gfx950 runner to an RBE worker that runs other arch

[mmakevic-amd]

Does RBE use TF_ROCM_AMDGPU_TARGETS? If yes I believe that way we'll send gfx950 when running this script on gfx950 runner to an RBE worker that runs other arch

Ah, that makes sense. I don't think there is a good way to automatically detect the target then? Should I just add gfx950 to the hardcoded list?

[leo-automation]

Does RBE use TF_ROCM_AMDGPU_TARGETS? If yes I believe that way we'll send gfx950 when running this script on gfx950 runner to an RBE worker that runs other arch

Ah, that makes sense. I don't think there is a good way to automatically detect the target then? Should I just add gfx950 to the hardcoded list?

Yes, but I am thinking doing it in the main if block might be better. Like below

for arg in "$@"; do if [[ "$arg" == "--config=ci_multi_gpu" ]]; then TAG_FILTERS="" TEST_TARGETS=("${TEST_TARGETS_MGPU[@]}") AMDGPU_TARGETS="${TF_ROCM_SGPU_AMDGPU_TARGETS:-gfx90a,gfx942}" fi if [[ "$arg" == "--config=ci_single_gpu" ]]; then TAG_FILTERS="${TAG_FILTERS},gpu,-multi_gpu,-no_oss" TEST_TARGETS=("${TEST_TARGETS_SGPU[@]}") AMDGPU_TARGETS="${TF_ROCM_SGPU_AMDGPU_TARGETS:-gfx950}" fi done

[mmakevic-amd]

Does RBE use TF_ROCM_AMDGPU_TARGETS? If yes I believe that way we'll send gfx950 when running this script on gfx950 runner to an RBE worker that runs other arch

Ah, that makes sense. I don't think there is a good way to automatically detect the target then? Should I just add gfx950 to the hardcoded list?

Yes, but I am thinking doing it in the main if block might be better. Like below

for arg in "$@"; do if [[ "$arg" == "--config=ci_multi_gpu" ]]; then TAG_FILTERS="" TEST_TARGETS=("${TEST_TARGETS_MGPU[@]}") AMDGPU_TARGETS="${TF_ROCM_SGPU_AMDGPU_TARGETS:-gfx90a,gfx942}" fi if [[ "$arg" == "--config=ci_single_gpu" ]]; then TAG_FILTERS="${TAG_FILTERS},gpu,-multi_gpu,-no_oss" TEST_TARGETS=("${TEST_TARGETS_SGPU[@]}") AMDGPU_TARGETS="${TF_ROCM_SGPU_AMDGPU_TARGETS:-gfx950}" fi done

Fixed, please take a look

[i-chaochen]

do we still have to hard code gfx? cannot we just get the rocminfo like here? https://github.com/openxla/xla/pull/36513/changes#diff-84cc06d38378056901dad1204d79317414d28d609a1fe884b4b1b52552749ac4R71-R76

[mmakevic-amd]

I originally did that, but as @leo-amd pointed out, it won't work when using RBE (see discussion above)

[i-chaochen]

so IIUC, the latest change is to specific gfx942 for sgpu and gfx950 for mgpu?

[mmakevic-amd]

so IIUC, the latest change is to specific gfx942 for sgpu and gfx950 for mgpu?

Yes, sGPU uses gfx90a and gfx942 and mGPU gfx950