Clarification on the HSA_FORCE_FINE_GRAIN_PCIE requirement

[tmh97]

Hey folks, Tom here from Cornelis networks.

We've begun using the rccl-tests suite to test the functionality of our libfabric provideropx+aws-rccl-plugin

We successfully ran these tests with no issue

• all_gather_perf
• all_reduce_perf
• broadcast_perf
• reduce_perf
• reduce_scatter_perf

These tests all fail due to Out Of Memory

• alltoall_perf
• alltoallv_perf
• gather_perf
• scatter_perf
• sendrecv_perf

When I set the HSA_FORCE_FINE_GRAIN_PCIE=1, all of the failing tests magically pass.

The docs say The HSA_FORCE_FINE_GRAIN_PCIE environment variable will need to be set to 1 in order to run the unit tests which use fine-grained memory type, however, I am running all of the tests with the Default: Coarse memory type.

I am hoping for some clarification on why this variable seems to improve behavior? Maybe some of the tests have "fine-grained memory type" by default? Any input would be greatly appreciated, thanks in advance for any help!

[nusislam]

What AMD GPU and rocm version are you using? This flag should not be required for rocm version >= 5.7

[tmh97]

ah the rocm version was my issue! I was using rocm 5.3.0

Thanks so much for the timely response!