You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We've begun using the rccl-tests suite to test the functionality of our libfabric provideropx+aws-rccl-plugin
We successfully ran these tests with no issue
all_gather_perf
all_reduce_perf
broadcast_perf
reduce_perf
reduce_scatter_perf
These tests all fail due to Out Of Memory
alltoall_perf
alltoallv_perf
gather_perf
scatter_perf
sendrecv_perf
When I set the HSA_FORCE_FINE_GRAIN_PCIE=1, all of the failing tests magically pass.
The docs say The HSA_FORCE_FINE_GRAIN_PCIE environment variable will need to be set to 1 in order to run the unit tests which use fine-grained memory type, however, I am running all of the tests with the Default: Coarse memory type.
I am hoping for some clarification on why this variable seems to improve behavior? Maybe some of the tests have "fine-grained memory type" by default? Any input would be greatly appreciated, thanks in advance for any help!
The text was updated successfully, but these errors were encountered:
Hey folks, Tom here from Cornelis networks.
We've begun using the
rccl-tests
suite to test the functionality of our libfabric provideropx
+aws-rccl-plugin
We successfully ran these tests with no issue
These tests all fail due to Out Of Memory
When I set the
HSA_FORCE_FINE_GRAIN_PCIE=1
, all of the failing tests magically pass.The docs say
The HSA_FORCE_FINE_GRAIN_PCIE
environment variable will need to be set to 1 in order to run the unit tests which use fine-grained memory type, however, I am running all of the tests with theDefault: Coarse
memory type.I am hoping for some clarification on why this variable seems to improve behavior? Maybe some of the tests have "fine-grained memory type" by default? Any input would be greatly appreciated, thanks in advance for any help!
The text was updated successfully, but these errors were encountered: