[ROCm] Enable deterministic rocBLAS mode

[ashishfarmer]

The PR adds a feature to disable atomics in rocblas calls thereby making the output deterministic when it is expected in pyTorch. This mode of rocBLAS can be exercised using the global setting torch.set_deterministic(True)

cc: @ezyang @jeffdaily @sunway513

[dr-ci]

💊 CI failures summary and remediations

As of commit f2afa4c (more details on the Dr. CI page):

---

✅ None of the CI failures appear to be your fault 💚

---

• 1/1 broken upstream at merge base 98fddc1 since Nov 30

---

🚧 1 ongoing upstream failure:

These were probably caused by upstream breakages that are not fixed yet:

• pytorch_linux_xenial_py3_clang7_onnx_ort_test2 since Nov 30
  • 🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 4 times.

[jeffdaily]

LGTM

[zasdfgbnm]

What if I call rocBLAS with deterministic=True, then set it to False, and call rocBLAS again? Would the performance of the second rocBLAS hurt by this?

This handle is not just used once and discarded. It will be returned to the pool when the operation finishes, so later the rocblas_atomics_not_allowed will remain in the handle until the end of PyTorch process.

[ashishfarmer]

Thanks @zasdfgbnm for the catch. In the case when the global setting is toggled after the handle is returned to the pool, if it is used again, it will still use no_atomics mode. Just added the check to query the mode each time getCurrentCUDABlasHandle() is called.

[facebook-github-bot]

@ezyang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@ezyang merged this pull request in b2ec21a.