Return atomic

[ngimel]

Per title. This is not used currently in the pytorch codebase, but it is a legitimate usecase, and we have extensions that want to do that and are forced to roll their own atomic implementations for non-standard types. Whether atomic op returns old value or not should not affect performance, compiler is able to generate correct code depending on whether return value is used. https://godbolt.org/z/DBU_UW.
Atomic operations for non-standard integer types (1,2 and 8 byte-width) are left as is, with void return.

[facebook-github-bot]

@ngimel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@ngimel merged this pull request in 58d7d91.

[jeffdaily]

@ngimel would you support a PR to have a non-returning interface in addition to your change here? Since atomic return values are not currently used in the pytorch codebase, it would be self-documenting to use something like atomicAddNoReturn. Specifically for ROCm, we can better optimize when the user opts in like this. Forcing atomic to return a value prohibits our optimization today.

Which extensions are you referring to that utilize atomic return values?

[ngimel]

Some internal custom ops (optimizers mostly) want to have old value, and they are compiling for double in half because that's how dispatch macros work, so they can't use built-in atomicAdd, and are forced to copy-paste things from THCAtomic adding return value. It's a legitimate request for someone to return old value, so having no mechanism for doing that is not ideal.
Sure, we can have atomiAddNoReturn. Out of curiosity, what's preventing ROCm from doing the same thing that nvcc does, and compile to fast instruction if return is ignored, and slow instruction if it's used? I posted godbolt in the PR description illustrating nvcc behavior.