torch-directml: different results from cpu and directml on float16 calculation

[lshqqytiger]

First of all, thank you for developing a library like DirectML.
I've been waiting for the float16 operation to be implemented in DirectML, and I tried it because it seems to have been implemented in 0.1.13.1.dev230119.
I am a student who is not a major yet, but I submitted this issue because I thought there was a problem with the difference in some calculations between cpu and DirectML as follows.

Environment
Python 3.10.8 | packaged by conda-forge | (main, Nov 24 2022, 14:07:00)
Windows 11 Build 25247
torch==1.13.1
torch-directml==0.1.13.1.dev230119
AMD Radeon RX 5700 XT
Radeon Driver 22.11.2

Code

import torch
import torch_directml

dml = torch_directml.device()

tensor1 = torch.randn(1, dtype=torch.float16) # tensor([0.6094], dtype=torch.float16)
tensor2 = torch.randn(1, dtype=torch.float16) # tensor([-0.5024], dtype=torch.float16)

tensor1 ** tensor2 # tensor([1.2822], dtype=torch.float16)
tensor1.to(dml) ** tensor2.to(dml) # tensor([1.2822], device='privateuseone:0', dtype=torch.float16)

tensor1 = tensor1.abs() # tensor([0.6094], dtype=torch.float16)
tensor2 = tensor2.abs() # tensor([0.5024], dtype=torch.float16)

tensor1 ** tensor2 # tensor([0.7798], dtype=torch.float16)
tensor1.to(dml) ** tensor2.to(dml) # tensor([0.7793], device='privateuseone:0', dtype=torch.float16)

# I found same issue from other operators too
tensor1 = torch.randn(1, dtype=torch.float16) # tensor([0.8931], dtype=torch.float16)

tensor1 + 0.02 # tensor([0.9131], dtype=torch.float16)
tensor1.to(dml) + 0.02 # tensor([0.9126], device='privateuseone:0', dtype=torch.float16)

tensor1 / 0.02 # tensor([44.6562], dtype=torch.float16)
tensor1.to(dml) / 0.02 # tensor([44.6250], device='privateuseone:0', dtype=torch.float16)

[vt-idiot]

I may be mistaken, but I think even torch itself does not return the same results when comparing torch running on a CPU to torch with CUDA.

https://discuss.pytorch.org/t/why-different-results-when-multiplying-in-cpu-than-in-gpu/1356/5

I know it definitely doesn't return identical results once you're, for example, using torch within stable diffusion. The same settings and seed will produce different images on a CPU vs. with CUDA.

[lshqqytiger]

Thank you for your kind reply and explanation.
Actually, when I applied it to stable diffusion, I found that GroupNorm and LayerNorm had strange outputs on float16, and it did not seem to be a problem that the computation results of cpu and DirectML were different.

[vt-idiot]

No problem, best of luck with the directml fork, I'm hoping you can get it working!