[DirectML] Divergent results between CPU and DirectML only on Intel Integrated GPU

[apleynes]

Issue

On a Windows laptop with an Intel iGPU and discrete NVIDIA GPU (NVIDIA Optimus), the results differ between CPU and DirectML running on Intel iGPU. CPU and DirectML running on NVIDIA GPU match and are within 1e-5 tolerance.

Originally discovered using ONNXRuntime DirectML (issue opened here: microsoft/onnxruntime#14214) but on further testing, torch-directml is also affected that indicates that the issue is in the underlying DirectML backend.

Using Python 3.9, torch 1.13.1 and torch-directml 0.1.13.dev221216 installed from PyPI

System information:
Intel Core i7-11800H with Intel UHD Graphics (Tiger Lake GT1)
NVIDIA RTX 3060 Laptop GPU
Laptop is running with NVIDIA Optimus to be able to switch between Intel iGPU and NVIDIA discrete GPU

Platform: Windows

OS Version: Windows 11 21H2 Build 22000.1335

ONNX Runtime Version or Commit ID
1.13.1

ONNX Runtime API
Python

Architecture
X64

Execution Provider
DirectML

Execution Provider Library Version
onnxruntime-directml 1.13.1

To Reproduce

Run the following code, updating the device IDs based on the ordering on your device:

import os
os.environ['KMP_DUPLICATE_LIB_OK'] = "TRUE"
import torch
import torch.nn as nn
import torch.onnx
import torchvision
import onnxruntime as rt
import numpy as np
import matplotlib.pyplot as plt
import torch_directml

# Print devices
print([torch_directml.device_name(d_idx) for d_idx in range(torch_directml.device_count())])
# On my machine, NVIDIA GPU is device 0, Intel iGPU is devices 1 and 2 (not sure why it's showing up twice)

import skimage.data
from torchvision.io.image import read_image
from torchvision.models.segmentation import fcn_resnet50, FCN_ResNet50_Weights
from torchvision.transforms.functional import to_pil_image

img = torch.from_numpy(skimage.data.cat().T)

## Run using CPU
# Step 1: Initialize model with the best available weights
weights = FCN_ResNet50_Weights.DEFAULT
model = fcn_resnet50(weights=weights)
model.to('cpu')
model.eval()

# Step 2: Initialize the inference transforms
preprocess = weights.transforms()

# Step 3: Apply inference preprocessing transforms
batch = preprocess(img).unsqueeze(0)

# Step 4: Use the model and visualize the prediction
with torch.no_grad():
    prediction_cpu = model(batch)["out"][0, 8]  # Class 8 is cat


## Run using NVIDIA GPU
# Step 1: Initialize model with the best available weights
weights = FCN_ResNet50_Weights.DEFAULT
model = fcn_resnet50(weights=weights)
device = torch_directml.device(0)  # !! Update device number here
model.to(device)
model.eval()

# Step 2: Initialize the inference transforms
preprocess = weights.transforms()

# Step 3: Apply inference preprocessing transforms
batch = preprocess(img).unsqueeze(0)

# Step 4: Use the model and visualize the prediction
with torch.no_grad():
    prediction_nvidia_gpu = model(batch.to(device))["out"].to('cpu')[0, 8]

## Run using Intel iGPU
# Step 1: Initialize model with the best available weights
weights = FCN_ResNet50_Weights.DEFAULT
model = fcn_resnet50(weights=weights)
device = torch_directml.device(1)  # !! Update device number here
model.to(device)
model.eval()

# Step 2: Initialize the inference transforms
preprocess = weights.transforms()

# Step 3: Apply inference preprocessing transforms
batch = preprocess(img).unsqueeze(0)

# Step 4: Use the model and visualize the prediction
with torch.no_grad():
    prediction_intel_igpu = model(batch.to(device))["out"].to('cpu')[0, 8]


# Print numerical comparisons
print(np.isclose(prediction_cpu.detach().numpy(), prediction_intel_igpu.detach().numpy(), atol=1e-4).all())  # Returns False, only returns True around atol=1e1
print(np.isclose(prediction_cpu.detach().numpy(), prediction_nvidia_gpu.detach().numpy(), atol=1e-5).all())  # Returns True

# Display image results
plt.figure()
plt.subplot(1, 4, 1)
plt.imshow(np.transpose(img, [1, 2, 0]))
plt.subplot(1, 4, 2)
plt.imshow(np.squeeze(prediction_cpu.detach().numpy()))
plt.colorbar()
plt.subplot(1, 4, 3)
plt.imshow(np.squeeze(prediction_intel_igpu.detach().numpy()))
plt.colorbar()
plt.subplot(1, 4, 4)
plt.imshow(np.squeeze(prediction_nvidia_gpu.detach().numpy()))
plt.colorbar()
# Note that colorbar scales will be different

Other models have the same issue.

[schattenan]

I noticed something similar with AMD GPUs vs CPU on two different PCs with a Radeon Pro WX 3200 and a 6650 XT. Both PCs use Windows 10 22H2 with the latest GPU drivers.

torch 1.13.0
torch-directml 0.1.13.dev221206
torchvision 0.14.0

The bug I noticed occurs after I call .eval() with a Resnet model, but only if the model is trained with DirectML. I wrote a little test script that can easily reproduce the bug with the following code:

import torchvision
import torch
import torch_directml

#Settings
epochs = 1000
use_directml = False

# Get Dataset
input = torch.rand(size=(1,3,128,128))
output = torch.rand(size=(1,1))

# Create AI model
model = torchvision.models.resnet18(weights=None, progress=False)
model.fc = torch.nn.Linear(512, 1)

# Set up training
device = torch_directml.device() if use_directml else 'cpu'
print(device)
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
criterion = torch.nn.L1Loss().to(device)

# Train
for epoch in range(epochs):
    input_data = input.to(device)
    output_desired = output.to(device)

    optimizer.zero_grad()
    output_predicted = model(input_data)
    loss = criterion(output_predicted, output_desired)
    loss.backward()
    optimizer.step()

    if (epoch+1) % 100 == 0:
        print({
            'epoch': epoch + 1,
            'loss': loss.item()
        })
print("--- Training done ---")

# Eval #1 before eval()
with torch.no_grad():
    input_data = input.to(device)
    output_desired = output.to(device)
    output_predicted = model(input_data)
    loss = criterion(output_predicted, output_desired)
    print("Test before eval()")
    print({
        'loss': loss.item()
    })

# Eval #2 after eval()
model.eval()

with torch.no_grad():
    input_data = input.to(device)
    output_desired = output.to(device)
    output_predicted = model(input_data)
    loss = criterion(output_predicted, output_desired)
    print("Test after eval()")
    print({
        'loss': loss.item()
    })

With use_directml = True I get following output:

privateuseone:0
{'epoch': 100, 'loss': 0.053089261054992676}
{'epoch': 200, 'loss': 0.0057080090045928955}
{'epoch': 300, 'loss': 0.010273993015289307}
{'epoch': 400, 'loss': 0.019928857684135437}
{'epoch': 500, 'loss': 0.043615713715553284}
{'epoch': 600, 'loss': 0.01498788595199585}
{'epoch': 700, 'loss': 0.007034868001937866}
{'epoch': 800, 'loss': 0.039565831422805786}
{'epoch': 900, 'loss': 0.00014317035675048828}
{'epoch': 1000, 'loss': 0.010241135954856873}
--- Training done ---
Test before eval()
{'loss': 0.011138230562210083}
Test after eval()
{'loss': 0.8617944121360779}

With use_directml = False (cpu training):

cpu
{'epoch': 100, 'loss': 0.008720815181732178}
{'epoch': 200, 'loss': 0.005333110690116882}
{'epoch': 300, 'loss': 0.011333465576171875}
{'epoch': 400, 'loss': 0.07911093533039093}
{'epoch': 500, 'loss': 5.163252353668213e-05}
{'epoch': 600, 'loss': 0.02338549494743347}
{'epoch': 700, 'loss': 0.055452004075050354}
{'epoch': 800, 'loss': 0.024184972047805786}
{'epoch': 900, 'loss': 0.041872113943099976}
{'epoch': 1000, 'loss': 0.017582297325134277}
--- Training done ---
Test before eval()
{'loss': 0.03921814262866974}
Test after eval()
{'loss': 0.04380892217159271}

It is strange how big the difference is between cpu training after eval() and directml training after eval() is. The loss for the cpu is barely different after .eval(). My assumption is that something bad is happening with the BatchNorm2d layers and how they keep their running stats.

[schattenan]

Just as expected, it seems that BatchNorm2d is the problem. DirectML training works fine with a small custom model without BatchNorm2d.

model = torch.nn.Sequential(
    torch.nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False),
    #torch.nn.BatchNorm2d(64),
    torch.nn.ReLU(inplace=True),
    torch.nn.MaxPool2d(kernel_size=3, stride=2, padding=1),
    torch.nn.AdaptiveAvgPool2d((1, 1)),
    torch.nn.Flatten(),
    torch.nn.Linear(64, 1)
)

privateuseone:0
{'epoch': 100, 'loss': 0.0033898353576660156}
{'epoch': 200, 'loss': 0.003071129322052002}
{'epoch': 300, 'loss': 0.0030232667922973633}
{'epoch': 400, 'loss': 0.0027652978897094727}
{'epoch': 500, 'loss': 0.0030717849731445312}
{'epoch': 600, 'loss': 0.0029827356338500977}
{'epoch': 700, 'loss': 0.005253493785858154}
{'epoch': 800, 'loss': 0.00330430269241333}
{'epoch': 900, 'loss': 0.0006208419799804688}
{'epoch': 1000, 'loss': 0.0025247931480407715}
--- Training done ---
Test before eval()
{'loss': 0.003795027732849121}
Test after eval()
{'loss': 0.003795027732849121}

So there seems to be a problem in how a BatchNorm2d layer trained with DirectML is converted from train() to eval().

[apleynes]

Thanks for reproducing the issue and the additional investigation @schattenan .

I did more tests using your code and the simple model. It seems to be an issue with most of the *Norm layers even before eval() occurs.

import torchvision
import torch
import torch_directml

#Settings
epochs = 1000
use_directml = False

# Get Dataset
input = torch.rand(size=(1,3,128,128))
output = torch.rand(size=(1,1))

# Create AI model
# model = torchvision.models.resnet18(weights=None, progress=False)
# model.fc = torch.nn.Linear(512, 1)
model = torch.nn.Sequential(
    torch.nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False),
    # torch.nn.BatchNorm2d(64),
    torch.nn.InstanceNorm2d(64),
    # torch.nn.LayerNorm([64, 64]),
    # torch.nn.GroupNorm(4, 64),
    torch.nn.ReLU(inplace=True),
    torch.nn.MaxPool2d(kernel_size=3, stride=2, padding=1),
    torch.nn.AdaptiveAvgPool2d((1, 1)),
    torch.nn.Flatten(),
    torch.nn.Linear(64, 1)
)

# Set up training
device = torch_directml.device() if use_directml else 'cpu'
print(device)
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
criterion = torch.nn.L1Loss().to(device)

# Train
for epoch in range(epochs):
    input_data = input.to(device)
    output_desired = output.to(device)

    optimizer.zero_grad()
    output_predicted = model(input_data)
    loss = criterion(output_predicted, output_desired)
    loss.backward()
    optimizer.step()

    if (epoch+1) % 100 == 0:
        print({
            'epoch': epoch + 1,
            'loss': loss.item()
        })
print("--- Training done ---")

# Eval #1 before eval()
with torch.no_grad():
    input_data = input.to(device)
    output_desired = output.to(device)
    output_predicted = model(input_data)
    loss = criterion(output_predicted, output_desired)
    print("Test before eval()")
    print({
        'loss': loss.item()
    })

# Eval #2 after eval()
model.eval()

with torch.no_grad():
    input_data = input.to(device)
    output_desired = output.to(device)
    output_predicted = model(input_data)
    loss = criterion(output_predicted, output_desired)
    print("Test after eval()")
    print({
        'loss': loss.item()
    })

Using BatchNorm2d behaves as you've described on both Intel iGPU and NVIDIA RTX 3060 Laptop GPU.

Using InstanceNorm2d works on CPU but runs into the following error when direct_ml = True on both Intel iGPU and NVIDIA GPU:

RuntimeError                              Traceback (most recent call last)
Cell In[29], line 42
     39 output_desired = output.to(device)
     41 optimizer.zero_grad()
---> 42 output_predicted = model(input_data)
     43 loss = criterion(output_predicted, output_desired)
     44 loss.backward()
...
File C:\anaconda3\envs\torch_directml_20230110\lib\site-packages\torch\nn\functional.py:2495, in instance_norm(input, running_mean, running_var, weight, bias, use_input_stats, momentum, eps)
   [2493](file:///c%3A/anaconda3/envs/torch_directml_20230110/lib/site-packages/torch/nn/functional.py?line=2492) if use_input_stats:
   [2494](file:///c%3A/anaconda3/envs/torch_directml_20230110/lib/site-packages/torch/nn/functional.py?line=2493)     _verify_spatial_size(input.size())
-> [2495](file:///c%3A/anaconda3/envs/torch_directml_20230110/lib/site-packages/torch/nn/functional.py?line=2494) return torch.instance_norm(
   [2496](file:///c%3A/anaconda3/envs/torch_directml_20230110/lib/site-packages/torch/nn/functional.py?line=2495)     input, weight, bias, running_mean, running_var, use_input_stats, momentum, eps, torch.backends.cudnn.enabled
   [2497](file:///c%3A/anaconda3/envs/torch_directml_20230110/lib/site-packages/torch/nn/functional.py?line=2496) )

RuntimeError: shape '[1, 64, 1, 1]' is invalid for input of size 0

Using LayerNorm works on both CPU and direct_ml = True

Using GroupNorm works on CPU but runs into the following error when direct_ml = True on both Intel iGPU and NVIDIA GPU:

RuntimeError                              Traceback (most recent call last)
Cell In[29], line 42
     39 output_desired = output.to(device)
     41 optimizer.zero_grad()
---> 42 output_predicted = model(input_data)
     43 loss = criterion(output_predicted, output_desired)
     44 loss.backward()
...
File C:\anaconda3\envs\torch_directml_20230110\lib\site-packages\torch\nn\functional.py:2528, in group_norm(input, num_groups, weight, bias, eps)
   [2526](file:///c%3A/anaconda3/envs/torch_directml_20230110/lib/site-packages/torch/nn/functional.py?line=2525)     return handle_torch_function(group_norm, (input, weight, bias,), input, num_groups, weight=weight, bias=bias, eps=eps)
   [2527](file:///c%3A/anaconda3/envs/torch_directml_20230110/lib/site-packages/torch/nn/functional.py?line=2526) _verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:]))
-> [2528](file:///c%3A/anaconda3/envs/torch_directml_20230110/lib/site-packages/torch/nn/functional.py?line=2527) return torch.group_norm(input, num_groups, weight, bias, eps, torch.backends.cudnn.enabled)

RuntimeError: The parameter is incorrect.

[schattenan]

I played around a bit more and did some additional printf debugging.
I noticed something interesting with Resnet18 during training. It seems that the running mean and the running var for the batchnorm layer is not being updated at all with DirectML. That might be why the L1 loss is suddenly jumping so much after calling .eval().

I don't know if this should become a new issue or is still related to the current one...

import torchvision
import torch
import torch_directml

#Settings
epochs = 10

# Get Dataset
input = torch.rand(size=(32,3,128,128))
output = torch.rand(size=(32,1))

# Create AI model
directml_model = torchvision.models.resnet18(weights=None, progress=False)
directml_model.fc = torch.nn.Linear(512, 1)

# Set up training
device = torch_directml.device()
directml_model = directml_model.to(device)
optimizer_directml = torch.optim.Adam(directml_model.parameters(), lr=0.00001)
classifier_directml = torch.nn.L1Loss().to(device)

# Train
for epoch in range(epochs):
    input_data = input.to(device)
    output_desired = output.to(device)

    optimizer_directml.zero_grad()
    output_predicted_directml = directml_model(input_data)
    loss_directml = classifier_directml(output_predicted_directml, output_desired)
    loss_directml.backward()
    optimizer_directml.step()

    print((loss_directml.item(), directml_model.bn1.running_mean, directml_model.bn1.running_var))
print("--- Training done ---")

[AtiqurRahmanAni]

I am facing exactly the same issue with my resnet34 pre-trined model. Validation loss does not decrease during training time. But in CPU and NVIDIA GPU, it works totally fine, I have investigated further but didn't get any solution to this problem. I have to use google colab GPU for this issue even though I have GPU. Can you guys give me a solution? My detailed post on this issue.

[spitis]

Also having this issue on AMD 6800H.

[elephantpanda]

I had a similar problem. The Stable diffusion Unet float16 works fine in CPU mode but gives garbage in DirectML mode on integrated Intel UHD graphics. (I have tested the DirectML on Nvidia P5000 and it works fine).
I have 12GB RAM so this was pushing the CPU to the limit.
Even though Intel UHD is too slow to be practical for Stable Diffusion, it would still be nice if it gave correct results even if you have to wait 10x as long for it!

Not sure if this is a problem with DirectML.dll, the UHD graphics driver or the Intel chip itself. (From reading the above seems like it is a DirectML.dll problem?????)

However, the VAE encoder and decoder both work fine.

Is there a work around? It seems like there isn't apart from wait until the next update of DirectML.dll

[zhangxiang1993]

Hi,
Thanks everyone for the investigations. We have the issue repoed with the code snippet below.
And it turns out that we didn't update the running_mean and running_var properly during training for BatchNorm, so the results from BatchNorm eval() diverge between CPU and DirectML. We have this fixed since 0.1.13.1.dev230119

import torch
import torch.nn as nn
import torch_directml
import copy
            
dml = torch_directml.device()

input = torch.randn(1,3,2,2).requires_grad_(True)
input_dml = input.clone().detach().requires_grad_(True).to(dml)

bn = nn.Sequential(nn.BatchNorm2d(3))
bn_dml = copy.deepcopy(bn).to(dml)

bn.train()
bn_dml.train()

bn(input).mean().backward()
bn_dml(input_dml).mean().backward()

bn.eval()
bn_dml.eval()

result = bn(input)
result_dml = bn_dml(input_dml)

print (bn[0].running_mean, bn[0].running_var)
print (bn_dml[0].running_mean.to("cpu"), bn_dml[0].running_var.to("cpu"))
print (result)
print (result_dml.to("cpu"))

results before 0.1.13.dev221216

tensor([-0.0361,  0.0263, -0.0174]) tensor([1.0693, 0.9980, 1.1509])
tensor([0., 0., 0.]) tensor([1., 1., 1.])
tensor([[[[-1.7556, -0.6313],
          [-0.1517,  1.2813]],

         [[ 1.4108, -0.5215],
          [-0.6443,  0.7028]],

         [[ 1.8491, -0.9036],
          [ 0.0154, -1.5458]]]], grad_fn=<NativeBatchNormBackward0>)
tensor([[[[-1.8516e+00, -6.8894e-01],
          [-1.9300e-01,  1.2888e+00]],

         [[ 1.4358e+00, -4.9473e-01],
          [-6.1738e-01,  7.2846e-01]],

         [[ 1.9663e+00, -9.8677e-01],
          [-8.9308e-04, -1.6758e+00]]]], grad_fn=<ToCopyBackward0>)

Results after fix

tensor([ 0.0219, -0.0896, -0.0286]) tensor([0.9171, 1.2682, 0.9420])
tensor([ 0.0219, -0.0896, -0.0286]) tensor([0.9171, 1.2682, 0.9420])
tensor([[[[-0.0474,  0.6663],
          [ 0.4644, -0.2603]],

         [[-2.0938, -2.0224],
          [ 1.4924, -0.2391]],

         [[-1.0361, -0.5650],
          [ 0.4722,  0.0674]]]], grad_fn=<NativeBatchNormBackward0>)
tensor([[[[-0.0474,  0.6663],
          [ 0.4644, -0.2603]],

         [[-2.0938, -2.0224],
          [ 1.4924, -0.2391]],

         [[-1.0361, -0.5650],
          [ 0.4722,  0.0674]]]], grad_fn=<ToCopyBackward0>)

[elephantpanda]

So what is the fix? Update DirectML.dll or onnxtunrime.dll?