[Issue]: Bad Pytorch performance on RX 9070 XT Windows 11

[skillmaker-dev]

Problem Description

Using RX 9070 XT for Pytorch training takes much more time, and the VRAM usage is a bit higher than the other NVIDIA cards that accomplish the same thing ~10 times faster, setting MIOPEN_FIND_ENFORCE=3 doesn't make a difference.

Operating System

Windows 11 10.0.26100

CPU

AMD Ryzen 5 9600X 6-Core Processor

GPU

AMD Radeon RX 9070 XT

ROCm Version

7.1.1

ROCm Component

No response

Steps to Reproduce

I ran this small training algorithm:

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms, models
from torch.utils.data import DataLoader
import time

def main():
    # --- Configuration ---
    BATCH_SIZE = 256 
    EPOCHS = 10
    LEARNING_RATE = 0.001
    
    if torch.cuda.is_available():
        device = torch.device("cuda")
        gpu_name = torch.cuda.get_device_name(0)
        print(f"✅ Training on: {gpu_name}")
        
        torch.backends.cudnn.benchmark = True
    else:
        print("❌ GPU not detected. Exiting.")
        return

    # --- 1. Data Preparation (CIFAR-10) ---
    print("⬇️  Preparing Data...")
    
    stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
    train_transform = transforms.Compose([
        transforms.RandomCrop(32, padding=4, padding_mode='reflect'),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(*stats)
    ])
    
    # Download dataset
    train_data = datasets.CIFAR10(root='./data_cifar', train=True, download=True, transform=train_transform)
    
    train_loader = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4, pin_memory=True)

    # --- 2. Model Setup (ResNet18) ---
    print("🧠 Initializing ResNet18...")
    model = models.resnet18(weights=None)
    model.fc = nn.Linear(model.fc.in_features, 10) 
    model = model.to(device)

    optimizer = optim.AdamW(model.parameters(), lr=LEARNING_RATE, weight_decay=1e-4)
    loss_fn = nn.CrossEntropyLoss()
    
    # Gradient Scaler for Mixed Precision (AMP)
    scaler = torch.amp.GradScaler('cuda')

    # --- 3. Training Loop ---
    print(f"\n🚀 Starting training for {EPOCHS} epochs...")
    print("-" * 65)
    
    total_start = time.time()

    for epoch in range(EPOCHS):
        model.train()
        epoch_start = time.time()
        running_loss = 0.0
        correct = 0
        total = 0
        
        for batch_idx, (data, target) in enumerate(train_loader):
            data = data.to(device, non_blocking=True)
            target = target.to(device, non_blocking=True)

            optimizer.zero_grad()

            with torch.amp.autocast('cuda'):
                output = model(data)
                loss = loss_fn(output, target)

            scaler.scale(loss).backward()
            scaler.step(optimizer)
            scaler.update()

            running_loss += loss.item()
            _, predicted = output.max(1)
            total += target.size(0)
            correct += predicted.eq(target).sum().item()

        epoch_time = time.time() - epoch_start
        acc = 100. * correct / total
        avg_loss = running_loss / len(train_loader)
        
        imgs_sec = 50000 / epoch_time
        
        mem_alloc = torch.cuda.memory_allocated() / 1024**3 # GB

        print(f"Epoch {epoch+1:02d}/{EPOCHS} | "
              f"Time: {epoch_time:.2f}s | "
              f"Speed: {imgs_sec:.0f} img/s | "
              f"Loss: {avg_loss:.4f} | "
              f"Acc: {acc:.2f}% | "
              f"VRAM: {mem_alloc:.2f}GB")

    total_time = time.time() - total_start
    print("-" * 65)
    print(f"Training Finished in {total_time:.2f} seconds.")

if __name__ == "__main__":
    main()

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

I ran the above training code on my 9070 XT on Windows using the latest preview driver with ROCm 7.1.1, however, I noticed that the performance is not that optimal, it is slower than a RTX 3090 Ti, RTX 5070, RTX 5070 TI and RTX 4090, I tested the same code agains all cards, here are the results:

RTX 4090

🚀 Starting training for 10 epochs...

Epoch 01/10 | Time: 23.13s | Speed: 2162 img/s | Loss: 1.5010 | Acc: 45.24% | VRAM: 0.37GB
Epoch 02/10 | Time: 5.77s | Speed: 8669 img/s | Loss: 1.1395 | Acc: 59.27% | VRAM: 0.37GB
Epoch 03/10 | Time: 5.69s | Speed: 8792 img/s | Loss: 0.9910 | Acc: 64.55% | VRAM: 0.37GB
Epoch 04/10 | Time: 5.70s | Speed: 8765 img/s | Loss: 0.9012 | Acc: 68.31% | VRAM: 0.37GB
Epoch 05/10 | Time: 5.79s | Speed: 8641 img/s | Loss: 0.8344 | Acc: 70.56% | VRAM: 0.37GB
Epoch 06/10 | Time: 5.91s | Speed: 8455 img/s | Loss: 0.7815 | Acc: 72.33% | VRAM: 0.37GB
Epoch 07/10 | Time: 5.41s | Speed: 9238 img/s | Loss: 0.7364 | Acc: 74.14% | VRAM: 0.37GB
Epoch 08/10 | Time: 5.51s | Speed: 9067 img/s | Loss: 0.6934 | Acc: 75.79% | VRAM: 0.37GB
Epoch 09/10 | Time: 5.36s | Speed: 9328 img/s | Loss: 0.6539 | Acc: 76.93% | VRAM: 0.37GB
Epoch 10/10 | Time: 5.45s | Speed: 9168 img/s | Loss: 0.6277 | Acc: 77.96% | VRAM: 0.37GB

---

🏁 Training Finished in 73.73 seconds.

RTX 5070 TI

🚀 Starting training for 10 epochs...

Epoch 01/10 | Time: 8.93s | Speed: 5599 img/s | Loss: 1.5014 | Acc: 45.37% | VRAM: 0.19GB
Epoch 02/10 | Time: 4.02s | Speed: 12449 img/s | Loss: 1.1598 | Acc: 58.41% | VRAM: 0.19GB
Epoch 03/10 | Time: 3.98s | Speed: 12555 img/s | Loss: 1.0013 | Acc: 64.32% | VRAM: 0.19GB
Epoch 04/10 | Time: 3.93s | Speed: 12721 img/s | Loss: 0.9109 | Acc: 67.79% | VRAM: 0.19GB
Epoch 05/10 | Time: 4.35s | Speed: 11491 img/s | Loss: 0.8385 | Acc: 70.40% | VRAM: 0.19GB
Epoch 06/10 | Time: 4.01s | Speed: 12463 img/s | Loss: 0.7735 | Acc: 72.73% | VRAM: 0.19GB
Epoch 07/10 | Time: 3.97s | Speed: 12602 img/s | Loss: 0.7341 | Acc: 74.08% | VRAM: 0.19GB
Epoch 08/10 | Time: 4.04s | Speed: 12382 img/s | Loss: 0.6844 | Acc: 75.84% | VRAM: 0.19GB
Epoch 09/10 | Time: 4.04s | Speed: 12388 img/s | Loss: 0.6625 | Acc: 76.66% | VRAM: 0.19GB
Epoch 10/10 | Time: 4.00s | Speed: 12500 img/s | Loss: 0.6313 | Acc: 77.77% | VRAM: 0.19GB

---

🏁 Training Finished in 45.28 seconds.

RTX 3090 TI

🚀 Starting training for 10 epochs...

Epoch 01/10 | Time: 58.00s | Speed: 862 img/s | Loss: 1.5238 | Acc: 44.37% | VRAM: 0.19GB
Epoch 02/10 | Time: 49.65s | Speed: 1007 img/s | Loss: 1.1807 | Acc: 57.46% | VRAM: 0.19GB
Epoch 03/10 | Time: 49.28s | Speed: 1015 img/s | Loss: 1.0188 | Acc: 63.49% | VRAM: 0.19GB
Epoch 04/10 | Time: 45.96s | Speed: 1088 img/s | Loss: 0.9191 | Acc: 67.26% | VRAM: 0.19GB
Epoch 05/10 | Time: 55.13s | Speed: 907 img/s | Loss: 0.8476 | Acc: 70.24% | VRAM: 0.19GB
Epoch 06/10 | Time: 50.17s | Speed: 997 img/s | Loss: 0.7893 | Acc: 72.36% | VRAM: 0.19GB
Epoch 07/10 | Time: 43.80s | Speed: 1142 img/s | Loss: 0.7477 | Acc: 73.69% | VRAM: 0.19GB
Epoch 08/10 | Time: 47.65s | Speed: 1049 img/s | Loss: 0.7040 | Acc: 75.28% | VRAM: 0.19GB
Epoch 09/10 | Time: 46.70s | Speed: 1071 img/s | Loss: 0.6688 | Acc: 76.51% | VRAM: 0.19GB
Epoch 10/10 | Time: 49.94s | Speed: 1001 img/s | Loss: 0.6380 | Acc: 77.83% | VRAM: 0.19GB

---

🏁 Training Finished in 496.33 seconds.

RTX 5070

🚀 Starting training for 10 epochs...

Epoch 01/10 | Time: 18.93s | Speed: 2641 img/s | Loss: 1.4961 | Acc: 45.58% | VRAM: 0.19GB
Epoch 02/10 | Time: 8.07s | Speed: 6199 img/s | Loss: 1.1673 | Acc: 58.31% | VRAM: 0.19GB
Epoch 03/10 | Time: 8.08s | Speed: 6186 img/s | Loss: 1.0195 | Acc: 63.64% | VRAM: 0.19GB
Epoch 04/10 | Time: 8.08s | Speed: 6185 img/s | Loss: 0.9173 | Acc: 67.53% | VRAM: 0.19GB
Epoch 05/10 | Time: 8.06s | Speed: 6206 img/s | Loss: 0.8420 | Acc: 70.13% | VRAM: 0.19GB
Epoch 06/10 | Time: 8.01s | Speed: 6240 img/s | Loss: 0.7952 | Acc: 71.85% | VRAM: 0.19GB
Epoch 07/10 | Time: 7.85s | Speed: 6368 img/s | Loss: 0.7447 | Acc: 73.74% | VRAM: 0.19GB
Epoch 08/10 | Time: 7.95s | Speed: 6289 img/s | Loss: 0.6954 | Acc: 75.52% | VRAM: 0.19GB
Epoch 09/10 | Time: 7.94s | Speed: 6299 img/s | Loss: 0.6677 | Acc: 76.62% | VRAM: 0.19GB
Epoch 10/10 | Time: 7.85s | Speed: 6372 img/s | Loss: 0.6376 | Acc: 77.44% | VRAM: 0.19GB

---

🏁 Training Finished in 90.84 seconds.

RX 9070 XT

🚀 Starting training for 10 epochs...

Epoch 01/10 | Time: 75.41s | Speed: 663 img/s | Loss: 1.5216 | Acc: 44.57% | VRAM: 0.73GB
Epoch 02/10 | Time: 74.85s | Speed: 668 img/s | Loss: 1.1706 | Acc: 57.78% | VRAM: 0.73GB
Epoch 03/10 | Time: 75.57s | Speed: 662 img/s | Loss: 1.0195 | Acc: 63.62% | VRAM: 0.73GB
Epoch 04/10 | Time: 75.82s | Speed: 659 img/s | Loss: 0.9204 | Acc: 67.37% | VRAM: 0.73GB
Epoch 05/10 | Time: 74.70s | Speed: 669 img/s | Loss: 0.8465 | Acc: 70.04% | VRAM: 0.73GB
Epoch 06/10 | Time: 74.16s | Speed: 674 img/s | Loss: 0.7821 | Acc: 72.55% | VRAM: 0.73GB
Epoch 07/10 | Time: 74.77s | Speed: 669 img/s | Loss: 0.7358 | Acc: 74.13% | VRAM: 0.73GB
Epoch 08/10 | Time: 74.38s | Speed: 672 img/s | Loss: 0.6927 | Acc: 75.60% | VRAM: 0.73GB
Epoch 09/10 | Time: 74.38s | Speed: 672 img/s | Loss: 0.6601 | Acc: 76.82% | VRAM: 0.73GB
Epoch 10/10 | Time: 74.11s | Speed: 675 img/s | Loss: 0.6350 | Acc: 77.74% | VRAM: 0.73GB

---

🏁 Training Finished in 748.16 seconds.

[adityas-amd]

@skillmaker-dev similar ROCm/rocm-libraries#2567, with added kernels that implements Winograd conv to increase performance. ROCm/rocm-libraries#3000

[madias123]

Even worse: I own a RX-9070XT and a RTX-5060 Ti-16GB: The 9070 is slow and completly unstable in every condition and every OS (only AI - no hardware problem) . The 250 Euro cheaper(!) 5060ti-16gb runs out of the box and is about 30-40% faster and stable as a rock.
Sorry AMD, you really missed the point again and PLEASE: remove every "AI" advertising. You're just making yourselves look ridiculous. Even the super promoted Amuse AI is dead now. (but to be honest, I use Amuse even on my NVIDIA card for some fast works)

[k-artem]

after ROCm/rocm-libraries#3000 results are next with Navi48XTW 32GB:

@skillmaker-dev thanks for report the issue and test scenario.

[skillmaker-dev]

@k-artem @adityas-amd I tried using the latest nightly build, but now the first epoch is taking too long.

Update:
On the first run using the latest nightly build:

Epoch 01/10 | Time: 70.01s | Speed: 714 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 02/10 | Time: 17.98s | Speed: 2781 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 03/10 | Time: 18.09s | Speed: 2764 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 04/10 | Time: 17.79s | Speed: 2811 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 05/10 | Time: 18.48s | Speed: 2705 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 06/10 | Time: 19.11s | Speed: 2617 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 07/10 | Time: 17.82s | Speed: 2806 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 08/10 | Time: 20.67s | Speed: 2419 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 09/10 | Time: 20.82s | Speed: 2401 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB
Epoch 10/10 | Time: 18.31s | Speed: 2731 img/s | Loss: nan | Acc: 10.00% | VRAM: 0.32GB

Training Finished in 239.08 seconds.

Second run:

Epoch 01/10 | Time: 19.68s | Speed: 2541 img/s | Loss: 1.5017 | Acc: 45.45% | VRAM: 0.32GB
Epoch 02/10 | Time: 17.94s | Speed: 2787 img/s | Loss: 1.1612 | Acc: 58.49% | VRAM: 0.32GB
Epoch 03/10 | Time: 19.48s | Speed: 2567 img/s | Loss: 0.9999 | Acc: 64.34% | VRAM: 0.32GB
Epoch 04/10 | Time: 20.17s | Speed: 2479 img/s | Loss: 0.9111 | Acc: 67.76% | VRAM: 0.32GB
Epoch 05/10 | Time: 21.81s | Speed: 2293 img/s | Loss: 0.8343 | Acc: 70.53% | VRAM: 0.32GB
Epoch 06/10 | Time: 21.19s | Speed: 2359 img/s | Loss: 0.7776 | Acc: 72.66% | VRAM: 0.32GB
Epoch 07/10 | Time: 18.01s | Speed: 2777 img/s | Loss: 0.7280 | Acc: 74.49% | VRAM: 0.32GB
Epoch 08/10 | Time: 18.50s | Speed: 2702 img/s | Loss: 0.6888 | Acc: 75.81% | VRAM: 0.32GB
Epoch 09/10 | Time: 21.39s | Speed: 2337 img/s | Loss: 0.6537 | Acc: 77.10% | VRAM: 0.32GB
Epoch 10/10 | Time: 21.62s | Speed: 2313 img/s | Loss: 0.6289 | Acc: 77.85% | VRAM: 0.32GB

Training Finished in 199.78 seconds.

I noticed something weird, the Loss on the first run is nan, what could cause this?

[k-artem]

@skillmaker-dev could you please try again with MIOPEN_FIND_MODE=2 and check first epoch timing, thanks in advance.

[skillmaker-dev]

@k-artem
The following results are after restarting the kernel on each change (fresh run).
MIOPEN_FIND_MODE=1

MIOPEN_FIND_MODE=2:

MIOPEN_FIND_MODE=3

MIOPEN_FIND_MODE=5

My computer froze, and I had to restart it

MIOPEN_FIND_MODE=6 the default one I guess.

Running the cell again without restarting the kernel no longer shows nan in the loss value (this run was using MIOPEN_FIND_MODE=3):

The above runs were using the latest rocm and pytorch nightlies.

[adityas-amd]

@skillmaker-dev if your issue is resolved, we can close this issue as you no longer see nan in loss value with MIOPEN_FIND_MODE=3.

[skillmaker-dev]

@adityas-amd The loss is different from nan only on the second run of the cell, meaning I have to run it first, get nan loss, then start it again, without restarting the kernel, if i restart the kernel it shows nan again.

This happens with all find modes.

[adityas-amd]

@skillmaker-dev I reproduced on 9070XT, I dont see nan, this was on first attempt (no kernel restarts or other terminal).. I tested this on AI PRO 9700 too, could you please try again with the lates nightly release pip install --index-url https://rocm.nightlies.amd.com/v2-staging/gfx120X-all/ --pre torch torchaudio torchvision ?