ROCm fork-safe DataLoader: multiprocessing forkserver für LoRA Training

[TAIM-Tuerkiye]

ROCm fork-safe DataLoader: multiprocessing forkserver für LoRA Training
⚠️ AI-generated content — created with assistance, manually reviewed.
Date: 2026-07-01
OS: Kubuntu 24.04 LTS
GPU: AMD Radeon AI PRO R9700 / RX 9070 XT (RDNA4, gfx1201)
ROCm: 7.2
PyTorch: 2.12.1+rocm7.2 (native RDNA4 support, no HSA_OVERRIDE needed)
Model: Qwen2.5-VL-7B + LoRA (r=8...16, alpha=32)

The Problem: fork + ROCm = Silent Deadlock
When using PyTorch’s DataLoader(num_workers>0) on ROCm, you get a silent GPU deadlock — no error, no crash,
the GPU just hangs at 99-100% utilization doing nothing.
Root cause: ROCm initializes the GPU context on the first import torch. After that, os.fork() creates child
processes that inherit this GPU context. The inherited context in the worker process conflicts with the parent’s GPU
context → deadlock.
Most people’s solution: num_workers=0 — which leaves the CPU underutilized and slows training.
The Fix: forkserver start method
The solution is to use Python’s multiprocessing forkserver start method, which spawns a clean Python server
process before any GPU context exists. Workers fork from this clean server → no GPU context in workers → deadlock
free.
Script Architecture (strict order)

=== FILE: train.py ===


# 1. forkserver BEFORE any torch/transformers import
import multiprocessing as mp
_GOT_FORKSERVER = False
try:
mp.set_start_method("forkserver", force=True)
_GOT_FORKSERVER = True
except RuntimeError:
pass # worker imports will raise this — that's fine

# 2. Dataset & helpers at module level (must be importable by workers)
from torch.utils.data import Dataset
from PIL import Image

class VLMJSONLDataset(Dataset):
_processor = None # class-level cache: loaded once per worker

 @classmethod
def _get_processor(cls):
if cls._processor is None:
from transformers import AutoProcessor
cls._processor = AutoProcessor.from_pretrained(
"Qwen/Qwen2.5-VL-7B-Instruct",
trust_remote_code=True,
local_files_only=True)
cls._processor.tokenizer.padding_side = "right"
return cls._processor

 def getitem(self, idx):
proc = self._get_processor()

... load image, apply processor

img = Image.open(...).convert("RGB")
return proc(text=..., images=img, ...)

# 3. Model + training loop guarded by name
if name == "main":
from transformers import Qwen2_5_VLForConditionalGeneration
from peft import get_peft_model, LoraConfig

 model = Qwen2_5_VLForConditionalGeneration.from_pretrained(...)
model = get_peft_model(model, LoraConfig(r=8, ...))

 dl = DataLoader(
dataset,
batch_size=1,
num_workers=2, # ✅ now works!
persistent_workers=True,
prefetch_factor=2,
pin_memory=True,
multiprocessing_context="forkserver", # explicit
)

 for batch in dl:
loss = model(**batch).loss
...

Why the Lazy Processor Pattern?
The HuggingFace AutoProcessor contains a Rust-backed tokenizer that is not picklable. Instead of loading it in
init (which gets pickled to workers), load it lazily in getitem and cache it as a class variable. This way:
The Dataset object stays picklable (no processor in pickle)
Each worker loads the processor once at first getitem call
~100 MB extra RSS per worker (2 workers → ~1.3 GB total)
ResultsConfig Speed VRAM Notesnum_workers=0 (baseline) 1.3 sp/s 16.8 GB CPU bottlenecknum_workers=2 + forkserver 1.72 sp/s 16.8 GB +32% speedup+ torch.compile ❌ 0.15 sp/s — 11.5x slower with LoRA + dynamic shapes+ PYTORCH_TUNABLEOP ~1.7 sp/s + 3 min overhead — Not worth the tuning time
What NOT to do
❌ HSA_OVERRIDE_GFX_VERSION — ROCm 7.2 + PyTorch 2.12.1 has native RDNA4 (gfx1201) support. The
override causes SIGSEGV / black screen.
❌ torch.compile — With LoRA adapters and dynamic sequence lengths, CUDAGraphs are incompatible.
Results in 0.15 sp/s instead of 1.72 sp/s.
❌ PYTORCH_TUNABLEOP — ROCm 7.2 default GEMM kernels are already optimal for RDNA4. TunableOp
benchmarks for 3+ minutes with <5% gain.Full working example

Environment

export PYTORCH_HIP_ALLOC_CONF="max_split_size_mb:512,garbage_collection_threshold:0.6"

No HSA_OVERRIDE, no TUNABLEOP, no compile


# Start
python3 train.py
--dataset /path/to/data/
--num-workers 2

Reference Hardware
CPU: AMD Ryzen 9 9950X3D (16C/32T) — ~200% CPU utilization during training
GPU: Radeon AI PRO R9700 (32 GB, RDNA4) — 16.8 GB used
RAM: 96 GB system RAM

This post was generated with AI assistance. The underlying research was done on the system described above. Hope it helps
someone avoid hours of debugging