[Bug] Fine-Tuning uses much more VRAM than advertised, causing OOMs; cannot actually fine-tune any big models

[mags0ft]

Hi,

I've continued to try out a few things in Unsloth Studio and noticed a weird behavior that might not be solely limited to Unsloth Studio, but to the library itself. I made sure to update everything beforehand.

Inexplicably to me, training on a dataset in a ChatML-like format (with a messages column where an array of objects with role and content keys are present) consumes much, much more VRAM than training on an Alpaca-like dataset (where there are only two columns, one for prompt and one for response). I might be messing up my terminilogy here though, so please correct me if I'm wrong.

I think this is super weird, as to my best knowledge about LLMs and training, the format of the dataset should absolutely not matter at all. Unsloth Studio is applying the model's own chat template to the data anyways, or am I getting this wrong?

Enough yapping, let me give you an example. Let's say we have this configuration:

# Base model:
unsloth/Qwen3.5-2B
QLoRA, 4-bit

# Hardware:
12 GB VRAM NVIDIA RTX 4070 Ti
32 GB system RAM (CPU)

# Training config:
Batch size: 2
Gradient accumulation steps: 4
Learning rate: 0.00005
Optimizer: AdamW 8-bit
Max steps: 30
Context length: 4096
Warmup steps: 5

# LoRA config:
Rank: 16
Alpha: 16
Dropout: 0
Variant: lora

The only thing we're changing is the dataset.

Let's first try a dataset where we only have prompt and response columns, so "Alpaca-like": gsm8k by OpenAI. Training on this dataset, with thousands of rows, is absolutely no problem. The model trains just fine, finishes, and can be used.

Swapping out the dataset by a ChatML one, no matter how small, we'll run into an OOM; I tried the first tiny dataset I came across on Hugging Face, in this case it's TeichAI/glm-4.6-250x. 250 rows only, ChatML format, nothing else. Same configuration, but we run into an OOM:

CUDA out of memory. Tried to allocate 7.58 GiB. GPU 0 has a total capacity of 11.56 GiB of which 4.07 GiB is free. Including non-PyTorch memory, this process has 6.83 GiB memory in use. Of the allocated memory 5.90 GiB is allocated by PyTorch, and 686.61 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

So weird, isn't it? I can't explain this to myself! So, I thought, maybe the dataset is just glitchy and doesn't follow some formatting rule. I made my own, definitely compliant and fully featured dataset and tried again, but even then, I get the same kind of OOM. Then I made an Alpaca-like custom dataset, and would you look at that: Trains just fine.

No idea what the issue could be... Let me remind you: this is a 4-bit 2B model @ 4096 context length training a r = 16, alpha = 16 QLoRA adapter on a 12 GB VRAM card. This shouldn't be a problem at all... likely a bug!

I hope we can get this fixed! I'll be happy to help in case there are any further questions.

Alpaca-like (works)	ChatML-like (broken)

Training configuration:

training:
  max_seq_length: 4096
  num_epochs: 1
  learning_rate: 0.00005
  batch_size: 2
  gradient_accumulation_steps: 8
  warmup_steps: 5
  max_steps: 30
  save_steps: 30
  eval_steps: 0.1
  weight_decay: 0.01
  random_seed: 3407
  packing: false
  train_on_completions: false
  gradient_checkpointing: unsloth
  optim: adamw_8bit
  lr_scheduler_type: linear
lora:
  lora_r: 16
  lora_alpha: 16
  lora_dropout: 0
  target_modules:
    - q_proj
    - k_proj
    - v_proj
    - gate_proj
    - up_proj
    - down_proj
    - o_proj
  use_rslora: false
  use_loftq: false
  finetune_vision_layers: true
  finetune_language_layers: true
  finetune_attention_modules: true
  finetune_mlp_modules: true

Best regards

[mags0ft]

Also interesting: Trying to train on allenai/olmo-2-hard-coded gives me:

Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)

(even though it has the same standard ChatML format with the "messages" column)

[rolandtannous]

@mags0ft what is the model in the first error? and if its custom , what is its base model?

[mags0ft]

It's always unsloth/Qwen3.5-2B at 4 bit, 4096 tokens context length.

[mykeehu]

I've also noticed that even though I'm using a Hermes 8B with 4-bit model load with a 64/64 LR/alpha and a 4096-pixel context, along with a 2/8 batch/gradient, it's already using well over 24 GB of VRAM (currently 38 GB). Is this normal?

[mags0ft]

Is this normal?

Hmm, at least it doesn't sound normal on first glance. Unsloth advertises a rather low VRAM usage and that's what I've been able to reproduce using many different notebook configurations already. 38 GB of VRAM for fine-tuning a 4-bit 8B model @ 4096 ctx is extremely high. I assume you mean that you're using 8 steps for gradient accumulation.

I've extensively documented the problems I came across already in this issue, which may be linked to this problem as well. There wasn't any further in-depth investigation into what might have caused the issues though, so I believe the thread is not of much help.

This may also just be a temporary bug. I love Unsloth and give a lot of credit to them, but using their product does feel like you're constantly using a nightly/development build instead of a well-tested, production-ready release...

[mykeehu]

I was wondering the same thing: if I can train a 70B model on my own machine (according to their ad), then why does an 8B model take up so much VRAM? Because of the Hungarian language, I have to use high grad and rank/alpha values, but this causes me to constantly exceed the 24 GB VRAM limit, and my one-and-a-half-hour training session ends up taking an extra hour. And no matter whether I choose 4-bit loading or standard loading, it’s as if nothing changes; similarly, I didn’t see any VRAM changes with the rank/alpha adjustments, whether set to 2/8 or 1/16.

[mags0ft]

Hey @mykeehu, I've spent hours upon hours investigating this issue to figure out if the excessive VRAM usage is genuinely coming from the dataset format, which would've been weird. I made sure to pull the latest version of Unsloth from the Docker Hub beforehand. See this chart:

So the dataset format does NOT have anything to do with VRAM usage. Surprise, surprise.

Full data I collected

This was my process

• convert a very popular Alpaca-like dataset (openai/gsm8k) into a ChatML-like one (mags0ft/gsm8k-chatml), everything else stays exactly the same - mind that these two are single-turn datasets!
• pick 4 models of different sizes from the same family with the same, dense architecture to test against under the same conditions (2048 seq length, 4-bit QLoRA, r=16, alpha=16, batch size = 1, grad accumulation = 4, max steps = 30, no evals)
• do each training run for 4 different times and calculate the mean of all peak VRAM utilizations, using the unmodified Unsloth Studio web UI
• plot the values

Sooo...

1. The VRAM usage does not randomly fluctuate per run. Great sign!
2. VRAM usage is indeed heavily elevated for how big the models are (remember the tiny sequence length and 4-bit loading), but it does not seem to be linked to the dataset format
3. I was unable to find another cause - the datasets are, in their contents, exactly the same. Just the format differs.

As Unsloth is such a popular framework, this issue we're having is apparently INSANELY rare... nobody else seems to have this problem so far. We're probably missing something obvious.

It would be great to get assistance from the maintainers if that's possible, because it's not the first time I have stumbled across this issue, and I don't know what to do about the elevated VRAM usage anymore... I've never been able to figure this out. Any ideas, @danielhanchen and @shimmyshimmer? (I hope tagging you is ok!)

[mykeehu]

Meanwhile, I went in a different direction. I noticed that VRAM usage accumulates under Windows but isn’t cleared afterward, which causes a drop in throughput. I adjusted this setting so that VRAM usage was actually cleared every 50 steps under Windows, but the throughput remained just as slow, even though I had freed up the VRAM.
{"optim": "adamw_torch", "torch_empty_cache_steps": 50, "dataloader_num_workers": 0}

[mykeehu]

Another thing I noticed is that during log operations, it always loads something extra into VRAM, so if there’s a log every 5 steps, it somehow always increases VRAM usage.
I've now set the value of torch_empty_cache_steps to 10, which is quite frequent, but it can speed up training by freeing up space in VRAM.

[mags0ft]

It'd be great to get some more feedback from the community/someone at Unsloth about how wide-spread this issue actually is. It feels like it might have already gotten lost among the dozens of new issues on here.

[mykeehu]

It's annoying because what was supposed to be a one-and-a-half-hour session turned into a nearly four-hour training session on a Hermes 8B route because of the chatml...

[mykeehu]

Just to let you know, I'm currently training a Qwen3-4B-Instruct-2507 model with the exact same parameters (grad: 32, ctx 4096, 64/128, 3675 data), and it's already using 30 GB of VRAM - and it just keeps growing... Just like an 8B model. Just kidding.

[mags0ft]

Just kidding.

Wait, now I am confused about whether you're joking or not 😵‍💫

[mykeehu]

No, seriously, regardless of the model, my VRAM usage goes up to around 30 GB with the specs listed above, whether it's 4B or 8B.