Losses are nan upon resuming training with resume=True [Yolov8]

[RaahimSiddiqi]

Search before asking

• I have searched the Ultralytics YOLO issues and found no similar bug report.

Ultralytics YOLO Component

Train

Bug

I am facing an issue where upon resuming an interrupted training (manual interruption after 51 completed epochs) the losses are all nan from the very beginning.

I had created epochs on every 5 checkpoints, so i'm loading last.pt (also tried epoch50.pt) but to no avail. The weird thing is that If I begin a new training from the last checkpoint without the "resume" parameter, it works just fine.

However, setting the device="cpu" resumes the training successfully with resume=True. This obviously is not a solution, as my dataset is too large to even consider using CPU.

It is worth noting i have many existing solutions, reducing batch size down to 2, changing amp to false, directly modifying the args.yml file, etc. Tried setting lr0 to 0.001. Also worth mentioning I have been checking many solutions online, but most people seem to have an issue with regular training, but my training only has this issue upon using "resume".

Please suggest some solutions.

Environment

Ultralytics 8.3.19 🚀 Python-3.10.15 torch-2.5.0+cu121 CUDA:0 (NVIDIA GeForce RTX 4090, 24564MiB)

Using Yolov8x.

All AMP checks are passed, and everything else is normal. I can run other new trainings without any issues.

Minimal Reproducible Example

Below is a simple example of what my code looks like. It is worth noting i have tried a dozen variations of this, reducing batch size down to 2, changing amp to false, directly modifying the args.yml file, etc. Also tried different checkpoints.

=========================

def train_yolov8(model_path, **kwargs):
    model = YOLO(model_path)
    model.train(**kwargs)

if __name__ == '__main__':
    epochs = 300
    train_yolov8(
        model_path=r"my/model/full/path",
        resume=True
    )

Additional

I am using the LVIS dataset.

Furthermore, I have faced this issue once a week or two ago as well. So it is not a one-time issue.

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[UltralyticsAssistant]

👋 Hello @RaahimSiddiqi, thank you for bringing this to our attention and for your interest in Ultralytics 🚀! This is an automated response to help guide you, and an Ultralytics engineer will assist you soon.

We recommend checking out our Docs for detailed guidance. You can find excellent examples for Python and CLI usage there that might help illuminate unexpected behavior or settings.

Since this is a 🐛 Bug Report involving nan losses upon resuming training, we ask you to please supply a minimum reproducible example. This will enable us to better investigate and debug the issue.

Meanwhile, please double-check and ensure your environment is up-to-date. Upgrade to the latest ultralytics package along with all requirements in a Python>=3.8 environment with PyTorch>=1.8:

pip install -U ultralytics

For more interactive support, consider joining our Discord 🎧 channel for real-time conversation or visiting our Discourse for in-depth discussions. The community on our Subreddit is also a great place to share knowledge.

Environments

YOLO can be executed in these up-to-date verified environments with dependencies preinstalled:

• Notebooks with free GPU:
• Google Cloud Deep Learning VM: Refer GCP Quickstart Guide
• Amazon Deep Learning AMI: Refer AWS Quickstart Guide
• Docker Image: See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are passing, confirming correct operation across various platforms and tasks.

We appreciate your patience and are here to support you! 🚀

[Y-T-G]

Can you provide the output after running this command in terminal: yolo checks?

And which model is this?

[RaahimSiddiqi]

Yolov8x.

Here is the output of yolo checks. I don't get why it is showing my OS as win 10. I have windows 11.

=======

Ultralytics 8.3.19 🚀 Python-3.10.15 torch-2.5.0+cu121 CUDA:0 (NVIDIA GeForce RTX 4090, 24564MiB)
Setup complete ✅ (28 CPUs, 31.8 GB RAM, 811.6/953.8 GB disk)

OS Windows-10-10.0.22631-SP0
Environment Windows
Python 3.10.15
Install pip
RAM 31.77 GB
Disk 811.6/953.8 GB
CPU Intel Core(TM) i7-14700K
CPU count 28
GPU NVIDIA GeForce RTX 4090, 24564MiB
GPU count 1
CUDA 12.1

numpy ✅ 1.26.4>=1.23.0
matplotlib ✅ 3.9.2>=3.3.0
opencv-python ✅ 4.10.0.84>=4.6.0
pillow ✅ 11.0.0>=7.1.2
pyyaml ✅ 6.0.1>=5.3.1
requests ✅ 2.32.3>=2.23.0
scipy ✅ 1.14.1>=1.4.1
torch ✅ 2.5.0+cu121>=1.8.0
torchvision ✅ 0.20.0+cu121>=0.9.0
tqdm ✅ 4.66.5>=4.64.0
psutil ✅ 6.1.0
py-cpuinfo ✅ 9.0.0
pandas ✅ 2.2.3>=1.1.4
seaborn ✅ 0.13.2>=0.11.0
ultralytics-thop ✅ 2.0.9>=2.0.0
torch ✅ 2.5.0+cu121!=2.4.0,>=1.8.0; sys_platform == "win32"

[Y-T-G]

Did you try downgrading PyTorch?

[RaahimSiddiqi]

I just tried torch 2.3.0 (along with the corresponding torchvision and torchaudio. Same issue. Did not change anything else. As a test, I set another training for 300 epochs (yolov8m) and cancelled it (after 60 epochs were completed), and tried to resume. Same issue. nan's almost immediately (after a second)

Ultralytics 8.3.19 🚀 Python-3.10.15 torch-2.3.0+cu121 CUDA:0 (NVIDIA GeForce RTX 4090, 24564MiB)

[RaahimSiddiqi]

Just tried upgrading to the latest ultralytics as well. Same issue. Here is my current yolo checks.

=================

Ultralytics 8.3.25 🚀 Python-3.10.15 torch-2.3.0+cu121 CUDA:0 (NVIDIA GeForce RTX 4090, 24564MiB)
Setup complete ✅ (28 CPUs, 31.8 GB RAM, 818.0/953.8 GB disk)

OS Windows-10-10.0.22631-SP0
Environment Windows
Python 3.10.15
Install pip
RAM 31.77 GB
Disk 818.0/953.8 GB
CPU Intel Core(TM) i7-14700K
CPU count 28
GPU NVIDIA GeForce RTX 4090, 24564MiB
GPU count 1
CUDA 12.1

numpy ✅ 1.26.4>=1.23.0
matplotlib ✅ 3.9.2>=3.3.0
opencv-python ✅ 4.10.0.84>=4.6.0
pillow ✅ 11.0.0>=7.1.2
pyyaml ✅ 6.0.1>=5.3.1
requests ✅ 2.32.3>=2.23.0
scipy ✅ 1.14.1>=1.4.1
torch ✅ 2.3.0+cu121>=1.8.0
torchvision ✅ 0.18.0+cu121>=0.9.0
tqdm ✅ 4.66.5>=4.64.0
psutil ✅ 6.1.0
py-cpuinfo ✅ 9.0.0
pandas ✅ 2.2.3>=1.1.4
seaborn ✅ 0.13.2>=0.11.0
ultralytics-thop ✅ 2.0.9>=2.0.0
numpy ✅ 1.26.4<2.0.0; sys_platform == "darwin"
torch ✅ 2.3.0+cu121!=2.4.0,>=1.8.0; sys_platform == "win32"

[Y-T-G]

What if you start training with amp=False and stop it and resume the same training?

[RaahimSiddiqi]

Seeing very strange behavior.
For recap: I had two trainings which failed to resume. A yolov8x and a yolov8m on the same dataset (COCO/LVIS).

After updating library version the yolov8m training is now resuming, but the val metrics (val=True) are all 0.
The yolov8x training is not resuming properly however, and loss is still NaN.

I have tried the amp experiment you mentioned, but i think it needs to reproduced and the same scale (high number of epochs, and on the full dataset). I tried two experiments, using a fraction of the original data (1%) and ran 20+ epochs. Both resumed successfully.

The yolov8m training:

[Y-T-G]

What's the original training command you were using (with all the hyperparameters)?

[RaahimSiddiqi]

Yolov8x training run, args.yml

===================
task: detect
mode: train
model: C:\Users\Raahim.Siddiqi\PycharmProjects\MLTraining\Projects\Yolo-Multiclass\training\runs\yolov8x-without-coco-merge\weights\last.pt
data: data.yaml
epochs: 500
time: null
patience: 20
batch: 16
imgsz: 640
save: true
save_period: 5
cache: false
device: '0'
workers: 8
project: runs
name: yolov8x-without-coco-merge
exist_ok: false
pretrained: true
optimizer: Adam
verbose: true
seed: 0
deterministic: true
single_cls: false
rect: false
cos_lr: false
close_mosaic: 50
resume: C:\Users\Raahim.Siddiqi\PycharmProjects\MLTraining\Projects\Yolo-Multiclass\training\runs\yolov8x-without-coco-merge\weights\last.pt
amp: true
fraction: 1.0
profile: false
freeze: null
multi_scale: false
overlap_mask: true
mask_ratio: 4
dropout: 0.0
val: false
split: val
save_json: false
save_hybrid: false
conf: null
iou: 0.7
max_det: 300
half: false
dnn: false
plots: true
source: null
vid_stride: 1
stream_buffer: false
visualize: false
augment: false
agnostic_nms: false
classes: null
retina_masks: false
embed: null
show: false
save_frames: false
save_txt: false
save_conf: false
save_crop: false
show_labels: true
show_conf: true
show_boxes: true
line_width: null
format: torchscript
keras: false
optimize: false
int8: false
dynamic: false
simplify: true
opset: null
workspace: 4
nms: false
lr0: 0.01
lrf: 0.01
momentum: 0.937
weight_decay: 0.0005
warmup_epochs: 3.0
warmup_momentum: 0.8
warmup_bias_lr: 0.1
box: 7.5
cls: 0.5
dfl: 1.5
pose: 12.0
kobj: 1.0
label_smoothing: 0.0
nbs: 64
hsv_h: 0.015
hsv_s: 0.7
hsv_v: 0.4
degrees: 0.0
translate: 0.1
scale: 0.5
shear: 0.0
perspective: 0.0
flipud: 0.0
fliplr: 0.5
bgr: 0.0
mosaic: 1.0
mixup: 0.0
copy_paste: 0.0
copy_paste_mode: flip
auto_augment: randaugment
erasing: 0.4
crop_fraction: 1.0
cfg: null
tracker: botsort.yaml
save_dir: runs\yolov8x-without-coco-merge

============

yolov8m training run, args.yml

===========

task: detect
mode: train
model: C:\Users\Raahim.Siddiqi\PycharmProjects\MLTraining\Projects\Yolo-Multiclass\training-coco\runs\yolov8m2\weights\last.pt
data: data.yaml
epochs: 300
time: null
patience: 20
batch: 16
imgsz: 640
save: true
save_period: 5
cache: false
device: '0'
workers: 8
project: runs
name: yolov8m2
exist_ok: false
pretrained: true
optimizer: Adam
verbose: true
seed: 0
deterministic: true
single_cls: false
rect: false
cos_lr: false
close_mosaic: 30
resume: C:\Users\Raahim.Siddiqi\PycharmProjects\MLTraining\Projects\Yolo-Multiclass\training-coco\runs\yolov8m2\weights\last.pt
amp: true
fraction: 1.0
profile: false
freeze: null
multi_scale: false
overlap_mask: true
mask_ratio: 4
dropout: 0.0
val: true
split: val
save_json: false
save_hybrid: false
conf: null
iou: 0.7
max_det: 300
half: false
dnn: false
plots: true
source: null
vid_stride: 1
stream_buffer: false
visualize: false
augment: false
agnostic_nms: false
classes: null
retina_masks: false
embed: null
show: false
save_frames: false
save_txt: false
save_conf: false
save_crop: false
show_labels: true
show_conf: true
show_boxes: true
line_width: null
format: torchscript
keras: false
optimize: false
int8: false
dynamic: false
simplify: true
opset: null
workspace: 4
nms: false
lr0: 0.01
lrf: 0.01
momentum: 0.937
weight_decay: 0.0005
warmup_epochs: 3.0
warmup_momentum: 0.8
warmup_bias_lr: 0.1
box: 7.5
cls: 0.5
dfl: 2
pose: 12.0
kobj: 1.0
label_smoothing: 0.0
nbs: 64
hsv_h: 0.015
hsv_s: 0.7
hsv_v: 0.4
degrees: 0.0
translate: 0.1
scale: 0.5
shear: 0.0
perspective: 0.0
flipud: 0.0
fliplr: 0.5
bgr: 0.0
mosaic: 1.0
mixup: 0.0
copy_paste: 0.0
copy_paste_mode: flip
auto_augment: randaugment
erasing: 0.4
crop_fraction: 1.0
cfg: null
tracker: botsort.yaml
save_dir: runs\yolov8m2

[Y-T-G]

I want to see the command. I can't understand from that which are the default and which are the ones you passed.

[RaahimSiddiqi]

from ultralytics import YOLO


def train_yolov8(model_path, **kwargs):
    model = YOLO(model_path)
    model.train(**kwargs)


if __name__ == '__main__':
    epochs = 500
    train_yolov8(
        model_path=r"C:\Users\Raahim.Siddiqi\PycharmProjects\MLTraining\Projects\Yolo-Multiclass\training\runs\yolov8x-without-coco-merge\weights\last.pt",
        data=r"data.yaml",
        name='yolov8x-without-coco-merge',
        epochs=epochs,
        batch=8,
        imgsz=640,
        patience=20,
        save_period=5,
        optimizer="Adam",
        project='runs',
        device='0',
        workers=8,
        plots=True,
        verbose=True,
        amp=True,
        val=False,
        dfl=1.5,
        box=7.5,
        cls=0.5,
        close_mosaic=int(max(epochs * 0.1, 10)),
        warmup_epochs=0,
    )

[glenn-jocher]

Thank you for sharing your training script. If you're experiencing issues with NaN losses when resuming training, consider checking your learning rate and data integrity. If the problem persists, please provide additional details for further assistance.