ultralytics 8.1.41 DDP resume untrained-checkpoint fix

[glenn-jocher]

May resolve #9419

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhanced the training process with improved fault tolerance and customization options. 🚀

📊 Key Changes

• Specified "backend" explicitly in distributed training setup.
• Added a condition to resume training only if self.resume is true, allowing more control over training resumption.
• Expanded argument updates for resuming training with reduced memory usage, adding "device" to "imgsz" and "batch".
• Simplified assertions and logging related to resuming training, making the process more straightforward and less error-prone.

🎯 Purpose & Impact

• Better Fault Tolerance: By enforcing a timeout in distributed training setups and allowing more flexible memory usage adjustments, the changes aim to reduce crash instances due to network issues or CUDA out-of-memory (OOM) errors. 🛡️
• Enhanced Customization: The addition of "device" to adjustable parameters when resuming training provides users more control over their training environment, potentially mitigating memory constraints. 🔧
• Streamlined Training Resume Process: These updates make resuming training more intuitive by removing unnecessary checks and clarifying logging messages. This leads to a smoother experience for users restarting training sessions. 🔄

In summary, these changes make the training process more robust, customizable, and user-friendly, particularly in distributed environments or when resuming after interruption.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Improved training resumption logic and package update checks in Ultralytics YOLO 🚀!

📊 Key Changes

• 🔼 Updated the version to 8.1.41.
• 🌐 Enhanced Distributed Data Parallel (DDP) setup by specifying the backend directly.
• 🔄 Improved the training resume feature to better handle model, device, and memory configurations.
• ⏭ Added robustness to auto-updating dependencies by allowing retries.
• 🛠 Fixed optimizer to properly handle non-float32 tensors in conversion for mixed precision training.

🎯 Purpose & Impact

• Version Bump: Keeps users on the latest, most stable release. 🆕
• DDP Setup: Ensures more reliable multi-GPU training by explicitly setting the backend, potentially improving performance and reducing bugs. ✔️
• Resuming Training: Makes it easier for users to pause and resume their training processes, even on different devices or with adjusted memory parameters, preventing loss of progress and improving user experience. 💾
• Dependency Updates: Increases the success rate of automatic dependency updates, reducing setup time and improving user experience. ⬆️
• Optimizer Tweak: Ensures more stable and accurate training in mixed precision, enhancing performance and reducing memory usage. 🚀

Together, these changes aim to enhance the usability, stability, and flexibility of Ultralytics YOLO, making advanced computer vision more accessible and efficient.

[codecov]

Codecov Report

Attention: Patch coverage is 87.50000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 76.63%. Comparing base (2cee889) to head (e37336f).

Files	Patch %	Lines
ultralytics/engine/trainer.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9453      +/-   ##
==========================================
- Coverage   76.66%   76.63%   -0.04%     
==========================================
  Files         120      120              
  Lines       15175    15174       -1     
==========================================
- Hits        11634    11628       -6     
- Misses       3541     3546       +5     

Flag	Coverage Δ
Benchmarks	36.30% <25.00%> (+<0.01%)	⬆️
GPU	38.23% <25.00%> (-0.11%)	⬇️
Tests	71.89% <87.50%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[glenn-jocher]

@Laughing-q I'm trying to solve a DDP training bug when starting from a partially completed checkpoint from a different dataset (with different class counts) in #9419 but I'm having a problem.

This PR fixes the bug report, but it breaks DDP resume. I can't figure how to do both.

• To test the PR train COCO8, crash training after a few epochs, and try to train VOC.yaml DDP from the partially trained COCO8 model.
• To test DDP resume, just train COCO8 DDP, crash then resume it.

Both should work correctly but I can't figure it out right now.

[glenn-jocher]

@Laughing-q this PR also resolves #9329, user asking to change device on resume.

[Laughing-q]

@glenn-jocher ok found the reason. Let me check what's the best solution here.

[Laughing-q]

@glenn-jocher DDP resuming broken is because self.args.resume is set to False when loading args from last checkpoint:

ultralytics/ultralytics/engine/trainer.py

Line 650 in ea52750

self.args = get_cfg(ckpt_args)

resume in the overrides we read from trainer when generating ddp script hence remains False.

ultralytics/ultralytics/utils/dist.py

Line 31 in ea52750

overrides = {vars(trainer.args)}

And this PR added an explicit condition to launch resume_training only if self.args.resume is explicitly set, then the DDP resuming broken.
Fixed in 97a2478

[glenn-jocher]

@Laughing-q I'm seeing a strange error when resuming. Resume works correctly, but then I get an error I've never seen before after a few successful resume epochs:

RuntimeError: Tensors of the same index must be on the same device and the same dtype except step tensors that can be CPU and float32 notwithstanding

To reproduce on the server:

yolo train device=6,7
# CTRL+C after 10 epochs
yolo train device=6,7 model=/usr/src/ultralytics/runs/detect/train/weights/last.pt resume

EDIT: Ah wait, this must be because of the recent FP16 optimizer checkpointing. I noticed that the step keys in the optimizer stay at FP16 while all the other tensors return to FP32 on resume. Maybe I need to explicitly cast the entire optimizer to FP32 on resume, or prevent the step keys from converting to FP16.

[glenn-jocher]

@Laughing-q ok I think I fixed this in c8523ea

The step keys in the optimizer will stay as FP32 and now resume works fully. The step keys are very lightweight so will not increase checkpoint size.

[glenn-jocher]

@Laughing-q merging this PR as 8.1.41 to get these fixes out while we figure out the LR-resume bug some more.