Multi-GPU does not upload to Hub Web

[willuhmjs]

Search before asking

• I have searched the HUB issues and found no similar bug report.

HUB Component

Training

Bug

When training with multiple GPUs, the data is not uploaded to HUB. It stays at 0% until it logs as disconnected.

Environment

Linux on HPC, latest Python and Ultralytics

Minimal Reproducible Example

1. Run any training with multiple GPUs

Additional

No response

[github-actions]

👋 Hello @willuhmjs, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

[UltralyticsAssistant]

@willuhmjs hello! 👋

Thank you for bringing this issue to our attention. It seems you're encountering a problem with data not uploading to the Ultralytics HUB when training with multiple GPUs. This is indeed unusual, and we appreciate your detailed report.

A few things to consider that might help resolve this issue:

1. Check Network Connectivity: Ensure that your High-Performance Computing (HPC) environment has stable internet access throughout the training process. Network instability could cause the upload process to fail.

2. Verify HUB Version: Make sure you're using the latest version of the Ultralytics HUB. Outdated versions might have compatibility issues or bugs that have been addressed in newer releases.

3. Review HPC Restrictions: Some HPC environments have strict firewall rules or network policies that might restrict data uploads. It's worth checking with your system administrator if there are any such limitations in place.

4. Single GPU Test: As a diagnostic step, try running a smaller training job with a single GPU to see if the issue persists. This can help isolate whether the problem is specifically related to multi-GPU setups.

5. Logs and Error Messages: If possible, review any logs or error messages generated during the training process. These can provide valuable clues about what might be going wrong.

For further troubleshooting and guidance, please refer to our documentation at https://docs.ultralytics.com/hub. If the issue persists, feel free to provide additional details such as logs or error messages, and we'll be happy to assist further.

Thanks for your contribution to the Ultralytics community! 🚀

[darkkun0824]

I'm having the same problem, using 8 A100 for calculations

[darkkun0824]

I'm having the same problem, using 8 A100 for calculations

one gpu just work fine

[UltralyticsAssistant]

@darkkun0824 hello again! 👋

It's insightful to know that the issue is specifically occurring in a multi-GPU setup, especially with high-end GPUs like the A100s, but works fine with a single GPU. This narrows down the scope of the problem significantly.

Given this additional context, here are a few more tailored suggestions:

1. Distributed Training Configuration: Ensure that your distributed training setup is correctly configured. Multi-GPU training often requires specific settings to manage data synchronization and communication effectively across GPUs.

2. Resource Monitoring: Monitor system resources during training. High-end GPUs like the A100 can process data very quickly, potentially leading to bottlenecks elsewhere in the system that might affect data upload processes.

3. Batch Size Adjustment: Consider adjusting your batch size. While larger batch sizes are feasible with more powerful GPUs, they might introduce unexpected behavior in how data is processed and uploaded.

4. Update Drivers and Libraries: Ensure that your NVIDIA drivers and CUDA libraries are up to date. Compatibility issues between drivers, CUDA, and PyTorch can lead to unexpected behavior in multi-GPU setups.

5. Consult HPC Documentation: Given the specialized nature of HPC environments, it's also a good idea to consult any available documentation or support resources specific to your HPC setup. There might be known issues or configurations specific to multi-GPU training in these environments.

If these steps do not resolve the issue, we recommend gathering detailed logs from your training sessions and reaching out for further support through our documentation portal at https://docs.ultralytics.com/hub. Detailed logs can provide crucial insights for deeper troubleshooting.

Thank you for your patience and for contributing to the Ultralytics community! We're here to help. 🌟

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐