Hold failed upload metrics and upload with next upload metrics

[hassaanfarooq01]

The task involves the upload_metrics function, which is executed on a separate thread. If the function exceeds the maximum number of retry attempts, the metrics_queue is intended to retain the metrics data. However, the main thread inadvertently removes this data. To address this, I have added a new variable, session.metrics_upload_failed_queue, to store the failed metrics. This allows for the failed metrics to be checked and updated before attempting to upload the metrics again. If the upload is successful, the metrics_upload_failed_queue is reset within the thread function.

However, there are some drawbacks or corner cases to consider. First, using session.metrics_upload_failed_queue to hold failed metrics can consume a significant amount of memory if failures continue to occur. Second, if the upload fails at the very end, the last set of metrics will not be pushed, I think which is an issue that cannot be avoided.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Improvements in handling failed metric uploads during model training sessions.

📊 Key Changes

• Added metrics_upload_failed_queue to store metrics when an upload fails during the training session.
• retry_request method in session.py clears metrics_upload_failed_queue upon a successful upload.
• If all retries fail, retry_request updates metrics_upload_failed_queue with the failed metrics.
• During on_fit_epoch_end in callbacks/hub.py, metrics from metrics_upload_failed_queue are re-added to the main metrics_queue for re-upload attempts.

🎯 Purpose & Impact

• Robustness: Ensures metrics are not lost due to intermittent network or service issues during a training session. 🛡️
• Data Integrity: Provides a way to retain important training progress data, improving result analysis and reproducibility. 📈
• User Experience: Streamlines the training process by quietly handling temporary upload issues, making the tool more reliable for users. 👩‍💻👨‍💻

[github-actions]

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I sign the CLA

---

1 out of 2 committers have signed the CLA.
✅ (glenn-jocher)[https://github.com/glenn-jocher]
❌ @hassaanfarooq01
hassaanfarooq01 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You can retrigger this bot by commenting recheck in this Pull Request}

[hassaanfarooq01]

I have read the CLA Document and I sign the CLA

[codecov]

Codecov Report

Attention: Patch coverage is 20.00000% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 75.74%. Comparing base (55ce6fa) to head (dff1b07).
Report is 2 commits behind head on main.

Files	Patch %	Lines
ultralytics/hub/session.py	20.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8513      +/-   ##
==========================================
- Coverage   75.76%   75.74%   -0.02%     
==========================================
  Files         117      117              
  Lines       14688    14693       +5     
==========================================
+ Hits        11128    11129       +1     
- Misses       3560     3564       +4     

Flag	Coverage Δ
Benchmarks	36.28% <0.00%> (-0.02%)	⬇️
GPU	39.01% <20.00%> (+<0.01%)	⬆️
Tests	70.83% <0.00%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[glenn-jocher]

@hassaanfarooq01 got it, changes look good, merging now. Make sure you follow these instructions though to ensure your account is recognized by the CLA bot:

hassaanfarooq01 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.