Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

evolve with mutil-gpu #6599

Closed
1 task done
alicera opened this issue Feb 10, 2022 · 12 comments
Closed
1 task done

evolve with mutil-gpu #6599

alicera opened this issue Feb 10, 2022 · 12 comments
Labels
question Further information is requested Stale

Comments

@alicera
Copy link

alicera commented Feb 10, 2022

Search before asking

Question

bash run.sh

#!/bin/bash
# Multi-GPU
for i in 0 1 2 3; do
  nohup python train.py --epochs 300 --data data/4cls.yaml --weights d11b_large-object-bright_20220207_best.pt --evolve --device $i > evolve_gpu_$i.log &
done

image
image

Additional

No response

@alicera alicera added the question Further information is requested label Feb 10, 2022
@glenn-jocher
Copy link
Member

@alicera what's the question?

@alicera
Copy link
Author

alicera commented Feb 11, 2022

Does it success to run for see the message?

@alicera
Copy link
Author

alicera commented Feb 14, 2022

Command

python train.py --epochs 300 --data coco128.yaml --weights yolov5s.pt --cache --evolve

image

every generation as runs/evolve/hyp_evolved.yam
When is the hyp_evolved.yam generated? Until 300 epoch ?

@glenn-jocher
Copy link
Member

@alicera hyp_evolved.yaml is updated every generation with the latest results.

@alicera
Copy link
Author

alicera commented Feb 15, 2022

I already run 70 epoch and the folder runs/evolve/exp/ only has the results.csv.
So my training have some problem?

@glenn-jocher
Copy link
Member

@alicera that's not correct. Evolution is very easy, you can start just by attaching the --evolve argument to the Colab notebook train cell. As you can see there will be 3 files in your runs/evolve/exp directory which are updated every generation:
https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb

Screenshot 2022-02-15 at 10 49 26

@alicera
Copy link
Author

alicera commented Feb 22, 2022

you can see there will be 3 files in your runs/evolve/exp directory which are updated every generation:
The result is used colab to generate?

If I use the command python train.py --epochs 300 --data coco128.yaml --weights yolov5s.pt --evolve on the linux terminal, it is also work ?
Because I use it on the terminal, it still no generate evolve.csv and hyp_evolve.yaml. But there is result.csv.

@glenn-jocher
Copy link
Member

@alicera yes example is shown in Colab. Evolution works in every environment, just like any normal YOLOv5 training.

@zaipinai
Copy link

@alicera yes example is shown in Colab. Evolution works in every environment, just like any normal YOLOv5 training.

I have a same problem with alicera. After training 150 epochs(I set it max to 150), it created hyp.yaml and hyp_evolved.yaml. It seems that the two are reversed, the hyp.yaml is more likely to be the result.
1.hyp.yaml 2.hyp_evolved.yaml 3.my original hyp
image
image
image

@glenn-jocher
Copy link
Member

@zaipinai good news 😃! Your original issue may now be fixed ✅ in a previous PR #6604. Current code no longer outputs hyp.yaml and applies an indexing fix that caused incorrect final results to display:

Screenshot 2022-03-10 at 15 43 24

To receive this update:

  • Gitgit pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
  • PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
  • Notebooks – View updated notebooks Open In Colab Open In Kaggle
  • Dockersudo docker pull ultralytics/yolov5:latest to update your image Docker Pulls

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

@zaipinai
Copy link

@zaipinai good news 😃! Your original issue may now be fixed ✅ in a previous PR #6604. Current code no longer outputs hyp.yaml and applies an indexing fix that caused incorrect final results to display:

Screenshot 2022-03-10 at 15 43 24

To receive this update:

  • Gitgit pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
  • PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
  • Notebooks – View updated notebooks Open In Colab Open In Kaggle
  • Dockersudo docker pull ultralytics/yolov5:latest to update your image Docker Pulls

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

Thanks for your help! I use modify YOLO v5 with transformer prediction head and it doesn't update. I will update it later. Sorry for bothering you for this mistake.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 10, 2022

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Access additional Ultralytics ⚡ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested Stale
Projects
None yet
Development

No branches or pull requests

3 participants