how long time it takes to train YOLOV3 on coco dataset

[SophieChang66]

hello, I want to known how long time it takes to train YOLOV3 on coco dataset , on which GPU device???

[jjehl]

really depends on the GPU, several hours to several days

[ahkarami]

for example, how long it takes on one GTX 1080 ti?

[ahsan856jalal]

If you have 80 classes then roughly train it for 2000*80=1,60,000 iterations and do the testing Regards Ahsan

…

On Mon, Apr 30, 2018 at 11:52 AM, ahkarami ***@***.***> wrote: for example, how long it takes on one GTX 1080 ti? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#742 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK9zjmNpDjmbRX7sMNW7fAQ-JfHSdRIZks5ttrS_gaJpZM4TqJSG> .

[jjehl]

3 days should be good.

[hengshan123]

Do you trained the yolo v3 on coco ? how long time it needs?

[zhangpzh]

Hello ! I found that the default max_batches (500,200) and the sub-divisions(16) are so large. Could you please share your customized yolov3 cfg file for MS-COCO training ? @jjehl @ahsan856jalal Thank you !

[jingtianyilong]

i've train a modified network base on yolov3-tiny. About 15 BFLOPs with 1070ti. 500000 iters takes about 3days. But strangely it takes even longer on 1080ti. I'm trying to figure it out

[mbalty]

Hi. I am training with coco_train2017 (118287 images), batch_size32. I am trying to train a class-unbiased object detector, by basically mapping all classes in coco to a class object. I am using a Titan Xp gpu and it took around 3.5 hours for the first epoch. Is that normal?
Also, my version of tensorflow-gpu is 1.5.0

[zhangpzh]

I totally use 2 weeks for the original coco cfg on four M40 gpus. batch_size=64, max_iterations=500k

[jingtianyilong]

I totally use 2 weeks for the original coco cfg on four M40 gpus. batch_size=64, max_iterations=500k

I was wondering if you got the similar result as the original weight file which is available on the website.

[mbalty]

Hi. I am training with coco_train2017 (118287 images), batch_size32. I am trying to train a class-unbiased object detector, by basically mapping all classes in coco to a class object. I am using a Titan Xp gpu and it took around 3.5 hours for the first epoch. Is that normal?
Also, my version of tensorflow-gpu is 1.5.0

I am now using the yolo-tiny model and it still takes around 3 hours on my first epoch. Am I wrong or is it a bit too much? From the logs, the gpu seems to be used but also my cpu is at 100%.

[zhangpzh]

Yes, finally after 50w+ iterations, I got 33.1 mAP. But it seems the learning rate should reduce faster to save more potential time. @jingtianyilong

[techkuji]

How much time it should take to train a yolov3 model with custom objects... And i have no gpu

[jjehl]

Really, without GPU, no way or it is questions of months... You have to understand that training on images needs a lot of calculation. The time depends on the power you have the size of the dataset and the number of classes.
For example, on coco with a multi gpu 4xGTX1080 if you want to train like J. redmond, so 500 000 iterations, it takes 10 days. But after 1 days the result is not so bad and the 9 days after are just for few percents better.
If you have a multi Titan X you can increase batch so reduce time.
For 1 class, 2000 images and a GTX1060 result can be good after 12 hours.

[techkuji]

Thank you for your reply and i want to know that in google colab i will get e GUP so how can i upload that folder('darknet') in to it??

…

On Wed 23 Jan, 2019, 2:34 PM Julien JEHL ***@***.*** wrote: Really, without GPU, no way or it is questions of months... You have to understand that training on images needs a lot of calculation. The time depends on the power you have the size of the dataset and the number of classes. For example, on coco with a multi gpu 4xGTX1080 if you want to train like J. redmond, so 500 000 iterations, it takes 10 days. But after 1 days the result is not so bad and the 9 days after are just for few percents better. If you have a multi Titan X you can increase batch so reduce time. For 1 class, 2000 images and a GTX1060 result can be good after 12 hours. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#742 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AfSp_YIO2LE8RanGDNn4FZzoOS99G2-hks5vGCWDgaJpZM4TqJSG> .

[abhishek1315]

@Arup276
Just clone it using this command:
!git clone https://github.com/pjreddie/darknet

[tloki]

I suppose that all of you are talking about training the whole network without pre-training or freezing (transfer learning)?

If I may ask - I tried running the PyTorch yolov3 implementation (https://github.com/westerndigitalcorporation/YOLOv3-in-PyTorch) and saw no validation-set approach in their training script? My practice till now was to take the model from epoch that performed the best on validation set (so for example after 100 epochs start validating after every epoch and dump the best models)...

I have an RTX2080Ti on disposal for a few (4) days and would like to think that I could achieve some performance training on 8 classes on Berkley dataset (70 000 train, 20 000 eval, 10 000 test).

[7thstorm]

I am running training on 4 classes on 26 million training images and approx 6 million test images.
Training on RTX 2080Ti. How do I calculate approx time to complete

[jenapss]

wow

[RezaSajadian]

running on an IBM p9 server with 4 Tesla V100 GPUs, and training on the entire coco2014 dataset, it took me well 5 days to finish all the 500200 max iterations of the yolov3 to finish.