Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repo Claims To Be YOLOv5 #5920

Closed
danielbarry opened this issue Jun 10, 2020 · 93 comments
Closed

Repo Claims To Be YOLOv5 #5920

danielbarry opened this issue Jun 10, 2020 · 93 comments

Comments

@danielbarry
Copy link

danielbarry commented Jun 10, 2020

Hey there,

This repo is claiming to be YOLOv5: https://github.com/ultralytics/yolov5

They released a blog here: https://blog.roboflow.ai/yolov5-is-here/

It's being discussed on HN here: https://news.ycombinator.com/item?id=23478151

In all honesty this looks like some bullshit company stole the name, but it would be good to get some proper word on this @AlexeyAB

@AlexeyAB
Copy link
Owner

AlexeyAB commented Jun 10, 2020

Read more about YOLOv5:


Comparison YOLOv3 vs YOLOv4 vs YOLOv5: WongKinYiu/CrossStagePartialNetworks#32 (comment)

CSPDarknet53s-YOSPP gets 19.5% faster model inference speed and 1.3% higher AP than YOLOv5l.

YOLOv4 achieves 133 - 384 FPS with batch=4 using OpenCV and at least 2x more with batch=32:
OpenCV_Vs_TensorRT


123036148-3e43a180-d3f5-11eb-926d-bbc810f0ea6a


Data from:

123036798-4b14c500-d3f6-11eb-97ed-63d99414e410


84604438-abf6ec80-aec8-11ea-8341-f4563ea51dbc

@danielbarry
Copy link
Author

@josephofiowa I've updated my comment to reflect you're not the author - sorry. I am just trying to get to the bottom of these dubious claims.

@fat-tire
Copy link

I'm still confused cuz i thought YOLOv3 was the final one due to ethical concerns.

@danielbarry
Copy link
Author

I'm still confused cuz i thought YOLOv3 was the final one due to ethical concerns.

It's the last project by pjreddie, but not the last word on YOLO or Darknet.

@AlexeyAB
Copy link
Owner

I'm still confused cuz i thought YOLOv3 was the final one due to ethical concerns.

image


Tables 8-10: https://arxiv.org/pdf/2004.10934.pdf

(Real-time detectors with FPS 30 or higher are highlighted here. We compare the results with batch=1 without using tensorRT.)

comparison_gpus


https://medium.com/@alexeyab84/yolov4-the-most-accurate-real-time-neural-network-on-ms-coco-dataset-73adfd3602fe?source=friends_link&sk=6039748846bbcf1d960c3061542591d7

Therefore, we only show results with batch = 1 and without using TensorRT on comparison graphs.

@AlexeyAB
Copy link
Owner

@glenn-jocher did a lot for the development and improvements of Yolo and showed a lot of ideas, he created at least 2 very good repositories on Pytorch. Thus, he gave Yolo a long life outside of Darknet. All this hype around the Yolov5 was not raised by him.

@AlexeyAB
Copy link
Owner

Some notes on comparison: https://github.com/ultralytics/yolov5

  • The latency shouldn't be measured with batch=32. The latency must be measured with batch=1, because the higher batch - the higher latency. The latency is the time of a complete data processing cycle, it cannot be less than processing a whole batch, which can take up to 1 second depends on batch-size

  • If there is used batch=32 for both Yolov5 vs EfficientDet (I don't know), then this is ok, but only for Yolov5 vs EfficientDet, and only for FPS (not for latency), it can't be compared with any other results where is batch=1

  • Size of weights: yolov5x.pt - 366 MB, yolov5s.pt - 27 MB

@AlexeyAB
Copy link
Owner

Invalid comparison results in the roboflow.ai blog: https://blog.roboflow.ai/yolov5-is-here/

Actually if both networks YOLOv4s and ultralytics-YOLOv5l are trained and tested on the same framework with the same batch on a commond dataset Microsoft COCO: WongKinYiu/CrossStagePartialNetworks#32 (comment)

  • weights size: YOLOv4s 245 MB vs YOLOv5l 192 MB vs YOLOv5x 366 MB

  • test-dev accuracy on MSCOCO: YOLOv4s-608 45% AP vs YOLOv5l-736 44.2% AP (YOLOv4 is more accurate)

  • speed with batch=16: YOLOv4s-608 10.3ms vs YOLOv5l-736 13.5ms (YOLOv4 is faster)

  • roboflow.ai shared the Latency-Accuracy chart withultralytics-YOLOv5 which are measured with batch=32 and then divided by 32, while latency must be measured with batch=1, because the higher batch - the higher latency, latency of 1 sample can't be less than latency of the whole batch, so real latency of YOLOv5 can be up to ~1 second with high batch-size=32-64

  • they stated 140 FPS for YOLOv5 (s/m/l/x ???) (what batch-size ???) while YOLOv4 achieves ~400 FPS just with batch=4 by using OpenCV-dnn or TensorRT on GPU RTX 2080ti (table above)


Second, YOLOv5 is fast – blazingly fast. In a YOLOv5 Colab notebook, running a Tesla P100, we saw inference times up to 0.007 seconds per image, meaning 140 frames per second (FPS)! By contrast, YOLOv4 achieved 50 FPS after having been converted to the same Ultralytics PyTorch library.

  1. Actually YOLOv4 is faster and more accurate than YOLOv5l if it is tested with equal settings batch=16 on the same framework https://github.com/ultralytics/yolov5 on a common dataset Microsoft COCO (while YOLOv5x is much more slower than YOLOv4, and YOLOv5s is much less accurate than YOLOv4)

CSPDarknet53s-PASPP-Mish: 608x608 (~YOLOv4) is +0.8AP more accuracte and 1.3x times (+30%) faster than YOLOv5l 736x736

Full true comparsion: WongKinYiu/CrossStagePartialNetworks#32 (comment)

CSPDarknet53s-PASPP-Mish: (~YOLOv4)

cd53s-paspp-mish 45.0% AP @ 608x608
Model Summary: 212 layers, 6.43092e+07 parameters, 6.43092e+07 gradients
Speed: 8.7/1.6/10.3 ms inference/NMS/total per 608x608 image at batch-size 16

YOLOv5l:

yolov5l 44.2% AP @ 736x736
Model Summary: 231 layers, 6.17556e+07 parameters, 6.17556e+07 gradients
Speed: 11.3/2.2/13.5 ms inference/NMS/total per 736x736 image at batch-size 16

  1. They compared size of models of small ultralytics-YOLOv5-version YOLOv5s (27 MB) with very low accuracy 26-36% AP on Microsoft COCO with big YOLOv4 (245 MB) with very high accuracy 41-43% AP on Microsoft COCO

Fourth, YOLOv5 is small. Specifically, a weights file for YOLOv5 is 27 megabytes. Our weights file for YOLOv4 (with Darknet architecture) is 244 megabytes. YOLOv5 is nearly 90 percent smaller than YOLOv4. This means YOLOv5 can be deployed to embedded devices much more easily.


  1. They compared speed of very small and much less accurate version of ultralytics-YOLOv5 with very accurate and big YOLOv4. They did not provide the most critical details for comparison: what exactly YOLOv5 version was used s,l,x,... what training and testing resolutions were used, and what test batch was used for both YOLOv4 vs ultralytics-YOLOv5. They did not test it on the generally accepted Microsoft COCO dataset, with exactly the same settings, and they did not test it on the Microsoft COCO CodaLab-evaluation server, to reduce the likelihood of manipulation.

Third, YOLOv5 is accurate. In our tests on the blood cell count and detection (BCCD) dataset, we achieved roughly 0.895 mean average precision (mAP) after training for just 100 epochs. Admittedly, we saw comparable performance from EfficientDet and YOLOv4, but it is rare to see such across-the-board performance improvements without any loss in accuracy.

@AlexeyAB AlexeyAB reopened this Jun 11, 2020
@danielbarry
Copy link
Author

@AlexeyAB Thank you for breaking that down. I think my suspicion of the comparisons was warranted.

I just noticed that their iOS app page calls their network YOLOv4: https://apps.apple.com/app/id1452689527

YOLOv4 is an updated version of YOLOv3-SPP, trained on the COCO dataset in PyTorch and transferred to an Apple CoreML model via ONNX.

Someone said that they were apparently very surprised when you released YOLOv4 as they were planning to also release YOLOv4. I think this really puts emphasis on the need for people to communicate their intentions.

@AlexeyAB AlexeyAB pinned this issue Jun 11, 2020
@rcg12387
Copy link

I just noticed that their iOS app page calls their network YOLOv4: https://apps.apple.com/app/id1452689527

YOLOv4 is an updated version of YOLOv3-SPP, trained on the COCO dataset in PyTorch and transferred to an Apple CoreML model via ONNX.

Someone said that they were apparently very surprised when you released YOLOv4 as they were planning to also release YOLOv4. I think this really puts emphasis on the need for people to communicate their intentions.

Yeah. I see it's from Ultralytics LLC, who now becomes of the creator of YOLOv5. I agree your opinion. IMO Ultralytics has intended to succeed to YOLO by implementing PyTorch version with several contributions. Anyway it is the encouraging news for PyTorch community even it doesn't have a significant superior to YOLOv4 of @AlexeyAB.

@danielbarry
Copy link
Author

I think there is a strong case for either project to adjust their name to reflect the works are not built upon one another and are not a fair comparison.

As YOLO started in the Darknet framework, this repository was somewhat endorsed by pjreddie, @AlexeyAB was first to the punch with YOLOv4, Ultralytics already had their own "flavour" of YOLOv3 for TF - it would make sense to rename YOLOv5. Even something small like "uYOLOv5", or "YOuLOv5" could be significant in distinguishing the works.

Otherwise who publishes YOLOv6, and is YOLOv6 the improvement from YOLOv4 or YOLOv5? I think this is incredibly confusing and serves nobody.

@josephofiowa
Copy link

It's Joseph, author of that Roboflow blog post announcing Glenn Jocher's YOLOv5 implementation.

Our goal is to make models more accessible for anyone to use on their own datasets. Our evaluation on a sample task (BCCD) is meant to highlight tradeoffs and expose differences if one were to clone each repo and use them with little customization. Our post is not intended to be a replacement nor representative of a formal benchmark on COCO.

Sincere thanks to the community on your feedback and continued evaluation. We have published a comprehensive updated post on Glenn Jocher's decision to name the model YOLOv5 as well as exactly how to reproduce the results we reported.

@AlexeyAB called out very important notes above that we included in this followup post and updated in the original post. Cloning the YOLOv5 repository defaults to YOLOv5s, and the Darknet implementation defaults to "big YOLOv4." In our sample task, both these models appear to max out their mAP at 0.91 mAP. YOLOv5s is 27 MB; big YOLOv5l is 192 MB; big YOLOv4 is 245 MB. For inference speed, Glenn's YOLOv5 implementation defaults to batch inference and divides the batch time by the number of images in the batch, resulting the reported 140 FPS figure. YOLOv4 defaults to a batch size of 1. This is an unfair comparison. In the detailed update, we set both batch sizes to 1, where we see YOLOv4 achieves 30 FPS and YOLOv5 achieves 10 FPS.

Ultimately, we encourage trying each on one's own problem, and consider the tradeoffs based on your domain considerations (like ease of setup, complexity of task, model size, inference speed reqs). We published guides in the post to make that deliberately easy. And we will continue to listen on where the community lands on what exact name is best for Glenn Jocher's YOLOv5 implementation.

@AlexeyAB

This comment has been minimized.

@rowhanm rowhanm mentioned this issue Jun 12, 2020
@rcg12387
Copy link

@josephofiowa Thank you for your blog post Responding to the Controversy about YOLOv5: YOLOv4 Versus YOLOv5. But I am a little confused about that you wrote. First, in the last sentence of the section "Comparing YOLOv4 and YOLOv5s Model Storage Size" you wrote like this:

The largest YOLOv5 is YOLOv5l, and its weights are 192 MB.

Then what about YOLOv5x?

Second, in the fourth sentence of the section "Comparing YOLOV4 and YOLOv5s Inference Time" you wrote like this:

On single images (batch size of 1), YOLOv4 inferences in 33 ms (30 FPS) and YOLOv5s inferences in 20ms (10 FPS).

It should be 100ms or 50 FPS for YOLOv5s, I might say.

Thank you for your post.

@AlexeyAB
Copy link
Owner

@rcg12387 Why do you think they should know arithmetic? )

@josephofiowa
Copy link

@rcg12387
Thanks for the model sizes question. We've updated the post to show all sizes:

Updated to include model size of all YOLOv5 models. v5x: 367mb, v5l 192mb, v5m 84mb, v5s 27MB. YOLOv5s is the model compared in this article. YOLOv4-custom refers to the model we have been testing throughout this post.
model-sizes

Thanks for your callout of the arithmetic error. It's corrected as is the accompanying graph:

On single images (batch size of 1), YOLOv4 inferences in 33 ms (30 FPS) and YOLOv5s inferences in 20ms (50 FPS). (Update June 14 12:46 PM CDT - In response to rcg12387's GitHub comment, we have corrected an error where we previously calculated YOLOv5 inference to be 10 FPS. We regret this error.)

Note: Glenn Jocher provided inference time updates and pushed an update to his repo so that times are reported as end-to-end latencies. We have included his comments in the post and pasted them below:

The times ... are not for batched inference, they are for batch-size = 1 inference. This is the reason they are printed to the screen one at a time, because they are run in a for loop, with each image passed to the model by itself (tensor size 1x3x416x416). I know this because like many other things, we simply have not had time to modify detect.py properly for batched inference of images from a folder.
One disclaimer is that the above times are for inference only, not NMS. NMS will typically add 1-2ms per image to the times. So I would say 8-9ms is the proper batch-size 1 end-to-end latency in your experiment, while 7 ms is the proper batch-size 1 inference-only latency.
In response to this I've pushed a commit to improve detect.py time reporting. Times are now reported as full end-to-end latencies: FP32 pytorch inference + posprocessing + NMS. I tested out the new times on a 416x416 test image, and I see 8 ms now at batch-size 1 for full end-to-end latency of YOLOv5s.

@AlexeyAB
We have included your COCO benchmark performance in the post as well. Thank you for providing this.

@AlexeyAB
Copy link
Owner

@josephofiowa But you still don’t know what is the difference between Inference time and FPS )

@AlexeyAB
Copy link
Owner

The latest comparison: ultralytics/yolov5#6 (comment)

84604438-abf6ec80-aec8-11ea-8341-f4563ea51dbc

@rcg12387
Copy link

@josephofiowa Thank you for your reply. I have read your updated post.
However, the sentence still remains in the updated post:

The largest YOLOv5 is YOLOv5l, and its weights are 192 MB.

In order to avoid any confusion you should correct this sentence like this: The largest YOLOv5 is YOLOv5x, and its weights are 367 MB.
Thanks.

@josephofiowa
Copy link

@josephofiowa Thank you for your reply. I have read your updated post.
However, the sentence still remains in the updated post:

The largest YOLOv5 is YOLOv5l, and its weights are 192 MB.

In order to avoid any confusion you should correct this sentence like this: The largest YOLOv5 is YOLOv5x, and its weights are 367 MB.
Thanks.

Yes, done. Thanks.

@josephofiowa
Copy link

josephofiowa commented Jun 15, 2020

@AlexeyAB Thanks. Following performance updates on ultralytics/yolov5#6.

As it is clear Glenn is going to continue to create performance updates (even in the time since the post went live and now) and eventually publish a paper, we will reference that thread in the post for where to find the most up-to-date performance discussion on the COCO benchmark.

@pfeatherstone
Copy link

pfeatherstone commented Jun 15, 2020

Just to throw a spanner in the works: https://github.com/joe-siyuan-qiao/DetectoRS and https://arxiv.org/pdf/2006.02334.pdf. They claim 73.5 AP50. (I know it has nothing to do with yolo and naming continuity)

@AlexeyAB
Copy link
Owner

@pfeatherstone
DetectoRS is 15x - 60x times slower than Yolo: https://arxiv.org/pdf/2006.02334.pdf

  • 54.7 AP - slower than 1 FPS (test-time augmentaton)
  • 51.3 AP - 3.9 FPS

So this is offtopic.

@rcg12387
Copy link

rcg12387 commented Jun 15, 2020

@pfeatherstone Please don't make a hasty conclusion. A merit of YOLO versions is their lightness and speed. Practitioners don't welcome non-realistic latency even though a model has a high precision. It's useless.

@pfeatherstone
Copy link

@AlexeyAB I agree it's off topic. But this thread was comparing latency, FPS and accuracy. I thought i might include other non-yolo based models. Maybe that is more suited to a forum.

@pfeatherstone
Copy link

Maybe @glenn-jocher should have branded yolov5 differently to avoid controversy. At the end of the day, pick the one that suits your needs best, I.e performance requirements and your custom dataset.

@AlexeyAB
Copy link
Owner

@pfeatherstone
There is YOLOv4-tiny released: 40.2% AP50, 371 FPS (GTX 1080 Ti): #6067

@pfeatherstone
Copy link

Thanks for the update. It feels like there is competition in the YOLO market...

@Kreijstal
Copy link

I just found out about the controversy believing that YOLOv5 was an upgraded version of YOLOv4

@HardLaugh
Copy link

python pytorch is popular. its a trend to use pytorch to train darknet model. differnet training system always make me confusing, for example efficientdet in tensorflow | pytorch and darknet backward grad in yolo layer

@AlexeyAB
Copy link
Owner

AlexeyAB commented Jul 28, 2020

YOLOv4 training and inference on different frameworks / libraries:

Pytorch-implementations:

TensorFlow: https://github.com/hunglc007/tensorflow-yolov4-tflite

OpenCV (YOLOv4 built-in OpenCV): https://github.com/opencv/opencv

TensorRT: https://github.com/ceccocats/tkDNN

Tencent/NCNN: https://github.com/Tencent/ncnn

TVM https://tvm.ai/about

OpenDataCam: https://github.com/opendatacam/opendatacam#-hardware-pre-requisite

BMW-InnovationLab - Training with YOLOv4 has never been so easy (monitor it in many different ways like TensorBoard or a custom REST API and GUI):

@pfeatherstone
Copy link

@AlexeyAB why use darknet to train models rather than pytorch? You’re time must be split between research and maintaining/updating darknet. Not trying to be funny or make a point, just trying to understand the reasoning. Wouldn’t you be more productive if you could just focus on models rather than fixing bugs or creating new layers in darknet ?

@pfeatherstone
Copy link

By the way, using darknet is also a great solution as a minimal inference framework on CPU as it can have very minimal dependencies. So I can see reasons from a personal point of view.

@pfeatherstone
Copy link

This has arrived https://arxiv.org/pdf/2007.12099v2.pdf. Another flavour of yolo...

@AlexeyAB
Copy link
Owner

@pfeatherstone #6350

@pfeatherstone
Copy link

@AlexeyAB thanks. Soz for the duplication

@LEEGILJUN
Copy link

So! which model is better between Yolo v4 and Yolo v5 recommendation for the general user??? and Young students fight about which model is good, but can't draw conclusions

@kossolax
Copy link

kossolax commented May 6, 2021 via email

@LEEGILJUN
Copy link

perfs are similar as it's use same backbones. The only difference is one is using darknet, the other pytorch. Le jeu. 6 mai 2021 à 09:44, LEEGILJUN @.***> a écrit :

So! which model is better between Yolo v4 and Yolo v5 recommendation for the general user??? and Young students fight about which model is good, but can't draw conclusions — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

it is samely similar also Scaled yolov4?

@kossolax
Copy link

kossolax commented May 7, 2021 via email

@dewball345
Copy link

Frankly, the only reason why I used yolov5 over yolov4 is that its documentation(I'm talking about the PYTORCH IMPLEMENTATION; Darknet doesn't really fit my needs) is way easier to understand. The detailed steps for exporting to Onnx, TensorFlow, etc. are all there as well as CLEAR directions on how to use your own data. I tried using the PyTorch implementation of yolov4 and I struggled to actually start training, due to OpenCV and some random value errors that popped up. The directions are super vague and don't give much detail on which directory to place your data, where to keep labels, images, etc. Yolov5 at least has a nice tutorial blog teaching how to use your own data as well as a template colab(from roboflow) that actually works. I was able to export to Onnx and TensorFlow in like 5-10 minutes, while I couldn't even train the PyTorch implementation, because it was so hard figuring out where to put what.

So I think that's really the only inherent advantage that the v5 has that makes it worth using over v4. I totally agree that the naming for it is really misleading, but I just wanted to suggest that more documentation be added to this repo and the PyTorch one.

@dewball345
Copy link

Frankly, the only reason why I used yolov5 over yolov4 is that its documentation(I'm talking about the PYTORCH IMPLEMENTATION; Darknet doesn't really fit my needs) is way easier to understand. The detailed steps for exporting to Onnx, TensorFlow, etc. are all there as well as CLEAR directions on how to use your own data. I tried using the PyTorch implementation of yolov4 and I struggled to actually start training, due to OpenCV and some random value errors that popped up. The directions are super vague and don't give much detail on which directory to place your data, where to keep labels, images, etc. Yolov5 at least has a nice tutorial blog teaching how to use your own data as well as a template colab(from roboflow) that actually works. I was able to export to Onnx and TensorFlow in like 5-10 minutes, while I couldn't even train the PyTorch implementation, because it was so hard figuring out where to put what.

So I think that's really the only inherent advantage that the v5 has that makes it worth using over v4. I totally agree that the naming for it is really misleading, but I just wanted to suggest that more documentation be added to this repo and the PyTorch one.

Also would like to mention that while the performance for yolov4 is so much better than yolov5, I think the ease(or the actual ability) to develop is a reasonable trade off.

I think my comment may be a bit unrelated, but just wanted to mention the difference from a developer(who just wants something to work decently)'s point of view

@danielbarry
Copy link
Author

I'm going to close out this issue - it's mostly been 'resolved' in terms of understanding exactly what happened and there is not much benefit to continue piling in on the issue.

Feel free to open a new ticket if new issues arise.

@GiorgioSgl
Copy link

GiorgioSgl commented Jun 10, 2021

I test YOLOv4-tiny and YOLOv5s on raspberry PI4
I got from the first 0.4FPS and in the second 1.5FPS, I will come out with a medium soon. 🔥

@AlexeyAB
Copy link
Owner

AlexeyAB commented Jun 10, 2021

@AlexeyAB
Copy link
Owner

@GiorgioSgl
Read more about YOLOv5:

Comparison of YOLOv5 vs Scaled-YOLOv4 / YOLOR: #7717
E2hCrxSXoAARYRQ
E2l83oPX0AYmkQO

@GiorgioSgl
Copy link

I have tried on TFLite framework and I get better results on FPS, but the performance of the mAP@.5 decrease to a ~27.
Next week I will give a try on your suggested framework.

@Utsabab
Copy link

Utsabab commented Feb 7, 2023

Detection on thermal Infrared Images. Several versions of YOLOv4 and YOLOv5 compared. Details in Table 2.

https://www.sciencedirect.com/science/article/pii/S1569843222001145

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests