Feature suggestions

[pfeatherstone]

Just putting some suggestions out there. Maybe we could organize these into projects.

• support scaled yolov4
• support yolov5 models
• add loss layer for training yolo models
• Create a fuse_conv_batchnorm visitor for enhanced performance during inference
• bipartite matching loss (Hungarian algorithm)
• transformer (for Detr like model)
• add GIoU, DIoU, CIoU options to loss_yolo
• see what happens when pre-training the (darkner53) backbone with Barlow twins loss using unlabelled imagenet, then fine tuning neck and heads with loss_yolo. Hopefully training the backbone with an unsupervised loss gives you better features than one trained specifically for classification. Presumably, with the backbone being frozen and therefore not requiring any gradient computation or batch-normalisation, this should accelerate training and reduce VRAM right ?
• Add cutmix augmentation
• Enhance CPU with NNPACK or oneDNN

[arrufat]

Yes, I want to do all those things at some point :)

I've already started working on YOLO scaled models, in particular YOLOv4x-mish, which can be found here.

My idea is to make a generic template class for yolo models, where the template type is the yolo model itself, then put each one in a separate unit, so that we can link to them (this greatly improves compilation time).

I've also started working on the to_label() part for yolo models, but it's not ready yet (lack of time these days). I will definitely push it unless someone wants to work on it. My dream would be to have the training part working, as well.

I am not sure how we can improve the performance. In my tests, dlib performs faster than pytorch for densenet, resnet and vovnet architectures and uses less memory (for small batch sizes, up to 4 or 8), but the tendency inverts for big batch sizes. If that is the case, dlib should be fast on single inference, so I am wondering if it's the post-processing (NMS, etc) that drags the peformance down... I want to test that at some point, as well.

[pfeatherstone]

When i was doing my tests, both onnx inference and dlib inference were doing NMS stuff. The NMS stuff is practically instantaneous (i haven't properly measured it though). I think it's something else that's causing bottlenecks. But your benchmarks are very interesting, and not what i was expecting based on my tests with yolov3. Properly profiling this at some point will be useful.

[arrufat]

Did you set CUDA_LAUNCH_BLOCKING to 1? I used that in all my benchmarks.

[pfeatherstone]

So when running all the yolo models with this repository, do you get similar performance to darknet and pytorch?

[pfeatherstone]

Did you set CUDA_LAUNCH_BLOCKING to 1? I used that in all my benchmarks.

No i've never used that. I imagine that would slow pytorch down.

[arrufat]

I added it because the creator of PyTorch suggested to, for proper benchmarking. arrufat/dlib-pytorch-benchmark#2

[pfeatherstone]

Oh ok. Fair enough. At the end of the day, running yolov3 with onnxruntime, pytorch or darknet yields roughly 65 FPS on 416x416 images. With dlib, i think i got around 45 FPS. If we can close that gap, that would be great.

[arrufat]

Oh ok. Fair enough. At the end of the day, running yolov3 with onnxruntime, pytorch or darknet yields roughly 65 FPS on 416x416 images. With dlib, i think i got around 45 FPS. If we can close that gap, that would be great.

I've just run yolov3 on darknet and dlib and I can confirm similar numbers, more precisely, on an NVIDIA GeForce GTX 1080 Ti:

	FPS	FPS	VRAM (MiB)	VRAM(MiB)
model (size)	darknet	dlib	darknet	dlib
yolov3 (416)	70	50	865	835
yolov4 (608)	32	22	1545	1742

I agree, it'd be cool if we could find out and fix the bottlenecks.

[pfeatherstone]

That's great thanks. Whenever i have time i will have a look. A decent profiler will go a long way. I always struggle to interpret sysprof. I'll try orbit again at some point. Though i had trouble building it last time i seem to remember.

[pfeatherstone]

It could be tensor.host() is called in a few places which introduce unnecessary barriers. I don't know enough about CUDA to be honest.

[arrufat]

As far as I know, tensor.host() is only called once in user code (to get the actual output of the network). I need to check if it's called somewhere else inside some layer implementation...

[pfeatherstone]

Yep building google's orbit profiler failed again. I'll have to do this at some point in my free time. Thanks @arrufat for investigating.

[davisking]

I'm not sure what causes this, but there shouldn't be any unnecessary tensor.host() calls. When the network is running it should all stay on the GPU. My guess is that darknet is making use of the fused conv+relu methods in cuDNN. dlib doesn't do that yet, it's still running those as 2 calls to cuDNN rather than one, which is a modest but non-trivial difference in speed if darknet is doing it like that.

…

On Mon, Jan 18, 2021 at 4:12 AM pfeatherstone ***@***.***> wrote: Yep building google's orbit profiler failed again. I'll have to do this at some point in my free time. Thanks @arrufat <https://github.com/arrufat> for investigating. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPYFRYIEWZIZISAT7UR74TS2P3RNANCNFSM4V2JGZ7A> .

[pfeatherstone]

@davisking Does dlib do fused convolution and batch normalisation ?

[pfeatherstone]

Darknet definitely does that. But then again, Pytorch doesn't and it achieves similar FPS to darknet, if not faster. I've seen onnxruntime achieve even faster FPS, but it does all sorts of crazy shit with graph optimization.