YOLOv5 Quantization

[gkoi00]

I am trying to load and quantize the YOLOv5 model, using the following code:

`device = torch.device('cpu')

model = torch.hub.load('ultralytics/yolov5', 'yolov5n', pretrained=True)

rand_in = torch.randn(1, 3, 640, 640)
quantizer = torch_quantizer(quant_mode, model, rand_in)
quantized_model = quantizer.quant_model
quantized_model = quantized_model.to(device)

quantized_model.eval()
results = quantized_model(rand_in)

if quant_mode == 'calib':
quantizer.export_quant_config()
if quant_mode == 'test':
quantizer.export_xmodel(deploy_check=False)`

The problem occurs when calling the model inference function, specifically i get the following error:

[wgshun]

The model should be loaded using the source code of the "ultratics/yolov5" project, and cannot be loaded using the "torch. hub. load" method; You also need to modify the 'forward' function in the yolo. py script:

def forward(self, x):
        for i in range(self.nl):
            x[i] = self.m[i](x[i])  # conv
        return x

[gkoi00]

Thank you very much!

[zijian98]

Hi, may I know how do you load the model using source code instead of using torch.hub?

Thank you.

[gkoi00]

Hello @zijian98!
They way i did it was to first clone the yolov5 repo and install requirements.txt like so:

git clone https://github.com/ultralytics/yolov5 
cd yolov5
pip install -r requirements.txt

Then import the DetectMultiBackend() function from yolov5/models/common.py
and use it to load either a model you trained yourself, or let it load a pretrained model.

from yolov5.models.common import DetectMultiBackend

model = DetectMultiBackend(weights='path/to/model')
#or
model = DetectMultiBackend(weights='yolov5s.pt')

The rest of the code I left he same.

[Zach227]

I followed the same method described in this thread to quantize yolov5 and it successfully converted to xmodel. However it gave one warning at the end:

WARNING: Logging before InitGoogleLogging() is written to STDERR
W20230626 11:18:07.572243  3639 tool_function.cpp:171] [UNILOG][WARNING] The operator named DetectMultiBackend__DetectMultiBackend_DetectionModel_model__Focus_model__Focus_0__Conv_conv__SiLU_act__input_5, type: aten::silu_, is not defined in XIR. 
XIR creates the definition of this operator automatically. You should specify the shape and the data_type of the output tensor of this operation by set_attr("shape", std::vector<int>) and set_attr("data_type", std::string)

nn.SiLU() is the default activation function for the Conv() class in the yolov5 source code. It is used many times in my model and it seems bad that this function is not defined in XIR. Is this an important warning to address? If so how would I fix it?

[gkoi00]

Yes, the SiLU function is not supported (as seen here https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/Currently-Supported-Operators). You need to replace it with LeakyReLU. The specific files that need to be modified are the common.py and experimental.py files, located in yolov5/models/, which are modified as follows.

#old
self.act = nn.SiLU
#new
self.act = nn.LeakyReLU(0.1, inplace=True)

After these changes you have to train this modified network.
I used Google Colab and it took approximately 4 hrs using this command:
!python train.py --img 640 --batch 16 --epochs 3 --data coco.yaml --weights yolov5n.pt
i got from here https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data

You have done it right if after you compile the model for your specific board, you get something like this with only 1 DPU subgraph:

[Zach227]

Thank you @gkoi00 @wgshun! That worked great. Would you also know the best way to interpret the output of the quantized model? After feeding an image through, I used the yolo non_max_suppression() function on the output. This should give a list for each detected object which contains the bounding box coordinates, a confidence score, and a classification. However, the results of this function didn't make sense which I assume is due to the fact that we changed the forward function in yolo.py.

Here is my quantization script which includes a custom dataset class and the non max suppression function: quantize.py

Any help in this area would be much appreciated.

[zijian98]

Hi @Zach227 , thanks for the reply. I have managed to solve this issue. However during deployment, I realized that it requires a .prototxt file which I do not know how to write it and using the prototxt file from similar model also resulted in segmentation fault. Any help would be appreciated. Thank You. From: Zachary ***@***.***> Sent: Monday, 3 July 2023 11:19 pm To: ***@***.***> Cc: Zi ***@***.***>; ***@***.***> Subject: Re: [Xilinx/Vitis-AI] YOLOv5 Quantization (Issue #1252) @zijian98<https://github.com/zijian98> I got this same error when I tried to train the model with the modified forward function in yolo.py. You need to use the original forward function for training, and then only use the shortened version for quantization. So use this for training: def forward(self, x): z = [] # inference output for i in range(self.nl): x[i] = self.m[i](x[i]) # conv bs, _, ny, nx = x[i].shape # x(bs,255,20,20) to x(bs,3,20,20,85) x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous() if not self.training: # inference if self.dynamic or self.grid[i].shape[2:4] != x[i].shape[2:4]: self.grid[i], self.anchor_grid[i] = self._make_grid(nx, ny, i) if isinstance(self, Segment): # (boxes + masks) xy, wh, conf, mask = x[i].split((2, 2, self.nc + 1, self.no - self.nc - 5), 4) xy = (xy.sigmoid() * 2 + self.grid[i]) * self.stride[i] # xy wh = (wh.sigmoid() * 2) ** 2 * self.anchor_grid[i] # wh y = torch.cat((xy, wh, conf.sigmoid(), mask), 4) else: # Detect (boxes only) xy, wh, conf = x[i].sigmoid().split((2, 2, self.nc + 1), 4) xy = (xy * 2 + self.grid[i]) * self.stride[i] # xy wh = (wh * 2) ** 2 * self.anchor_grid[i] # wh y = torch.cat((xy, wh, conf), 4) z.append(y.view(bs, self.na * nx * ny, self.no)) return x if self.training else (torch.cat(z, 1),) if self.export else (torch.cat(z, 1), x) And use this for quantizing: def forward(self, x): for i in range(self.nl): x[i] = self.m[i](x[i]) # conv return x — Reply to this email directly, view it on GitHub<#1252 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AUP6QL5KGYLRU2TGMVS2Q3LXOLPIVANCNFSM6AAAAAAYVOFJS4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[yadus97]

Hi may I know how you found out the application docker image for running yolov5?
or do you have any application code for running yolov5 on board?

[IkrameBeggar]

Yes, the SiLU function is not supported (as seen here https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/Currently-Supported-Operators). You need to replace it with LeakyReLU. The specific files that need to be modified are the common.py and experimental.py files, located in yolov5/models/, which are modified as follows.

#old
self.act = nn.SiLU
#new
self.act = nn.LeakyReLU(0.1, inplace=True)

After these changes you have to train this modified network. I used Google Colab and it took approximately 4 hrs using this command: !python train.py --img 640 --batch 16 --epochs 3 --data coco.yaml --weights yolov5n.pt i got from here https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data

You have done it right if after you compile the model for your specific board, you get something like this with only 1 DPU subgraph:

Hello I am having a huge drop of map@0.5. It dropped from 93% to 1% after the quantization. Do you have any recommendations

[DrLEO11153024]

I am currently trying yolov5 quantification and an error occurred. How can I correct it?

RuntimeError: Given groups=1, weight of size [32, 3, 6, 6], expected input[1, 1, 640, 640] to have 3 channels, but got 1 channels instead

[Muffinxz]

@Zach227 I tried to use the code you provided and I am getting this error "ImportError: cannot import name 'TryExcept' from 'utils' (unknown location)"

Can you tell me how I can resolve this issue please. Or Anyone, I am kinda stuck and I got a pretty short deadline coming ahead. Any help would be really appreciated

[60rw311]

@Muffinxz I'm having the same error, did you solve it?

[SoroushSemer]

@Muffinxz @60rw311 I had the same issue the way, I resolved it by moving the quantizer python script under the yolov5 directory.

This is because the common.py is trying to look for the utils directory but since it is inheriting the root directory of your quantizer script it cannot find it. (or at least i think)

[MuhammadBabarAli]

Hi, I took the quantization script from this tutorial: https://www.hackster.io/LogicTronix/yolov5-quantization-compilation-with-vitis-ai-3-0-for-kria-7b005d and made the necessary changes to the files. Can anyone please tell me how I can evaluate the quantized model, I used the yolov5 val.py but got errors when using the quantized .pt model and got mAP 0 when using the quantized .onnx model. Any help would be appreciated.