cv2.remap slows down the model inference

[bigsausage]

System Information

python version 3.9.6
cv2 version 4.8.0

Detailed description

i find that the inference time of yolo model would be slow down when i use cv2.remap to the image

Steps to reproduce

t0_total = 0
t1_total = 0
t2_total = 0
t0_point = 100
t1_point = 200
t2_point = 300
for i in range(t2_point+1):
    img = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)
    torch.cuda.synchronize()
    t = time.time()
    results = model.predict(source=img, show=False, imgsz=640, device=0)
    torch.cuda.synchronize()
    if i < t0_point:
        t0_total += time.time()-t
    elif i< t1_point:
        t1_total += time.time()-t
    else:
        t2_total += time.time()-t

print("avg t0 time",t0_total/t0_point)
print("avg t1 time",t1_total/(t1_point - t0_point -1))
print("avg t2 time",t2_total/(t2_point - t1_point -1))

the model is yolov8 model

when i run this code , i get the result as

avg t0 time 0.012496998310089111
avg t1 time 0.016826453835073144
avg t2 time 0.029507776703497374

but without the line img = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT) i got :

avg t0 time 0.013513808250427245
avg t1 time 0.013428420731515595
avg t2 time 0.013519821744976622

so i'm confused why the remap operation would affect the yolov8 inference?
i supose that cv2 operates on cpu and my yolov8 model inference on gpu, is it the data tranport thing or the multi-thread thing,
i'm so confused

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[dkurt]

Use cv.resize instead.

[bigsausage]

cv2.remap

i use cv2.remap to do distortion correction; my images are from camera, i'm afraid that cv2.resize can not sactisfied.

[dkurt]

What are shapes of img before and after remap? Are they the same?

[bigsausage]

What are shapes of img before and after remap? Are they the same?

yes, they're the same before and after remap, both (3200,1800) ,

[dkurt]

Does model perform data dependent computation? Like non-maximum suppression. In such case the content of the image affect performance. Without remap you probably get no detections while correct pipeline with remap requires extra compute.

[bigsausage]

Does model perform data dependent computation? Like non-maximum suppression. In such case the content of the image affect performance. Without remap you probably get no detections while correct pipeline with remap requires extra compute.

yes, yolo models perform data dependent computation.
but my code above use the same image each time
all of the 300 time use the same image so their results should be the same

[dkurt]

Please add more details about PyTorch version, CUDA version, attach model and add a minimal working reproducer.

[bigsausage]

sorry for the late reply ..
pytorch version is 2.0.1
cuda is 11.8
model is yolov8 pose model , i add it into the file
codes are as below :

from ultralytics import YOLO
import time
import cv2
import numpy as np
import torch

model = YOLO("pose_model.pt",task='pose')
img = cv2.imread("./test.jpg")
# this is for the warm up
results = model.predict(source=img, show=False, imgsz=640, device=0)

# camera param
K = np.array(
        [[1201.3967181633418, 0.0, 909.7424436183744], [0.0, 1203.635467250557, 534.1590658991514],
         [0.0, 0.0, 1.0]]) 
D = np.array(
    [[-0.0978537375125563], [-0.03841501213366177], [-0.03612764818273854], [0.05276041355808103]]) 
width, height = 1920, 1080
p = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(K, D, (width, height), None)
map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, D, None, p, (width, height), cv2.CV_32F)


t0_total = 0
t1_total = 0
t2_total = 0
t0_point = 100
t1_point = 200
t2_point = 300
for i in range(t2_point+1):
    img = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)
    torch.cuda.synchronize()
    t = time.time()
    results = model.predict(source=img, show=False, imgsz=640, device=0)
    torch.cuda.synchronize()
    if i < t0_point:
        t0_total += time.time()-t
    elif i< t1_point:
        t1_total += time.time()-t
    else:
        t2_total += time.time()-t

print("avg t0 time",t0_total/t0_point)
print("avg t1 time",t1_total/(t1_point - t0_point -1))
print("avg t2 time",t2_total/(t2_point - t1_point -1))

pose_model and test.jpg are in the zip file
111.zip

[dkurt]

but my code above use the same image each time
all of the 300 time use the same image so their results should be the same

Can you confirm this statement? Acoording to the code sample, image overrides every iteration:

img = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

Can you try the following?

inp = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)
results = model.predict(source=inp, show=False, imgsz=640, device=0)

[bigsausage]

but my code above use the same image each time
all of the 300 time use the same image so their results should be the same

Can you confirm this statement? Acoording to the code sample, image overrides every iteration:

img = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

Can you try the following?

inp = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)
results = model.predict(source=inp, show=False, imgsz=640, device=0)

i tried the code and still get the similar result~