[BUG] setWarpTransformFourPoints() unexpected behaviour

[jakaskerl]

Describe the bug
setWarpTransformFourPoints()works as expected in some scenarios, but acts strange in some other scenarios. The problem is that it seems to work great with some point configuration, but moving one point of the polygon by a single pixel causes the warp to zoom-in too much.
I tried to figure out what causes the problem, but to no avail. I assumed it was something related to the data-types and how the warp was programmed.

Minimal Reproducible Example - not entirely minimal to ease debugging

# !/usr/bin/env python3

import depthai as dai
import cv2
import numpy as np

# Create pipeline
pipeline = dai.Pipeline()

"""Main nodes"""
# Camera
colorCameraNode = pipeline.create(dai.node.ColorCamera)
# Properties
colorCameraNode.setPreviewSize(600, 400)
colorCameraNode.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
colorCameraNode.setInterleaved(False)
colorCameraNode.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
# Img Transf
imgTransfNode = pipeline.create(dai.node.ImageManip)
# Properties
imgTransfNode.setMaxOutputFrameSize(2000 * 1500 * 3)

"""LinkIn/Out nodes"""
# Camera
colorCameraOutNode = pipeline.create(dai.node.XLinkOut)
colorCameraOutNode.setStreamName("colorCameraOutQueue")
# Config In
imgTransfConfigInNode = pipeline.create(dai.node.XLinkIn)
imgTransfConfigInNode.setStreamName("imgTransfConfigQueue")
# Tranf Img out
imgTransfOutNode = pipeline.create(dai.node.XLinkOut)
imgTransfOutNode.setStreamName("imgTransfOutQueue")

"""Connections"""
# Camera Node to Camera Out
colorCameraNode.preview.link(colorCameraOutNode.input)
# Camera Node to Img Transf Node
colorCameraNode.preview.link(imgTransfNode.inputImage)
# Config to Img Transf Node
imgTransfConfigInNode.out.link(imgTransfNode.inputConfig)
# Img Transf Node to Img Out
imgTransfNode.out.link(imgTransfOutNode.input)

# Connect to device
device = dai.Device(pipeline)

# Create queues
colorCameraOutQueue = device.getOutputQueue(
    name="colorCameraOutQueue",
    maxSize=4
)
imgTransfConfigQueue = device.getInputQueue(
    name="imgTransfConfigQueue",
    maxSize=4
)
imgTransfOutQueue = device.getOutputQueue(
    name="imgTransfOutQueue",
    maxSize=4
)

normalized = False
key = ord('b')

good =         [[304.0, 69.25],  [424.0, 64.5],   [434.5, 183.0],  [314.5, 187.75]] ## good
bad =          [[303.3, 68.8], [423.8, 64.3], [433.5, 182.8], [313.0, 187.3]] ## bad

while key != ord('q'):
    # Send an updated config key press
    cfg = dai.ImageManipConfig()
    
    point2fList = []

    if key == ord('g'):
        is_good = True
    elif key == ord('b'):
        is_good = False
    elif key == ord('i'):
        bad[-1][1] -= 1
    elif key == ord('k'):
        bad[-1][1] += 1
    elif key == ord('j'):
        bad[-1][0] -= 1
    elif key == ord('l'):
        bad[-1][0] += 1

    points = good if is_good else bad
        
    for p in points:
        pt = dai.Point2f()
        pt.x = p[0]
        pt.y = p[1]
        point2fList.append(pt)

    cfg.setWarpTransformFourPoints(point2fList, normalized,)
    imgTransfConfigQueue.send(cfg)

    for q in [colorCameraOutQueue, imgTransfOutQueue]:
        pkt = q.get()
        name = q.getName()
        frame = pkt.getCvFrame()
        if points:
            for p in points:
                cv2.circle(frame, (int(p[0]), int(p[1])), 2, (255, 0, 0), -1)

        cv2.imshow(name, frame)

    key = cv2.waitKey(1)

Expected behavior
setWarpTransformFourPoints()should warp and crop an image to fit a polygon defined by 4 input points. Each corner of the polygon defined by these 4 points should be clearly (though possibly warped) visible in the warped image.

Screenshots
Screenshot of incorrect image warp. Blue points are corners of the warp. They are not visible in the transformed image.

Correct image warp:

Moving the correct image warp bottom left corner (by one pixel to the right) results in a zoomed in image:

Moving the corner even more, makes it even worse (like in the 1st image).

Things I have tried

• Setting the pixel values to integers instead of floats - doesn't make a difference - looks like the values are rounded to int anyway
• Checking the area of the polygon - doesn't seem to have an impact
• Moving the points into irregular polygons / rectangles with high aspect ratio - no difference

Figured it could be related to dai.Point2f or some calculation error due to data type.

Thanks.

[deepernewbie]

Hi

rr = dai.RotatedRect()

setCropRotatedRect(rr)
does the same what I have observed is

if any of rr.size.width, rr.size.height is odd it zooms in too much so as a work around

if I assign to the closest event number while setting them up
rr.size.width, rr.size.height = (2*((box[2]-box[0])//2),2*((box[3]-box[1])//2))

it could be something related under the hood, give it a try for your case as well

I hope it helps

[asahtik]

Hi @deepernewbie ,
The problem with setWarpTransformFourPoints was that by default, the image manip configuration keeps the aspect ratio of the original image.
Can you try setting the keep aspect ratio option to false using cfg.setKeepAspectRatio(false) to see if that improves anything?