Clipping incorrect when rotating an omx render component while having its dest outside of the visible screen

[dividuum]

Let me try to describe what happens. I took a FullHD video and am rendering it to a FullHD output. I manually place the render component of the omx pipeline to be outside of the visible screen. This can be done with omxplayer like this:

omxplayer test.mp4 --win '500 0 2420 1080' --orientation 0

The result is the following and looks correct:

But if I rotate the render component by 90, 180 or 270 degree the following happens. The example image is create when running omxplayer like this:

omxplayer test.mp4 --win '500 0 2420 1080' --orientation 180

It seems that when using any rotation other than 0, the video is somehow clipped into the visible area. In the above image I would expect the output to look like the red area in this manually put together screenshot:

The same thing happens then the video is vertically outside of the screen. In that case 0 and 90 degree rotation works while 180 and 270 don't. Each screenshot is generated while running

omxplayer test.mp4 --win '0 500 1920 1580' --orientation $ORIENTATION

Expected result when rotated by 0 degree:

Expected result when rotated by 90 degree:

Weird result when rotated by 180 degree:

Weird result when rotated by 270 degree:

I'm running

Jul  2 2017 13:02:31 
Copyright (c) 2012 Broadcom
version 2b697c01b84a5ca2f33e4a1be47650cc59faa076 (tainted) (release)

Linux raspberrypi 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux

Can this be fixed?

[popcornmix]

Can you confirm. With omxplayer test.mp4 --win '500 0 2420 1080' --orientation 180
you would like video to be rendered to pixel 0,0, outside the given destination rectangle?

[dividuum]

For omxplayer test.mp4 --win '500 0 2420 1080' --orientation 180, the video pixel 0, 0 (top left of the video) should be rendered to the invisible (since outside of the screen) coordinates 2420, 1080. Right now it seems to be rendered to 1920, 1080 instead. Similarly the video pixel 1920, 1080 (bottom left pixel) should be at 500, 0. Right not it's clipped and invisible.

The summary would probably be: If I render a video partially offscreen, the output should make sense regardless of the rotation setting. Right now 0 behaves like I would except, while other rotations don't.

[popcornmix]

I'm talking about destination pixels.
Your picture with the red rectangle drawn on looks like you are expecting the destination pixel 0,0 to be rendered to. Am I understanding that correctly?

[dividuum]

I'm not sure I understand, but the image with the red border is what I'm expecting when rendering a 180 degree rotated video outside of the screen. Instead I get the result in the screenshot before that.

I would expect when I set a destination rect to some coordinates, that the video is fully rendered into that area. Regardless of whether that rect is outside the screen or the video is rotated. The screenshot with the red area shows how it's supposed to look like. This works perfectly when the video isn't rotated.

[dividuum]

Is it clear what I'm expecting vs what actually happens? If not, I might try to create a GIF that shows the problem.

[dividuum]

Here's a GIF showing the problem in an animated form: I'm playing a video and adjusting it's target with OMX_DISPLAY_SET_DEST_RECT and the OMX_IndexConfigDisplayRegion call. I'm also setting the rotation of the video with the OMX_IndexConfigCommonRotate call to all 4 rotations (0, 90, 180, 270). You can see how the video behaves once it crosses the screen boundary. The output using rotation 0 makes sense. All other settings seem to clamp some of the output to the screen coordinates.

(Less choppy youtube version)

Does that help?

[popcornmix]

I think I understand. We have a clamp function in dispmanx that checks for out-of-range dest rectangle coordinates and clamps them to screen edges (adjusting source coordinates to compensate).

I believe that function does the right thing for orientation=0 but not for other orientations.
Let me see if I can handle the other cases (it is tricky code so might take some effort to get right).

[dividuum]

Thanks a lot! I'm happy to test any future firmware that fixes this. Oh. And if you need any easy way to test all this, I can put up info-beamer code that runs the above test. Would that be helpful?

[dividuum]

How to reproduce the above GIF output:

Put the following code in node.lua in some directory. Put any video named test.mp4 next to it.

-- save as node.lua
gl.setup(NATIVE_WIDTH, NATIVE_HEIGHT)

local vid = resource.load_video{
    file = "test.mp4",
    raw = true,
}

-- local font = resource.load_font "font.ttf"

vid:layer(-1)

local s = 2
local r = {0, 90, 180, 270}

function node.render()
    local rotation = r[math.floor(sys.now()/4) % 4 + 1]
    vid:rotate(rotation)
    gl.clear(0,0,0,0)
    -- font:write(0, 0, "Rotation " .. tostring(rotation), 100, 1,1,1,1)
    local x = math.sin(sys.now() * 5) * 500 + WIDTH/2
    local y = math.cos(sys.now() * 5) * 300 + HEIGHT/2
    vid:target(x - 1920/s, y-1080/s, x+1920/s, y+1080/s)
end

Then run this code with info-beamer like this:

$ info-beamer /path/the/the/directory

If you want the overlay that informs you about the current rotation, put any truetype font as font.ttf into the directory and uncomment the two font related lines in the above snippet. Let me know if you run into problems setting all this up.

[dividuum]

Thinking a day over what you said:

[...] checks for out-of-range dest rectangle coordinates and clamps them to screen edges (adjusting source coordinates to compensate).

For some reason I didn't expect it to work like this internally. But that makes perfect sense. So I guess it's also possible to fix this completely in "user space" by adjusting source/dest/rotation accordingly. I'll look into that. That might also have the added benefit of making it easier to check if the transformed video is completely off-screen and avoid those greenish artifacts that sometimes result from that.

[dividuum]

It works. With the help of a transformation matrix I now project the video into screen space, clamp there and transform back into video space. Then I set the dest/source/rotation values. Works pretty well. I guess I no longer need a firmware based fix for this. Feel free to close depending on what you think the firmware should do.

[JamesH65]

Closing this issue as questions answered/issue resolved.