GoPro Superview edges warping

[p-masek]

Is there an existing feature request for this?

• I have searched the existing requests

Description

Stabilization of GoPro Superview footwage works fine except I can see warping edges and footage does not look as smooth as with eg. wide FOV.I have tried other stabilization methods but result is more-or-less the same.
Phenomen quite visible on this footage (see sky and edges) : https://youtu.be/OIdSuL9H9Yo
Maybe it is something which cannot be improved as Superview requires quite heavy lens corrections - so just asking... Need to say that original footage was very shaky.

[temp-64GTX]

Confirming this thing with Superview. It also was in ReelSteady GO 1.0.23. But in version 2 they managed to fix it. Footage looks smooth, and sky are not shaking at all. So it's can be done.
I have a gopro 10. Is there any way that i can help?

[AdrianEddy]

The problem with Superview is that their algorithm is not documented anywhere and it's quite complicated, so best we can do is guess some approximation.

[temp-64GTX]

Ok. If this is not a strange non-linear lens distortion. (This is not?)
And if this is not some time syncronization (video-gyro) problems. Which i think isn't that.
Then maybe it is a some kind of non-linear scale of gyro-data. Because what else it could be? I have a lot footages with heavy shaking. And it isn't just a "little sky tremor on the edge of the frame". it's very noticeable shaking of almost whole frame. And the more gopro shakes when recording - the more it noticeable in final stabilized video. So i thought that a gyro-data interpretation is can be our problem. For example - in high amount of degrees (or radians, whatever) program does not rotate virtual camera enough (or rotate it more than enough).
I also attached the picture of my idea.

[AdrianEddy]

No, that's not it, it's definitely not accurate implementation of the non-linear distortion (yes, it's non-linear) - take a look here #43

[temp-64GTX]

Alright. Photoshop have a few lens profiles for GoPro. Maybe it can help somehow

GoPro Lens.zip

.

[p-masek]

I would not say problem would be in lens profile optically - I think correction is just fine. Problem is just during movement. Could it be rolling shutter (from some reason higher than in 4k 4:3 modes)?

[AdrianEddy]

@pmasek It could be rolling shutter, did you try to change the rolling shutter value and see if that changes anything? Can you send me the video btw?
@temp-64GTX Unfortunately there's no superview profile, so it doesn't help us. Superview is a special mode with additional processing in-camera

In any case, what we currently have implemented is not 100% perfect, so I'm still betting that this is the cause of this problem.

Pixel-wise, superview is the same as 4:3 mode, so generally I don't see a point of shooting superview if you want to stabilize in post. Just use 4:3. Gyroflow will output 16:9 from 4:3 by default and you'll have more-or-less the same result, but stabilized properly

[p-masek]

@AdrianEddy

1. Finally - I do not think it is rolling shutter. I have tried also 4k60fps Superview (so far I have used 5k60fps Superview and warping is the same (I think readout speed from sensor would be faster with lower resolution so less rolling shutter)
2. I use 5K60fps Superview as it is tiny sharper than 4:3 4K60fps (especially in shadows) so I like image little bit more (only downside is that warping)
3. 5K60fps Superview here - https://drive.google.com/file/d/1bqmlZyuA3D0yFvl-hU6ZX87bcXW8M9yV/view?usp=sharing
4. 4k60fps Superview here - https://drive.google.com/file/d/14VPI4-z9VxDqI0knyqIdzKzIgm0Xltea/view?usp=sharing

[p-masek]

And one more note:
5. If I use Standard in-camera stabilization for superview and put it into Gyroflow then:

• there is no warping when only optical corrections are used
• warping is added when additional Gyroflow stabilization is used - but in this case is warping much lesser...almost OK (as less GF
  stabilization is needed)

In-camera stabilized 5k60fps Superview video added here:
https://drive.google.com/file/d/1Bg-f8S9NtHeelCL3LHuROl1xf29WK7up/view?usp=sharing

[temp-64GTX]

@AdrianEddy
i will glad to try 4k 4:3, but i already have a lot of footages that shoot at 4k Superview.

[p-masek]

It looks like GF stabilization does not count that original Superview image is basically "squeezed" 4:3.
Also image is squeezed more in ~1/4 of upper and lower part of the image so it is unproportional and that is why there is more warping in these parts of the image.
So I guess it is that "unproportional squeeze" of the superview image which causes that...

[temp-64GTX]

ok, i've got some interesting results.
So.
I made some transformation matrix. It transforms "4k Superview" to "4:3 4000x3000".
And i have some Superview footage, that shakes as hell after stabilization. But after my transformation to 4:3 and stabilization - it's much more stable (but not as stable as reelsteady do). So, that's just a good start to future progress. I can provide you all my "research" data

.

[AdrianEddy]

and what is your transformation matrix? All the research data is attached in #43, there is html/js test page to calculate the pixel transform

[temp-64GTX]

Ok, let's take that GX013087.mp4 video from our good fellow p-masek.
If we will put it in GyroFlow, and do a standard stabilization, then we will have this: https://www.youtube.com/watch?v=hDb19HdkJxs
As we see - clouds are shaking/warping very noticeably.

And now we take source unstabilized video, and do my transformation, and then put it into GyroFlow as a "4:3 4000x3000" video. And take gyro-data from original video. Stabilized result will look that: https://www.youtube.com/watch?v=WoBkzN1BuVM
We can compare and see that sky is almost not shaking/warping anymore.

Now, what i did. First - i scaled original superview 3840x2160 frame to 4000x3000. From corners to corners:

Then i did my transformation matrix. I forced to do a lot of horizontal lines, because they are more complicated, then vertical. Goal is to make the superview look like 4:3. Almost pixel-to-pixel. In my case, accuracy is about 5 pixels. More or less. I did it manually, by mouse-dragging that 975 points. Well, i did only a necessary quarter, but that's not the point. Here is what i've got:

These lines - just guide-lines. Surface, that deforming by these guide lines is more dense:

As you can see on matrix image - vertical lines are stays vertical. There is no radial distortion. They just moving to left and right from center. It must be easy to implement that in parametric way.

But horizontal lines are much more complicated. From the center - they warping to top and bottom. But at the left and right sides - it warping much less. And look at the top line. From left corner - it goes down, then changing direction to up a little, to center, and again down, and again up to the right corner:

Notes that frames of image are not the same. At this picture we can see, that new frame is little bit bigger from left (and right) sides. Old original 4:3 frame is orange here. And new frame is cyan. Look at the corner (left top). it also above old frame, but when we moving to right - it goes under to the old frame:

Ok. So. I transforming original video:

And put them two into GyroFlow with this identical parameters of stabilization:

As for my transformed video, i got gyro-data from original video. And assigned to it a "Gopro 4000x3000" lens profile:

For some reason, it need to be synchronized with -10 ms:

And result is on video-links, which we see earlier it this text.
My video is also not ideal. Maybe it's because of interpolation of transformation matrix. If we look at the textel density - we can see that it is not quite smooth. It's a little bit wavy. And it not suppose to be. It must be smooth. My instruments are not very suitable for this case. It's 3ds max, actually. So, if you will implement this matrix in a parametric way - it will give more correct and accurate results.

I attached my matrix in vector and raster format here:
Rastro.zip
vector.zip

[p-masek]

@AdrianEddy Any chance that above changes suggested by @temp-64GTX could be somehow implememented into next releases?

[AdrianEddy]

no because it's not a solution, it's just more guessing. Also it's hard to translate that to an equation, and we don't support mesh warping the image

[temp-64GTX]

it's not guessing. It's ready-to-use solution. What can be easier. I deconstruct it to the atoms, even children will understand.
Without Superview support - program is unuseble. I guess i need to go and buy ReelsteadyGo. Ok. No problems.

[AdrianEddy]

It's ready-to-use solution.

How so? It's a mesh. How can I use it?

I guess i need to go and buy ReelsteadyGo. Ok. No problems.

If you need proper Superview support, then yes, you should.

[AdrianEddy]

/bounty $200

[algora-pbc]

💎 $200 bounty • Gyroflow

Steps to solve:

1. Start working: Comment /attempt #355 with your implementation plan
2. Submit work: Create a pull request including /claim #355 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Additional opportunities:

• 🔴 Livestream on Algora TV while solving this bounty & earn $200 upon merge! Comment /livestream once live

Thank you for contributing to gyroflow/gyroflow!

Add a bounty • Share on socials