Add video rotation API

[stefanalund]

Ideally video rotation should be handled without expensive computations.

[superdump]

I think the camera sources need to do something sensible by default, have an option for signaling orientation separately but also have an option to manually set the orientation.

A sensible default behavior that was suggested was to set 'up' as the device orientation at the point where video capture starts.

[superdump]

I'm not sure if tracking the 'up' direction and always rotating the video accordingly is a good option or not. We probably need to test to see what is most intuitive.

[ikonst]

Does gstreamer have an element for this?

[stefanalund]

Yes, I believe this could be used http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-videoflip.html

[ikonst]

Isn't it better to rotate when displaying, i.e. GL transform?
Shuffling bytes around sound like a waste.

Also, what I meant to ask was - is there a GStreamer source element that gives orientation / position sensors?

[stefanalund]

For sure. But can we assume OpenGL rendering at all times?

[ikonst]

I would think that, in any case, rotation during drawing is better.

(I'm no video expert, and I can understand using videoflip for simplicity, but how can that ever be better than passing the desired rotation/flip as metadata along with the video frame?)

[superdump]

I think the plan for this should be to see if the camera API gives some way to request a specific video orientation from the camera. As long as that is effective and I feel like it's a possibility on mobile devices.

It is very easy for us to rotate and flip when rendering and it's also easy but ineffective to do it before encoding. We want to avoid anything ineffective if possible.

One more possibility is that after the work on texture memory sharing, maybe we can use GL elements for the transformations between the camera and encoder on iOS for basically free.

[ikonst]

See the part about physically rotating buffers here: https://developer.apple.com/library/ios/qa/qa1744/_index.html

I guess Apple are pretty experienced with this, and though they promise hardware-accelerated rotation (which our videoflip certainly isn't), they discourage you.

Does WebRTC say something about how to signal orientation without rotating buffers?

[superdump]

No, you have to do custom signalling, which @pererikb has demonstrated.

[stefanalund]

Any consensus on how we should solve this?

[superdump]

We may have to try a few things to see what works best. The problem, from my perspective, is that there is no standard way to handle signaling of video orientation in WebRTC. This means it is up to the application.

My default inclination is that we should do what the user expects. I have a proposal for what we should try, but as soon as something feels wrong, I hope we can figure out why and change it. Here is what I propose for the C-level OpenWebRTC behaviour:

• consider 'up' to be the physical 'up' of the device when opening the camera
• self-view should look like a mirror
• remote-view should look normal
• lock the orientation after deciding what is up

The downside of that is that you lose the cool thing that FaceTime does where you can switch between landscape and portrait and physical device up is always up. I don't really know why you would want to film upside down, but maybe you do.

Another mode of operation that may feel good is just that - physical up is always up and you signal that somehow, or rotate the video on the fly and hope the receiver can handle that.

Ideally I'd like to implement all of these and see which works best. :)

[ikonst]

I just want to ensure that, in the process, we don't physically rotate any
video buffers. Performance and energy efficiency is paramount.

On Wed, Mar 4, 2015 at 9:29 PM, Robert Swain notifications@github.com
wrote:

We may have to try a few things to see what works best. The problem, from
my perspective, is that there is no standard way to handle signaling of
video orientation in WebRTC. This means it is up to the application.

My default inclination is that we should do what the user expects. I have
a proposal for what we should try, but as soon as something feels wrong, I
hope we can figure out why and change it. Here is what I propose for the
C-level OpenWebRTC behaviour:

• consider 'up' to be the physical 'up' of the device when opening the
  camera
• self-view should look like a mirror
• remote-view should look normal
• lock the orientation after deciding what is up

The downside of that is that you lose the cool thing that FaceTime does
where you can switch between landscape and portrait and physical device up
is always up. I don't really know why you would want to film upside down,
but maybe you do.

Another mode of operation that may feel good is just that - physical up is
always up and you signal that somehow, or rotate the video on the fly and
hope the receiver can handle that.

Ideally I'd like to implement all of these and see which works best. :)

—
Reply to this email directly or view it on GitHub
#150 (comment)
.

[superdump]

But what if the video has completely the wrong orientation and both your self-view and the video the other side receives are flipped horizontally so that filmed text is backwards and they're rotated 180 degrees from what you expect? And you somehow can't rotate your device to fix it. 😄

I think we have to evaluate what feels best and what the costs are. Like any engineering problem really, find the trade off that produces the overall best result. Usability is just as important as performance. I will be disappointed if we cannot find something that is both very efficient and feels right.

I guess the situation we have now is that the camera always delivers a fixed orientation as described on the page you linked, on iOS. On desktop machines we can assume, I think, that the camera is correctly oriented. Android has been the most problematic I think.

I want to actually check the behaviors and test how they feel to figure out what makes sense and what is sensible performance-wise. My gut says we can probably do something on iOS that is very efficient and feels right. Android I'm less sure.

[ikonst]

What I mean to say is – let's please make sure we signal the orientation right up to displaying, where rotation is cheap, and avoid rotating actual buffers.

And, having said that, I'll stop giving armchair advice :)

[superdump]

Sure, as much as possible. Was there mirroring support as well as rotation? In any case, we can do all of that cheaply in GL. The real problem is whether any transformations are necessary and whether they can be efficiently done before encoding on iOS. First step on ios - do nothing and see if it feels right.

[superdump]

So, it seems I had missed something and there is a way to signal rotation/orientation in WebRTC:

Section 4: https://datatracker.ietf.org/doc/draft-ietf-rtcweb-video/

So, we need to know, at least on Android and iOS, which way is up and implement support for the CVO signaling and signaling of CVO in the SDP.

However, if no other WebRTC clients support CVO, we may have to fall back to either codec-specific rotation signaling or physical rotation before encoding.

[sdroege]

FWIW, we could do rotation before encoding on iOS/OSX (and after some more work on Android) in GL too as we get GL'ish things from the camera and can also pass GL data to the encoder.

Best is of course to just signal the orientation and let the clients doing the displaying do the work. There's probably a reason why even digital cameras usually don't rotate photos but just put the orientation as metadata into the file ;)

[ikonst]

FYI I've added the 'orientation' property to avfvideosrc:
https://bugzilla.gnome.org/show_bug.cgi?id=747378

Still I don't think it's a good idea since:
a) Apple says it comes with a cost, however miniscule
b) It's Apple-specific
c) It actually rotates the buffer and renegotiates caps, and I'm not sure how the pipeline will react to that

We should probably rotate in GL, either by building the missing gltransformation (requires Graphene library) or — probably better — implementing in glshader.

As for rotating like in https://github.com/EricssonResearch/openwebrtc-examples/blob/master/ios/NativeDemo/NativeDemo/NativeDemoViewController.m#L315, glimagesink takes it rather badly (bugs...).

[superdump]

@Rugvip has implemented #332 and #333 which are a good start towards this. We will ultimately also need GL rotation elements that actually adjust the dimensions of the output buffers. And then we need to adjust the default settings to do something sensible and integrate sensible behaviour into the example applications.

[superdump]

Merged, but we want GL elements to do this and then to optimise which elements are used and where the rotation happens.

[KerwinMa]

Sorry still posting the comment to this closed issue, I am not able to see the rotation issue is being fixed. Rotate the view seems working fine. BTW, anywhere that has instructions to follow how to compile local changed code with Cerbero? Thanks.

[superdump]

What rotation is not working for you? After the commits for this you can now set mirroring and rotation on OwrVideoRenderer and OwrVideoPayload. So to rotate before sending, you need to set the rotation and mirroring on the OwrVideoPayload.

[KerwinMa]

Thank you @superdump I will try your suggestion out locally.
BTW, any suggestions on how to use local changed code if I want to update openwebrtc code before commit to public?

[stefanalund]

We follow common practices on GitHub: Make a fork, modify the code in your fork and when you are done you can submit a Pull Request (if you want).

[KerwinMa]

Thank you @stefanalund kindly help. I mean to modify the code locally, what I should do to test the changes. Do I just need to run (for iOS):
./cerbero-uninstalled -c config/cross-ios-universal.cbc package -f openwebrtc
As I noticed it will fetch everything again:
[(1/32) libiconv -> fetch ]

[stefanalund]

Are you making changes to OpenWebRTC itself or only an app that is using it? For the first case @Rugvip can provide better guidance, in the latter case you don't need to rebuild anything.

[KerwinMa]

I mean the OpenWebRTC. I am wondering how to use cerbero to do it as this is the official suggestion to compile it. Seems there is no any guide saying how to compile local OpenWebRTC changes. It will be google we have some document for this as well. I notice there is a Makefile in openwebrtc folder. Maybe, i should start it from there. Thanks.

[IngJohEricsson]

Hi

By no means an expert in this area but.. This is what works for me. Rob helped me with this to make it easier to debug yet another feature that is in the pipe (SCReAM congestion control for video)

In the cerbero-uninstalled shell
cd to the build directory. In the linux case it is something like linux-x86
cd further to eg openwebrtc-0.3
Do:
make && make install
Run the openwebrtc-daemon
Alot if the above is from memory and I am likely a bit off on the folder names but hope that it helpes a little atleast. I am at the office again on Monday and can then double check things

Ingemar

Sent from Moxier Mail
(http://www.moxier.com)

----- Ursprungligt meddelande -----
Från: KerwinMa notifications@github.com
Till: EricssonResearch/openwebrtc openwebrtc@noreply.github.com
Skickat: 2015-05-16 17:08
Ämne: Re: [openwebrtc] Add video rotation API (#150)

I mean the OpenWebRTC. I am wondering how to use cerbero to do it as this is the official suggestion to compile it. Seems there is no any guide saying how to compile local OpenWebRTC changes. It will be google we have some document for this as well. I notice there is a Makefile in openwebrtc folder. Maybe, i should start it from there. Thanks.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/150#issuecomment-102635664.

[KerwinMa]

Thank you @IngJohEricsson. I will have a try.