Remote video not showing when video-orientation is set in SDP message

[Dryra]

Hello,
I don't really know if this is an issue from the MixedReality-WebRTC itself or something else but it never hurts to ask.

So the Hololens app is communicating with an Android app with WEBRTC, Video and Audio, everything works fine except for something that I noticed recently within the Unity VideoRemote: there was no image coming from the remote peer but I can hear audio, after some investigations, I noticed that if I remove the line a=extmap:4 urn:3gpp:video-orientation\r\n from the sdp message on the Hololens Peer, the remote video works, but the scaling is wrong (kind of distorted)., and also when I change the Android apps orientation, the Remote video also works.

I'm using the release version.

If you need any other information, please let me know, any help is really appreciated.

[zhuangce]

@Dryra Hi ，i'm very initerested in your way of communicate hololens with android app ,i'm a web-development noob ,and i spend lots of time try to implement but failed ,can you tell me how to make it work , thanks

[Dryra]

Hello @zhuangce, it is not an easy task to implement, you have to do a lot of research as I can't really tell you how it exactly works, here is a repo that can help you start up and maybe understand more, https://github.com/vivek1794/webrtc-android-codelab. Good Luck!

[zhuangce]

@Dryra oh ,thank you for you reply ,i'll try it

[djee-ms]

@Dryra sorry I have no idea about this. It sounds like you're using an SDP extension 3gpp:video-orientation, and I am not familiar with it. I can see that it is used by default in the Google implementation, so there is support for it at least. But we currently do not support any phone platform yet (see #28), so we haven't had a chance to test things much in that domain so far. Hopefully that will change in the near future, but at the moment I can't provide much help.

One thing you could try maybe would be to recompile everything from sources so you get the PDBs for the Google code and can access the debug logs of libwebrtc (they are printed with OutputDebugString() to the Visual Studio Output window), which hopefully prints an error message that could tell us what the problem is.

When you say "when I change the Android apps orientation the Remote video also works", is it with or without the 3gpp:video-orientation part? It is possible our video processing/rendering doesn't handle well portrait mode videos and/or dynamically changing orientation; to be fair I don't recall ever trying that.

[Dryra]

@djee-ms thank your for your reply and your suggestion, but I unfortunately cannot invest much time on debugging this issue, so I will go with easy workarounds.

When you say "when I change the Android apps orientation the Remote video also works", is it with or without the 3gpp:video-orientation part? It is possible our video processing/rendering doesn't handle well portrait mode videos and/or dynamically changing orientation; to be fair I don't recall ever trying that.

it is with the 3gpp:video-orientation part, I have to Android app and the Unity app in front of me , at first when both peers connect, the unity app does not display any remote stream, and after checking, the I420RemoteVideoFrameReady callback in RemoteVideoSource.cs does not get fired, but when I change the orientation of the Android app from portrait to landscape at that moment, the callback gets fired and then I can see the remote Video.

I guess I call myself lucky that it even works cross device (Android > Hololens) !

I will update this thread if I find any information that could be useful in future developments.
Thank you for your help! Best regards.

[djee-ms]

Ok thanks. Sounds like it might be a bug if the remote callback doesn't start immediately but gets started after changing the video orientation. Though without a repro it's hard to tell. Feel free to reopen if you have more time to investigate, and we can have a look again.

[Dryra]

I will definitely investigate more when I have the time and will let you know. In the meanwhile I have some other issues, but will open them separately.

[qazqaz12378]

Hello,
I had the same problem.
the I420RemoteVideoFrameReady callback in RemoteVideoSource.cs does not get fired.
Is there a solution now?

[Dryra]

Hello @qazqaz12378 ,
I found a workaround which is not perfect, as it stretches out the remote video in the hololens a little bit, just for the time being: check for the string "a=extmap:4 urn:3gpp:video-orientation\r\n" in the SDP answer or offer that the Hololens is sending, and remove it before sending.

[zhuangce]

@Dryra hello ,i'm kind of successes to build the communication between hololen and android using your suggestion , but the android app will crash after receive offer from hololen ,if i send offer again ,the app will work correctly ,i'm wondering if you have the problem?

[Dryra]

@zhuangce: No I currently do not have the crashing problem related to sending or receiving offers. You would have to check the Error Stack in the Android app and see what's causing the crash exactly, maybe it's a timing problem when the android is receiving the offer for the first time.

[jameszhong2008]

Remove “a=extmap:4 urn:3gpp:video-orientation\r\n” does work for me when make a video call, but join a conference meeting with multiple peers, maybe it is not enough.
I found that VideoSinkWants also influence the result.
in peer_connecton.cpp PeerConnection::OnAddTrack metheod

rtc::VideoSinkWants sink_settings{};
sink_settings.rotation_applied =
true; // no exposed API for caller to handle rotation

device rotation state
portrait , ARGBRemoteVideoFrameReady was not called.
landscape, ARGBRemoteVideoFrameReady works.

after set sink_settings.rotation_applied = false
both orientation of Android device work, ARGBRemoteVideoFrameReady works
but image rotation is wrong in portrait position.
@djee-ms

[jameszhong2008]

Hello @Dryra @djee-ms
Just figured out it, we need to rotate image by our self.
two steps:

1. peer_connecton.cpp PeerConnection::OnAddTrack metheod
   change rotation_applied = false
   rtc::VideoSinkWants sink_settings{};
   sink_settings.rotation_applied = false;

2. video_frame_observer.cpp VideoFrameObserver::OnFrame metheod

rtc::scoped_refptrwebrtc::VideoFrameBuffer buffer(
frame.video_frame_buffer());
// rotate frame if needed
if (frame.rotation() != webrtc::kVideoRotation_0 ||
buffer->type() == webrtc::VideoFrameBuffer::Type::kI420)
{
auto frameTemp = webrtc::VideoFrame(
webrtc::I420Buffer::Rotate(*buffer->GetI420(), frame.rotation()),
webrtc::kVideoRotation_0, frame.timestamp_us());

OnFrameCallback(frameTemp);

} else {
OnFrameCallback(frame);
}

// do the callback things
void VideoFrameObserver::OnFrameCallback(
const webrtc::VideoFrame& frame) noexcept {

....
i420_callback_(...)
argb_callback_(...)

}

maybe we should add rotate angle arg in i420_callback_ and argb_callback_ in the future.

[djee-ms]

@Dryra is this still an issue for you? Can you please try on master, because I remember fixing some issue with ARGB scratch buffers and invalid striding when frame rotates (see #435) which may have been causing this bug (not sure).

[djee-ms]

(using the 2.0.0-preview.1 prebuilt packages is also OK, they contain that fix for #435)

[Dryra]

@djee-ms I unfortunately can't get my hands on a HL2 at the moment, but I will test this as soon as I do. Thank you!

[madlymad]

I am experiencing a quite similar issue when trying to make a WebRTC video call between Windows 10 Client with MixedReality-WebRTC 2.0.0 and Android mobile client which uses prebuild org.webrtc:google-webrtc:1.0.32006.

A video call can established between the two clients but the remote video appears in the MixedReality-WebRTC client only when the Android client is in landscape. Android client does not have any problem showing remote or local video.

I've done a couple of tries in the Android client implementation regarding the capturing stream but none of them had any effect in the MixedReality-WebRTC client.

• In terms of the Android client I changed the capture to a square video ie. 720x720 the problem was the same as only when Android device change orientation to Landscape the windows client (MixedReality-WebRTC) can show its video.
• Tried to use capturing width-height parameters both ways, but with same results

videoCapturer?.startCapture(1280, 720, 30)
videoCapturer?.startCapture(720, 1280, 30)

• Remove orientation attribute "a=extmap:13 urn:3gpp:video-orientation" completely from SDP before sending it to the windows client as suggested above but this resulted in a crash at the windows client, so I guess it no longer a possible workaround.

It seems that MixedReality-WebRTC is not able to handle incoming portrait video.

[PureCoderPan]

Hello @Dryra @djee-ms
Just figured out it, we need to rotate image by our self.
two steps:

1. peer_connecton.cpp PeerConnection::OnAddTrack metheod
   change rotation_applied = false
   rtc::VideoSinkWants sink_settings{};
   sink_settings.rotation_applied = false;
2. video_frame_observer.cpp VideoFrameObserver::OnFrame metheod

rtc::scoped_refptrwebrtc::VideoFrameBuffer buffer(
frame.video_frame_buffer());
// rotate frame if needed
if (frame.rotation() != webrtc::kVideoRotation_0 ||
buffer->type() == webrtc::VideoFrameBuffer::Type::kI420)
{
auto frameTemp = webrtc::VideoFrame(
webrtc::I420Buffer::Rotate(*buffer->GetI420(), frame.rotation()),
webrtc::kVideoRotation_0, frame.timestamp_us());

OnFrameCallback(frameTemp);

} else {
OnFrameCallback(frame);
}

// do the callback things
void VideoFrameObserver::OnFrameCallback(
const webrtc::VideoFrame& frame) noexcept {

....
i420_callback_(...)
argb_callback_(...)

}

maybe we should add rotate angle arg in i420_callback_ and argb_callback_ in the future.

Where can find peer_connecton.cpp and video_frame_observer.cpp source code, my only found mrwebrtc dll in my project about MixedReality WebRTC ? I have same problem need to try。Any suggestion will appreciated.

[djee-ms]

See the answer on #661 you opened to ask the same thing.