A bug when changing mobile screen orientation during a video call

[ashfik2000]

During a video call between two users A and B:
if A changes the mobile orientation , localPreviewContainer and remoteVideoContainer are changing ok in VideoHandlerFragment
but in the mobile of user B: localPreviewContainer rotates fine and remoteVideoContainer disappears !!!!

tracing the logcat everything looks fine for VideoHandlerFragment.java and RemoteVideoLayout.java except the following error for SrtpCryptoContext.java :

(SrtpCryptoContext.java:195)#checkReplay: Discarding RTP packet with sequence number ٢٩٠٥١, SSRC ١٧٠٨٨١١٤٥٧ because it is outside the replay window! (roc ٠, s_l ٣٧٦٩٧), guessedROC ٠

Note: i changed the video codec to vp8 and made it the first priority (like as stated in atalk's faq) but the problem still the same.

[cmeng-git]

aTalk uses ZRTP library from Jitsi for video stream encryption, where it implements Replay protection. It was found that the replay protection was unable to re-sync properly in most cases when the video was resumed after an interruption. When this happened, all the received encrypted stream data were discarded by Replay Protection routine, leading to remote view has no video data to playback.

I had once tried to fix the problem but seem it is not a simple problem to fix. I had reported the problem to Jitsi development team long time ago, but they had yet to offer a fix. I am unable to locate the reported bug thread in Jitsi currently, may be they have closed the issue.

If you disable the video encryption, I believe the video call resumes normally when the screen is rotated, or camera is restart after a pause.

[cmeng-git]

Found the Jitsi bug reported thread:

SRTPCryptoContext#checkReplay may failed when resume from a paused secure video call #442

[ashfik2000]

Following what you said: < If you disable the video encryption, I believe the video call resumes normally when the screen is rotated, or camera is restart after a pause.>

i disabled the video encryption from settings but the problem still exist: " the localPreviewContainer rotates fine and remoteVideoContainer disappears !!!!"....
logcat still contains:
"2021-03-02 11:43:21.626 23293-23293/org.atalk.android W/(VideoHandlerFragment.java:633)#lambda$handleRemoteVideoEvent$0$VideoHandlerFragment: Remote video removed."

The only new thing is that the message "(SrtpCryptoContext.java:195)#checkReplay: Discarding RTP packet with sequence number ٢٩٠٥١, SSRC ١٧٠٨٨١١٤٥٧ because it is outside the replay window! (roc ٠, s_l ٣٧٦٩٧), guessedROC ٠" is no longer appears.

[cmeng-git]

This is a normal operation when the remote sender rotates the camera, it is current aTalk implementation to re-init the remote veiw.

"2021-03-02 11:43:21.626 23293-23293/org.atalk.android W/(VideoHandlerFragment.java:633)#lambda$handleRemoteVideoEvent$0$VideoHandlerFragment: Remote video removed."

You are right, there seems to be a problem with video call when one of the parties changes the phone orientation. The last check for device rotation is end 2018. Since then there was a major change to video call to support Conversations. Need to investigate the actual cause.

[cmeng-git]

Please perform the test with the following options:
a. Use only VP8 codec
b. Camera resolution set to 640x480

Found that the success rate when device orientation is changed is no good when the resolution is set too high; it requires a high CPU speed to handle the video rotation.

Also found a problem in android camera implementation, at least on all my Samsung android devices Note8, Note5, Note3 and J730.

Android camera incorrectly calculates the preview size for 720*480 resolution when device is in portrait mode.

[ashfik2000]

Also found a problem in android camera implementation, at least on all my Samsung android devices Note8, Note5, Note3 and J730.

not only samsung devices, i tried on samsung and Huawei devices and Pixel3a API 26 (AVD)

Please perform the test with the following options:
a. Use only VP8 codec
b. Camera resolution set to 640x480

OK i will try this and repport you back.

[ashfik2000]

Please perform the test with the following options:
a. Use only VP8 codec
b. Camera resolution set to 640x480

i did the above and the problem still exists.

[cmeng-git]

What are the two devices you are using for testing?
And what is the failure rate?

Please also note you must disable encryption for the test.

[ashfik2000]

What are the two devices you are using for testing?

Samsung Galaxy J7, S9+ and Huawei Y7Prime.

Please also note you must disable encryption for the test.

was done before testing.

Allow me to remind again:
The device which is rotated has no problem and both localPreviewContainer and remoteVideoContainer are appearing
the problem is in the other party phone, remoteVideoContainer disappears

[ashfik2000]

i dont know if the upgarde from Camera.CameraInfo to android.hardware.camera2 will solve this problem or not.

[cmeng-git]

Early on I was testing with my Note-8 and Note3, with video resolution set to 1208x720; I saw that the remote video failure was 100%. When I change the resolution to 640x480 and uses Note8 and Note5, there was an improvement in remote video display success rate, although it is not 100%. I am still looking at way to improve this behavior, but not sure what will be the final result. aTalk video call under screen orientation change is very complex, involving many routines and a good codec (currently only VP8 implementation supports orientation change) implementation to work properly.

i dont know if the upgarde from Camera.CameraInfo to android.hardware.camera2 will solve this problem or not.
Personally not sure, until I have ported to sue camera2 to test it out. Camera2 new API has undergone major changes, making porting very challenging.

[cmeng-git]

Just released aTalk v2.6.0. aTalk has completely changed the way video streaming is handled on device rotation (required VP8 codec) and camera switching. It has also improved the SRTP Replay Protection reliability under these conditions.

[ashfik2000]

I tried aTalk v2.6.0 and I noticed that switching camera works fine but decreasing video resolution from 1280x720 to 720x480
results in poor quality video call so i had to go back using the previous version.

[cmeng-git]

aTalk is designed to support video resolutions: 1920x1080, 1440x1080, 1280x720, 960x720, 720x480*, 640x480, 320x240
and android OS API-21 min. The new implementation does not change this user option, it only sets the default to 720x480 on initial aTalk installation.

During a video call, aTalk is required to perform all the tasks below:
a. audio/video data encode/decode
b. audio/video rtcp encryption and decryption
c. streaming video format transformation and perform video swap/flip rendering
d. media call data in streaming and receiving
e. remote and local video rendering for display etc
The android device CPU resource loading and network bandwidth increase quickly as the video resolution increases.

Pending on the android device resources and network bandwidth support, user can set the media call streaming video resolution via the main menu | Settings...| Video resolution.

[ashfik2000]

I think that the poor quality and rendering issue is related to VP8 codec becuase when i changed to H264 codec a great improvement was done with video call resolution and rendering (i tested on 3 diferent mobile devices and Android Emulator neglecting that H264 does not provide screen rotation ) and these tests were done more than once on the same android device resources and network bandwidth

[cmeng-git]

I believe this is a known fact between the two codecs' performance. See

https://medium.com/@brandonaaskov/vp8-vs-h-264-originally-written-posted-on-may-19-2014-3b432d58abaa

“When comparing VP8 and x264, VP8 also shows 5–25 times lower encoding speed with 20–30% lower quality at average. For example x264 High-Speed preset is faster and has higher quality than any of VP8 presets at average.”

[ashfik2000]

👍 thanks... I hope that h264 codec will provide screen rotation soon.

[cmeng-git]

aTalk v2.6.1 has added device orientation change support for h264 codec.