H264 Video decoding issues on rk3288

[rjam]

Hi,

We've integrated with linphone's SDK 4.4 (4.4.0-13-gc99cb9c88) into our app and have managed to successfully make calls between our android device and our sip intercom, however the video received on the app always appears distorted, like so:

The device receiving the call is an android tablet running Android 8.1 (27) and the info received by linphone labels it as { 'rockchip', 'rk3288', 'rk3288' }. I'm aware of other issues related to rockhip chipsets, and this on in particular (namely #738 and #773), and I do see some logs that seem to suggest the hw decoder might be at fault here as well?

When receiving 320x240 video:

2020-09-04 14:18:56:039 [/mediastreamer] MESSAGE MediaCodecDecoder: new format 320x240 :
2020-09-04 14:18:56:043 [/mediastreamer] ERROR Mismatch between decoder new format and output image detected: 320x240 vs 240x240
2020-09-04 14:18:56:046 [/mediastreamer] MESSAGE H26xDecoder: first frame decoded 240x240

When receiving 640x480 video:

MediaCodecDecoder: new format 640x480 :
    {crop-right=639, color-format=0, crop-bottom=479, slice-height=480, image-data=java.nio.HeapByteBuffer[pos=0 lim=104 cap=104], mime=video/raw, crop-left=0, width=640, stride=640, crop-top=0, height=480}
Mismatch between decoder new format and output image detected: 640x480 vs 240x320
H26xDecoder: first frame decoded 240x320
allocate_gl_textures: allocated new textures[0] (256 x 512)

However, approximately a year ago, we developed a similar app using Linphone's sdk (4.2?) that was able to correctly receive video calls on this same device, so I'm wondering if there's any configuration that has since become default and we can change to workaround this issue? I've tried downgrading to 4.2 with no luck (same issue, maybe something changed on the pre-built sdks?) and also forcing the usage of OpenH264 by using OpenH264DownloadHelper and setting H264Helper.setH264Mode(H264Helper.MODE_OPENH264, core) but that doesn't seem to make a difference on the decoder used?

Any guidance would be appreciated.

full_log_320x240

[Viish]

Hi,
Have you tried to make a call from our Linphone android app (or iPhone, desktop, etc...) instead of your SIP intercom?
Cheers,

[rjam]

Hi Viish 👋 to be honest I haven't, I did reproduce the issue on the sdk's sample app to discard anything wrong with our integration, but since the intercom has worked previously and we could still access its rtsp feed separately without issues I didn't double check that. I'll do so and get back to you.

[rjam]

Just a quick update fyi: running the linphone app on the tablet and a cell phone, we were able to have the video work fine during a call. We're now going to run a few more tests to check if the issue is in the intercom itself or in the sdk/tablet combination/configuration.

[Viish]

If using a Linphone app the result is OK it means the problem lies in your hardware encoder.

[rjam]

After a few more tests using the linphone app we've confirmed that calls between the intercom and a phone work fine:

But the calls between the intercom and the tablet, even with the linphone app, are still broken:

Given this, I don't think the intercom can be at fault. Is there a way force a software decoder on the sdk to rule out the tablet's hw?

[Viish]

Please try with the following APK on your tablet:
https://www.linphone.org/snapshots/android/linphone-android-debug-4.4.0-alpha.3+a679e365.apk

[rjam]

Please try with the following APK on your tablet:
https://www.linphone.org/snapshots/android/linphone-android-debug-4.4.0-alpha.3+a679e365.apk

Same outcome as before 😞

[Viish]

Try to add your device info in this table, rebuild the SDK and then the app:
BelledonneCommunications/mediastreamer2@e757ca9

[rjam]

Try to add your device info in this table, rebuild the SDK and then the app:
BelledonneCommunications/mediastreamer2@e757ca9

@Viish what would be a stable branch to build the SDK of? I tried release/4.4, made the change and rebuilt. It builds all the way to the "Generating the SDK (zip file and aar)" task but then it fails because it can't find the annotations import which doesn't look like a build error to me:

> Task :compileReleaseJavaWithJavac FAILED
/linphone-sdk/build/linphone-sdk/android-arm64/share/linphonej/java/org/linphone/core/ConfiguringState.java:22: error: cannot find symbol
import androidx.annotation.Nullable;
                          ^
  symbol:   class Nullable
  location: package androidx.annotation
/linphone-sdk/build/linphone-sdk/android-arm64/share/linphonej/java/org/linphone/core/ConfiguringState.java:23: error: cannot find symbol
import androidx.annotation.NonNull;

(..a bunch more similar errors)

cmake command wascmake -D LINPHONESDK_PLATFORM=Android -D ENABLE_NON_FREE_CODECS=ON ..

[Viish]

If you have this issue means you aren't building release/4.4 branch but master.
Clear you build directory, update submodules and start build process again.

[rjam]

Thanks for the tip, after updating the submodules and rebuilding from scratch I got the sdk to build properly. When importing it into the project I always get a link error from libc++_shared.so though:

java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.app.debug-0KpKr7K_7KH6LcSylIsGnQ==/base.apk"],nativeLibraryDirectories=[/data/app/com.app.debug-0KpKr7K_7KH6LcSylIsGnQ==/lib/x86, /system/lib, /vendor/lib]]] couldn't find "libc++_shared.so"

I've tried cleaning and rebuilding but no luck, always the same. I've added LinphoneSdkBuildDir to my gradle.properties and to make sure i'm using my local build I've replaced the maven repo with the local one by doing so:

maven { url file(LinphoneSdkBuildDir + '/maven_repository/') }
//maven { url "https://linphone.org/maven_repository/" }

The app builds fine, but crashes at runtime. Any idea?

[Viish]

Your device is x86 and by default our SDK builds are only armv7/arm64.
Add the following to your cmake options: -DLINPHONESDK_ANDROID_ARCHS=x86

[rjam]

The addition of {"rockchip", "rk3288", "rk3288"}, to the list of "tv" devices on the h264 decoder did the trick, the video works fine now! 🎉

I see there are already some other references of the rk3288 on that file, is this new device reference something that you'd be willing to add to the official build so that we don't have to maintain our separate SDK just for this one change?

Thanks again for the help

[Viish]

I'm glad it works.
Yes I will integrate it upstream and it will be part of next 4.4 and 4.5 releases.

[rjam]

Awesome! I see the change was already merged to ms2, can you give me an estimate of when the next sdk build to include this might be released to the maven repo?

[Viish]

For master (4.5.0-alpha) it is already available, for 4.4 branch it should be available this week as we plan to release an update for the SDK (4.4.2) & the app (4.3.1).

[Viish]

Btw, until we tag the 4.4.2 release, you can use the following AAR from maven: 4.4.2-pre.13+2de967f

[Roger1103]

将{"rockchip", "rk3288", "rk3288"},h264解码器上的“电视”设备添加到列表中可以解决问题，现在视频可以正常工作！🎉

我看到该文件上已经有rk3288的其他引用，这个新设备引用是您愿意添加到官方版本中的东西，这样我们就不必为这一更改而维护我们单独的SDK ？

再次感谢您的帮助

I have the same problem,Can you help me.My device is RK3399

[Roger1103]

顺便说一句，直到我们标记了4.4.2发行版为止，您可以使用maven中的以下AAR：4.4.2-pre.13 + 2de967f

@Viish 我有同样的问题，能帮我吗。我的设备是RK3399

[ALLien-Lee]

@Viish hey brother, it's nice to see this issue, our company's tablet products always have the problem. our tablets has two platforms, RK3288 and RK3399, currently tested the version of 4.5.0-Alpha, RK3288 decoding is successed, but RK3399 not yet, I hope the sdk update can support RK3399.

[Viish]

As stated in BelledonneCommunications/linphone-sdk#62 (comment), RK3399 support was added yesterday into master (4.5.0-alpha), so you need a version 4.5.0-alpha-125 or newer.
Please not it hasn't been built & uploaded to our maven repository yet, so unless you build the SDK yourself it is expected that it doesn't work yet.

[ALLien-Lee]

As stated in BelledonneCommunications/linphone-sdk#62 (comment), RK3399 support was added yesterday into master (4.5.0-alpha), so you need a version 4.5.0-alpha-125 or newer.
Please not it hasn't been built & uploaded to our maven repository yet, so unless you build the SDK yourself it is expected that it doesn't work yet.

Nice to hear from you and see that you have added support for rk3999 to master.However, it's difficult for me to build the SDK,so I sincerely still hope that the alpha-125 will be updated to maven_repository in the near future. thank u.

[Viish]

It is now available on our Maven:

• 4.4.2-pre.15+f6a86ca
• 4.5.0-alpha.126+18d3a91

[ALLien-Lee]

It is now available on our Maven:

• 4.4.2-pre.15+f6a86ca
• 4.5.0-alpha.126+18d3a91

Thank you very much, the RK3399 table is working.

[fengfeilong0529]

@Viish Hi,it's nice to see this issue,i met the same question.My device is rk3126c,is there any sdk version that has already support this device now?If not,i can just try to rebuild the SDK.

[Viish]

@fengfeilong0529 Try adding your device to this table: https://gitlab.linphone.org/BC/public/mediastreamer2/-/blob/master/src/voip/h26x/media-codec-h264-decoder.cpp#L216
If it works, I'll add it for our next release.

[fengfeilong0529]

ok,i'll have a try,thanks!

…

---- 回复的原邮件 ---- | 发件人 | Sylvain ***@***.***> | | 日期 | 2022年01月14日 17:17 | | 收件人 | ***@***.***> | | 抄送至 | ***@***.******@***.***> | | 主题 | Re: [BelledonneCommunications/linphone-android] H264 Video decoding issues on rk3288 (#1153) | @fengfeilong0529 Try adding your device to this table: https://gitlab.linphone.org/BC/public/mediastreamer2/-/blob/master/src/voip/h26x/media-codec-h264-decoder.cpp#L216 If it works, I'll add it for our next release. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: ***@***.***>

[fengfeilong0529]

When I build the 4.4 SDK, I got this error.What does it mean?@Viish
Could you please add {"rockchip","Sasincomm S09","rk3126c"} to the list for a 4.4 debug branch and add a generated aar file to Maven repository? It's so difficult to build the SDK! ^_^

[Viish]

When I build the 4.4 SDK, I got this error.What does it mean?

As said in the logs: Could not find the patch program. So install the patch tool and you should be fine.
Also don't build 4.4 SDK, latest release is 5.0.

[fengfeilong0529]

When I build the 4.4 SDK, I got this error.What does it mean?

As said in the logs: Could not find the patch program. So install the patch tool and you should be fine. Also don't build 4.4 SDK, latest release is 5.0.

Hi，I've tried to add {"rockchip","Sasincomm S09","rk3126c"} to BelledonneCommunications/mediastreamer2@e757ca9 ，and it worked fine! So, can you add it to the next version please?

[Viish]

Done in https://gitlab.linphone.org/BC/public/mediastreamer2/-/merge_requests/599, will be part of 5.2 SDK release, in the meantime it's on master branch.

[Viish]

Merge request has been merged, will be available tomorrow in maven as latest 5.2.0-alpha