FPS drop and lags during streaming

[111913]

Hi Thibault! First of all, thanks for the library!

Implementing streaming to my project I followed your experience as an example, but I ran into some difficulties. Do you mind if I ask few questions?

When you start the application, it is noticeable that the camera frame rate is below 30, although the value is set to 30, and when you start the stream, the fps drops even more. As far as I understood, this is due to the call to the holder.setFixedSize() method every time the surface changes.

I tried to call holder.setFixedSize() only in surfaceCreated() and then there are no problems with fps, but in this case, when the stream starts, the information about the set fps does not reach the server, which leads to errors and the stream does not play.

Can you please tell me how to fix this? I would be grateful for any advice!

[ThibaultBee]

Hi!
The first thing I can think of is that your CPU is overload. What is your device and Android version?
Do you have the same issue with the example application?
Do you have the logcat to see if anything wrong?

The preview has nothing to do with the live stream. Your camera device send frames to the preview and to the encoder. Changing something on the preview should not change anything on the live stream.

[111913]

Thank you for the answer! I have Redmi Note 8 pro:
Android version - 11
MIUI - 12.5.3
CPU - MediaTek Helio G90T
RAM - 8GB

Yes, with your example application has the same issue.
Logs attached, also a screenshot Profiler. The application was launched with this SurfaceHolder.Callback::

    private val surfaceViewCallback = object : SurfaceHolder.Callback {
        var nbOnSurfaceChange = 0

        override fun surfaceCreated(holder: SurfaceHolder) {
            nbOnSurfaceChange = 0
        }

        override fun surfaceChanged(holder: SurfaceHolder, format: Int, width: Int, height: Int) {
            require(context != null)

            try {
                nbOnSurfaceChange++
                if (nbOnSurfaceChange == 2) {
                    viewModel.startPreview(holder.surface)

                } else {

                    val choices = context!!.getCameraOutputSizes(
                        SurfaceHolder::class.java,
                        viewModel.cameraId
                    )

                    chooseBigEnoughSize(choices, width, height)?.let { size ->
                        holder.setFixedSize(size.width, size.height)
                    }
                }
            } catch (e: Exception) {
                Log.e(TAG, "!!!Can't start preview due to ${e.message}!!!", e)
            }
        }

        override fun surfaceDestroyed(holder: SurfaceHolder) {
            viewModel.stopPreview()
            binding.surfaceView.holder.removeCallback(this)
        }
    }

LogsAfterStartingStream.txt
LogsAfterStoppingStream.txt

[ThibaultBee]

Hi,

Ok, everything seems alright. Your CPU is really low which is really good.
Anyway, I found this: https://stackoverflow.com/questions/57485050/how-to-increase-frame-rate-with-android-camerax-imageanalysis/57490987#57490987. It is for CameraX but the same settings apply to camera2.
As I only test on Nexus and Pixel, I might have miss something on other constructors. I want to be sure what setting really work on your phone.
It might take a while to test. Can you test on the sample of https://github.com/ThibaultBee/StreamPack/tree/bugfix/framerate?

[111913]

I tested your example, fps is still low, but another problem has been added =).
The camera capture has become dark, although the room is light, there is also a problem with the aspect ratio, but it was there before too. I have attached screenshots.

I didn't change the video settings. They stood by default:
Encoder - AVC/H.264
Resolution - 1280x720
Framerate - 30
Video start bitrate - 2000

LogsAfterStartingStream.txt
LogsAfterStoppingStream.txt

Is there anything else I can be useful?

[ThibaultBee]

Ok.
The dark effect is because AE is disabled.
Anyway, I have changed https://github.com/ThibaultBee/StreamPack/tree/bugfix/framerate.
Can you retest it?

[111913]

Hi Thibault!

Dark effect is gone, CPU does not overload, problems with fps remain, but I noticed that when switching to the front camera, fps does not drop.

I also wanted to clarify with you why call holder.setFixedSize () in surfaceChanged? Considering what you wrote: "The preview has nothing to do with the live stream. Your camera device send frames to the preview and to the encoder. Changing something on the preview should not change anything on the live stream". If it is needed for the correct aspect ratio of the surfaceView, then this doesn't work. I found this example: https://github.com/android/camera-samples/blob/main/Camera2Video/utils/src/main/java/com/example/android/camera/utils/AutoFitSurfaceView.kt on google and it works well with aspect ratio. I don't have much development experience, so any advice from your side would be useful to me =).

LogsAfterStartingApp.txt
LogsAfterStartingStream.txt
LogsAfterStoppingStream.txt

[ThibaultBee]

The code in surfaceChanged comes from the former Android camera sample.
I replace this dirty mechanism by an AutoFitSurfaceView. Can you test again the sample of https://github.com/ThibaultBee/StreamPack/tree/bugfix/framerate?

[111913]

Now there are no problems with fps, at least in SurfaceView, but the stream does not play. The same situation occurred when I called holder.setFixedSize () only in surfaceCreated (), which I wrote about earlier. As our backend developer explained to me, this happens because the fps information does not reach the server, which breaks the transcoding and leads to errors.

LogsAfterStartingApp.txt
LogsAfterStartingStream.txt
LogsAfterStoppingStream.txt

[ThibaultBee]

Now there are no problems with fps, at least in SurfaceView, but the stream does not play. The same situation occurred when I called holder.setFixedSize () only in surfaceCreated ()

Glad to hear. It's a good start :) but I don't understand why :(

As our backend developer explained to me, this happens because the fps information does not reach the server, which breaks the transcoding and leads to errors.

Which fps information are you talking about? Have you try with ffplay (ffplay -fflags nobuffer srt://0.0.0.0:9998?mode=listener)?

[111913]

Which fps information are you talking about?

For example, if start stream using your library, the following information comes to the server:

• Stream #0:0[0x100]: Video: hevc (Main) (HEVC / 0x43564548), yuv420p(tv), 720x1280, 30 tbr, 90k tbn, 90k tbc
• Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 130 kb/s

But the following information should come:

• Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 100 tbc
• Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 122 kb/s

Have you try with ffplay (ffplay -fflags nobuffer srt://0.0.0.0:9998?mode=listener)?

Unfortunately not yet, but I want to try. As soon as I do this, I will inform you of the results :).

[ThibaultBee]

Oh, I guess your server expects the VUI in the SPS (especially the timing_info).
In H.264, VUI is not a mandatory field. vui_parameters_present_flag indicates if VUI is present in the SPS.
Both SPS and PPS are generated by the Android encoder API (aka MediaCodec) and MediaCodec chooses to not add VUI in the SPS.
You should consider to support incoming stream that does not have VUI.

[ThibaultBee]

Hi @BakhromMuydinov,
Have you fixed your issue?

By the way, I added AutoFitSurfaceView in the SDK instead of the sample.

[111913]

Hi @ThibaultBee! I apologize for not getting in touch for a long time.

Yes, we fixed it, thank you very much!
But there is still a problem with the correct aspect ratio of the received stream. I also launched a stream on ffplay, there is the same problem.

What can be done about it?

Stream #0:0[0x20]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt470bg/bt470bg/smpte170m), 720x1280, 29.58 tbr, 90k tbn
Stream #0:1[0x21]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 130 kb/s

[ThibaultBee]

Hi,

Glad you fixed your issue!
You are using StreamPack in portrait. Did you notice the same effect in landscape?

Regards,
Thibault

[111913]

Hi!

The situation is better in landscape orientation, but still this effect, in my opinion, is slightly present.

Stream #0:0[0x20]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt470bg/bt470bg/smpte170m), 1280x720, 29.58 tbr, 90k tbn
Stream #0:1[0x21]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 131 kb/s

[ThibaultBee]

Hi!
Sorry for the delay, I have been really busy, + covid,...
Landscape seems alright on my side if you want to be sure, you should film a square and check if it is still a square on ffplay display.
Also, there is an issue in portrait but I try to fix it. Can you test https://github.com/ThibaultBee/StreamPack/tree/bugfix/portrait ?

Regards,
Thibault

[111913]

Hi!
Ohh.. I wish you a speedy recovery!!
Now everything looks correct in portrait, landscape, and square =).

Thank you very much!!

[ThibaultBee]

Hello,

I merged the branch.
I am waiting for a last development before releasing 1.4.0. Are you okay to wait few days?

Also, could we have a chat on what you are going to do with this library, the features you are using, not using or missing? I am pretty sure you can find me on linkedin with my name.

Regards,
Thibaultt

[111913]

I am waiting for a last development before releasing 1.4.0. Are you okay to wait few days?

No problem! I'm not in a hurry =).

Unfortunately, I'm not on LinkedIn, but if you want, we can chat on Telegram.

[ThibaultBee]

Hi,

These issues are fixed. Fixes will be shipped with next release.

Best regards,
Thibault :)