FPS fluctuations in stream after updating to iOS 15.4

[rouzbeh-abadi]

Hi,
After updating the iOS and iPadOS to 15.4 from 15.3.1, the FPS occasionally drops to a very low value for a second and then goes back up to near 30 fps, making streaming very laggy.

in iOS 15.3.1 everything works as expected:

Any recommendation?

[edgenod]

Recommendations? Linux? Windows? Android? Harmony OS? or keep a version that is working from Apple.

Have you tried reporting this to the closed-source vendor Apple? What was their response? They run locked-down proprietary code, and most likely screwed something up in their code, so it should be on them to diagnose and fix their stuff, or pinpoint why it fails on their OS and not on other OSes.

[PonyHugger]

Hi there!
We are experiencing exactly the same issue! Until we updated to 15.4, everything was fine.

Fps:

frame height:

combined:

You can see, that the frame rate is fine for a short period of time, when the frame size changes...

Maybe this problem is related to this thread https://developer.apple.com/forums/thread/702891 ?

Cheers!

[PonyHugger]

Hello,

someone at the apple forums thread has posted, that the behavior might be related to frames, having the same presentation timestamp:

I encountered the same problem, and finnaly I found the reason behind the problem after varied tries, maybe useful for you.

In my case, if the VideoToolbox encoder receives continuous two frames at least with the same pts, the encoder will decline output bitrate very quickly..

it's ok: pts: 0, 0.2, 0.4, 0.6, 0.8 ....

it's bad (two frame with the same pts 0.4) pts: 0, 0.2, 0.4, 0.4, 0.6, 0.8 ....

You may check pts of the frames passed to the encoder whether or not the same, if so, discard the later one or update the pts of the later.

Source: https://developer.apple.com/forums/thread/702891?answerId=709669022#709669022

Unfortunately we were not able to find the place, to get the frame's meta data and manipulate it, to ensure, that there are no frames with the same presentation timestamp.

Is there someone who knows, which file to edit, in order to get this fixed?

Cheers!

[jianjunz]

Hi,

The presentation timestamp is defined here for H.264, and here for HEVC.

[PonyHugger]

Hi @jianjunz ,

thank you very much, we'll give it a try...!

[PonyHugger]

Hi @jianjunz,
unfortunately, the presentation timestamp looks fine. We've tested a couple of devices, which were running great before updating to 15.4 or 15.4.1.

• ipad (5th gen): fine
• iphone 10: affected
• iphone 13 pro max: affected (from time to time?!)

Can you confirm, that you are having the same problem on an ios device with ios 15.4 or 15.4.1?

[rouzbeh-abadi]

Hi @PonyHugger, @jianjunz
We have still that issue on iOS 15.4, 15.4.1, and 15.5 beta 1 and beta 2

• iPad Pro (4th Gen): affected
• iPhone 13: affected
• iPhone 13 Pro Max: affected

I also checked the presentation timestamp, and it looks pretty fine; here, you can see the WebRTC logs in a frame-dropping season that we got in the debug schema. The timestamps before and after dropping look ok, and the frame dropping season started with "Reporting high QP", Do you have any suggestions to fix it @jianjunz

2022-04-22 13:50:27.987362+0200, timestamp: 258304.831000
2022-04-22 13:50:28.152174+0200, timestamp: 258304.997000
2022-04-22 13:50:28.354679+0200, timestamp: 258305.197000

(quality_scaler.cc:304): Reporting high QP, framedrop percent 80
(video_stream_adapter.cc:478): Scaling down resolution, max pixels: 552960
(video_stream_encoder_resource_manager.cc:604): Downgrade counts: fps: {quality:0cpu:0}, resolution {quality:1cpu:0}
(video_stream_encoder.cc:1894): Updating sink restrictions from QualityScalerResource to { max_pixels_per_frame=552960 }
(resource_adaptation_processor.cc:229): Resource "QualityScalerResource" signalled kOveruse. Adapted down successfully. Unfiltered adaptations: { res=1 fps=0 }
(video_source_sink_controller.cc:76): Pushing SourceSink restrictions: max_fps=60 max_pixel_count=552960 target_pixel_count=null
(video_adapter.cc:275): Frame size changed: scaled 632 / out 1714 / in 1714 Changes: 11 Input: 1280x720 Scale: 3/4 Output: 960x540 fps: 60/-1 alignment: 2
(video_stream_encoder.cc:1248): Video frame parameters changed: dimensions=960x540, texture=1.
(video_stream_encoder.cc:788): ReconfigureEncoder:
Simulcast streams:
0: 960x540 fps: 60 min_kbps: 30 target_kbps: 5120 max_kbps: 5120 max_fps: 60 max_qp: 56 num_tl: 1 active: true

(RTCVideoEncoderH264.mm:376): Initial encoder frame rate setting 60 is larger than the maximal allowed frame rate 52.
(RTCVideoEncoderH264.mm:586): Resetting compression session due to non-matching pixel format.
2022-04-22 13:50:28.480997+0200, timestamp :258305.297000
2022-04-22 13:50:28.485119+0200, timestamp :258305.330000
(RTCVideoEncoderH264.mm:788): Generated keyframe
2022-04-22 13:50:28.519033+0200, timestamp: 258305.364000

[zzw315]

I have the same problem。the video frame fps is very low，because webrtc drops a lot of frames。
On ios 15.4.1 , Videotoolbox makes large bitrates on baseline3.1 mode.
I used 2 methods to solve the problem:

1. By modifying the maximum bit rate limit value, it is 1.5 times by default and 1.2 times by modification

RTCVideoEncoderH264.mm
// The ratio between kVTCompressionPropertyKey_DataRateLimits and
// kVTCompressionPropertyKey_AverageBitRate. The data rate limit is set higher
// than the average bit rate to avoid undershooting the target.
const float kLimitToAverageBitRateFactor = 1.5f; ////you can change this value to 1.2f

2. By changing the encoder settings, change to，baseline3.3 or highline5.1
   It is H264 profileLevel。

[PonyHugger]

Hello!

@rouzbeh-abadi, thanks for your feedback!

@zzw315, thank you very much! We'll give it a try!

[Myshny]

@zzw315
How do you change H264 profileLevel ? I faced with same issue in iPhone 12 mini IOS 15.6 - frames dropped time to time.
In IOS/Swift I have

let h264Parameters: OWTVideoCodecParameters = OWTVideoCodecParameters()
h264Parameters.name = OWTVideoCodec.H264 //codec
h264Parameters.profile = "" //string

What should I put in there? For baseline3.3 or highline5.1 ?