Error using MediaRecorder creating HLS segments

[tlaz4]

Hi, this is a great library.

I am attempting to use the MediaRecorder to create hls segments, but ffmpeg encounteres the following error during transmuxing:

Application provided invalid, non monotonically increasing dts to muxer in stream 1: 2217000 >= 2217000

I am using the MediaRecorder as in the server example, except adding both audio and video tracks from a peer connection. The error only occurs on the video tracks, audio works perfectly.
I create the MediaRecorder objects as follows:

HLS_MANIFEST = "live/playlist.m3u8" HLS_SEGMENTS = "live/%s.ts" HLS_OPTS = { 'hls_list_size': '3', 'hls_time': '4', 'hls_segment_type': 'mpegts', 'hls_flags': 'delete_segments+discont_start', 'hls_start_number_source': 'datetime', 'strftime': '1', 'use_localtime': '1', 'hls_segment_filename': HLS_SEGMENTS, }

recorder = HLSRecorder(HLS_MANIFEST, format='hls', options=HLS_OPTS)

Any insight into this would be appreciated, thanks!

[jlaine]

I unfortunately have no idea what's going on here, but there is a good chance the error stems from code outside aiortc. How can I reproduce this myself?

[tlaz4]

I am running server.py as is described in the README, however the only difference is the the on track event at line 136:

`
@pc.on("track")
def on_track(track):
log_info("Track %s received", track.kind)

    if track.kind == "audio":
        recorder.addTrack(track)
    elif track.kind == "video":
        recorder.addTrack(track)

`

I just want to save the captured video + audio for now, so i removed adding any local tracks to send back to the peer.

I run it as python server.py --write-audio file.mp4 and using the provided javascript, create a peer connection with audio and video, setting the video codec to H264.

Maybe this isnt an issue and I am just misunderstanding, are you required to add a local track to send to the peer connection? If i add pc.add(track) into the ontrack event for type video, everything works.

Thanks for your insight.

[ashkov]

@tlaz4 Did you solve this? I have the same issue with the original MediaRecorder. On the random frame from client webcamera I get
webrtc_1 | non-strictly-monotonic PTS
webrtc_1 | Application provided invalid, non monotonically increasing dts to muxer in stream 0: 77312 >= 77312

and MediaRecorder stop working further.
A resulting piece of movie is not readable by VLC.

@jlaine Can I avoid this exception if I don't need strict video especially in the internet stream?

[d3im]

Hi, could You please provide HLSRecorder class (since I don't see it in source)?
Did You try to remux only (not reencoding) of H264?
Thanks, I'm looking for working solution also.

[d3im]

It seems HLSRecorder is just renamed MediaRecorder. Tried it that way and see also:

WARNING:libav.libx264:DPB size (3 frames, 22032 mbs) > level limit (2 frames, 18000 mbs)
WARNING:libav.libx264:MB rate (220320) > level limit (108000)
WARNING:libav.libx264:specified frame type (3) at 0 is not compatible with keyframe interval
WARNING:libav.libx264:forced frame type (3) at 250 was changed to frame type (2)
WARNING:libav.libx264:forced frame type (3) at 250 was changed to frame type (1)
WARNING:libav.libx264:forced frame type (3) at 500 was changed to frame type (2)
WARNING:libav.libx264:forced frame type (3) at 500 was changed to frame type (1)
WARNING:libav.libx264:non-strictly-monotonic PTS
ERROR:libav.hls:Application provided invalid, non monotonically increasing dts to muxer in stream 1: 6411000 >= 6411000
WARNING:libav.libx264:forced frame type (3) at 750 was changed to frame type (2)```

[whitphx]

Hi, I encountered this problem with examples/server/server.py.
I found this error vanishes if I omit relay.subscribe like below

            pc.addTrack(
                VideoTransformTrack(
                    track, transform=params["video_transform"]
                )
            )
            if args.record_to:
                recorder.addTrack(track)

This is not a solution since the two track consumers, pc and recorder, compete the output frames from the track on a first-come, first-served basis without MediaRelay,
though at least I just want to report that I think there is something wrong around MediaRelay...

[levistoddard]

Did anyone else happen to get to the bottom of this? I'm working through debugging for myself atm... Removing the MediaRelay doesn't make a difference and it still occurs

[bmsan]

I'm facing the same issue but intermittently.
I've added some logging inside MediaRecorder.__run_track()
I'm logging the received pts. It seems that at frame level time current_pts > previous_pts even when the error ocurrs.

What I have noticed but it might only be coincidence is that when things are ok the delta_pts > 4000, while when the issue is encountered delta_pts ~ 2800 with one sporadic delta_pts > 4000 (followed a few moments later by a non-strictly-monotonic PTS warning)

I'm attaching two logs:
Working case: https://gist.github.com/bmsan/f0f8d8191dcf781166470eed0f6e3e5c#file-working-example

Error case: https://gist.github.com/bmsan/f0f8d8191dcf781166470eed0f6e3e5c#file-example-with-error

In the error case there can be seen 2 issues:

• non-strictly-monotonic PTS - at least in my case it doesn't seem to break anything
• Application provided invalid, non monotonically increasing dts to muxer in stream - this causes the video recording to fail

The second issue seems to appear when the packet dts doesn't increase (which can be seen at the end of the bad log.

I've created a local workaround for the second issue(Which for me causes the crash) by dropping packages with nonincreasing DTS - but it is a dirty workaround because you lose some frames by doing this. - Initial testing seems to prevent the crashing error.
Edit: I am seeing some artifacts in the resulting videos at certain time moments and I think they might be related to my dropped package. I'll look into this further.

@jlaine would you consider such a workaround(mentioned above) a candidate for a pull request? If so I can create one.

[bmsan]

After trying to understand more in depth what is going on,
with the help of this pyav-ffmpeg timing guide : I noticed that decoder sometimes gives me a slightly higher fps(> 30), which resulted in two consecutive frames that I feed into the encoder to get the same dts and here things break.

For other people looking at this, you can check if this is the case using the following computation inside MediaRecorder.__run_track()

enc_time_base = context.stream.codec_context.time_base
enc_dts = frame.pts * frame.time_base / enc_time_base

For my hardcoded usecase(where I'm expecting 30FPS video streams), I created a MediaRecorderFixed30FPS, where I rewrite the frame.pts and frame.time_base information such that the encoder gets exactly 30 FPS.
@levistoddard maybe this helps.

I see two more generic solutions:

• before calling context.stream.encode(frame) compute the expected packet.dts and see if it increments vs the previous_packet.dts. If not go one of the following ways:
  1. Either drop the frame altogether
  2. increase(offset) the frame.pts such that the packet.dts is incremented, when/if possible decrement the offset(decrease the frame.pts) but making sure the packet.dts still gets incremented. - this will introduce a small timing jitter not sure if noticeable - but it might be more desirable this way than dropping frames.
     @jlaine Would you prefer one of the either two solutions presented here? Please let me know and I can take a stab at it.

[levistoddard]

@bmsan I wasn't able to get to the bottom and ended up writing the service I needed in node-webrtc instead which is working great for the time being. Just one comment on your "I'm expecting 30FPS video streams" — I'd be a little careful with that assumption if your consuming the other side of a MediaStream from a browser as I found in practice that you'll receive variable FPS depending on the quality of the connection

[bmsan]

@levistoddard Very good point.
For the time being I'm using an (unpublished modified version) where if I detect the FPS going sporadically slightly higher than 30 FPS(which was causing the issue at least in my case), and I introduce a slight time jitter to limit it to 30 FPS.

[CLTanuki]

That is the cause: PyAV-Org/PyAV#730

[bmsan]

@CLTanuki the behavior seems to be a little bit different, if I'm not understanding things wrong.

In PyAV-Org/PyAV#730 there seems to be a check regarding the codec id that is failing Invalid argument.

In this issue we are not encountering the codec check fail, but the DTS of the transmitted packed is sometimes non-monotonically increasing(two consecutive packages might have the same DTS). This triggers an error in ffmpeg.

This is due the instantaneous FPS of the webcam sometimes being slightly higher than 30 fps(the value that the MediaRecorder is configured with).

[CLTanuki]

@bmsan , this could not be fixed even with setting exact framerate for webcam. Only with patching framerate in MediaRecorder or setting pts and time_base manually, which is breaking manifest generation.
Reproducing:

• python 3.8 / 3.9
• pyav 8.0.2 / 8.0.3
• kernel 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux
• libblas3 3.9.0-3

Not reproducing:

• python 3.7
• pyav 8.0.2 / 8.0.3
• kernel 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux
• libblas3 3.8.0-2

[bmsan]

@CLTanuki I was thinking something in the lines that you mentioned:

Inside MediaRecorder before calling context.stream.encode(frame) compute the expected packet.dts and see if it increments vs the previous_packet.dts. If not go one of the following ways:

• Either drop the frame altogether
• increase(offset) the frame.pts such that the packet.dts is incremented, when/if possible decrement the offset(decrease the frame.pts) but making sure the packet.dts still gets incremented.
  • this will introduce a small timing jitter not sure if noticeable
  • but it might be more desirable this way than dropping frames.

[CLTanuki]

It was not the case.
Pass this options to the recorder and the issue would be fixed:

'vbsf': 'hevc_mp4toannexb',
'x264opts': 'keyint=24:min-keyint=24:no-scenecut',