Concat and real time streaming

[jb-alvarado]

Sorry that I ask my question here, but I hoped that I get some thought from you.

I have write a playout solution, based on python and ffmpeg. It works, but not 100% clean, that is way I would like to rewrite it based on PyAV.

I have play a bit around with your module, but some things I can't figure out:

1) When I want to concat multiple clips, would you recommend to buffer them? So when the script change from one clip to the next, that there is no delay.

2) The most easy way to concat can be done like this:

import os

import av

dir_path = os.path.dirname(os.path.realpath(__file__))

output = av.open('concat.mkv', 'w')

out_stream_v = output.add_stream('libx264', 50)
out_stream_a = output.add_stream('aac')
out_stream_v.width = 512
out_stream_v.height = 288
out_stream_v.options = {'preset': 'faster', 'crf': '24'}
out_stream_a.options = {}


list = [
    'waves.mp4',
    'waves.mp4'
]

for clip in list:
    input_ = av.open(os.path.join(dir_path, 'test_clips', clip))

    in_stream_v = input_.streams.video[0]
    in_stream_a = input_.streams.audio[0]

    for packet in input_.demux(in_stream_v, in_stream_a):
        if packet.dts is None:
            continue

        type = packet.stream.type

        for frame in packet.decode():
            if type == 'video':
                packet = out_stream_v.encode(frame)
            elif type == 'audio':
                packet = out_stream_a.encode(frame)

            output.mux(packet)

output.close()

But now I have the problem, that my dts and pts are not monotonically. So how can I generate a clean increasing dts and pts packet stream? And also: What would be, when the values are increasing over multiple day, or weeks, do I have to rest the values one time?

3) How can I encode in real time? I saw in this tutorial that he works with a timer and an older player example from you uses also a clock. Could you give me here a hint, for my use case?

As I say I would appreciate your thought on that very much!

Have a nice day!

[jb-alvarado]

I have now done some more tests regarding the PTS problem and came to the following solution (at 50FPS clips):

new_video_pts = 0
new_audio_pts = 0

for clip in list:
    input_ = av.open(os.path.join(dir_path, 'test_clips', clip))
    print(input_)

    in_stream_v = input_.streams.video[0]
    in_stream_a = input_.streams.audio[0]

    for packet in input_.demux(in_stream_v, in_stream_a):
        if packet.dts is None:
            continue

        type = packet.stream.type

        for frame in packet.decode():
            if type == 'video':
                frame.pts = new_video_pts
                packet = out_stream_v.encode(frame)
                new_video_pts += 256

            elif type == 'audio':
                frame.pts = new_audio_pts
                packet = out_stream_a.encode(frame)
                new_audio_pts += 1024

            output.mux(packet)

The Problem with this is, that as soon as a clips comes, where the sample rate is different to the one before, it is not working anymore. With an audio resampler I was also not successful.

What do you think about this?

About the question: resetting the pts value, I figure out when the pts value is an int64 value, then I don't have to reset anything, I have enough to count for many years :)...
Question 1) I can also check off.

Then stays only question 3): how can I encode, or decode in real time for rtmp streaming use cases.

[jlaine]

For clips with identical encodings, a more efficient way is to use FFmpeg's built-in concat format, see:

https://trac.ffmpeg.org/wiki/Concatenate

av.open('mylist.txt', format='concat')

Concerning your code for handling different frame rates, I see you are changing the pts, but you are not changing the time_base so your timestamps are probably garbage.

[jb-alvarado]

The problem with the concat demuxer is, that you can not change dynamical the content of the list nor the list by it self. That is way I need a custom solution.

I belief my code have more mistakes, but thank you for pointing this out about the time_base value. Another problem will be to, when there comes clips with shorter or longer audio lines than the video, this would end up in non sync a/v.

But beside this is there an -re input option in PyAV?