Recording: Don't use stateful filters in ffmpeg video processing (2.5 backport) #16050
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Because the input videos for BigBlueButton recording processing switch resolution and aspect ratio, the filter chain gets re-initialized, and any state in the filters is lost. This causes problems with the following filters: `color`: Timestamps restart from 0, rather than continuing at the point where they left off. `fps=start_time=12.345`: After reset, the fps filter thinks it's at the start of the file again, so the next frame it sees gets duplicated output for timestamps from the `start_time` until it catches back up. `setpts=PTS-STARTPTS`: The 'STARTPTS' is re-read as the first pts the filter sees after reinitialization, so timestamp of the next frame is reset to 0. (In practise, this didn't cause any problems because the duplicate frames created by the fps filter had the original start time.) The end result of all of these issues is that a lot of duplicate frames were created with invalid timestamps, which eventually get discarded by ffmpeg. But a lot of time is wasted, causing recordings to sometimes take hours to process when they should be ready in minutes. The fixes are as follows: * The `color` filters are used to generate the background and substitutes for missing videos. Move them out to separate filter chains by using the 'lavfi' input format, which lets you use a filter as if it was an input file. * Rather than use the `fps` filter's `start_time` feature, use the `trim` filter to remove early frames. * The actual start pts is already known by the script, so replace `setpts=PTS-STARTPTS` with `setpts=PTS-12.345/TB`, substituting in the absolute time.
kepstin
changed the title
Even with the filter changes made, there's still some cases where filter chain hangs can result from filter reconfigurations. To solve the issue completely, I have split out pre-processing video files to separate ffmpeg processes, so that the filter chain for compositing will not ever be reconfigured. Each input video now has a separate ffmpeg process run for it which does the scaling, padding, and video extending steps. To avoid issues with disk space usage and extra cpu usage or quality loss, the output from these separate processes is sent to the compositing ffmpeg process as uncompressed video in a pipe. To simplify the setup, named pipes (special files) are used rather than setting up pipes in the ruby code programmatically. The extra ffmpeg processes are configured to log to files, and when complete their log output is copied to the recording processing log. Processes are joined to ensure zombie processes are not created, and the return codes of all the processes are checked so errors can be detected. Due to the overhead of transferring video through pipes, this might be a bit slower than the 2.4 recording processing - but on the other hand, some of the video decoding and scaling happens in parallel, so it might balance out.
The tpad filter is problematic on the variable-framerate webcam files, and the result can end up being hangs (or, at least, very slow processing) in the compositing. Move the tpad filter to the compositing process where it can run after the fps filter has converted the video to constant framerate. It still needs to run before the start trimming, so switch to using the trim filter rather than the fps filter's start_pts feature.
|
Kudos, SonarCloud Quality Gate passed!
|
|
Hi, How can we enable generating mp4 to use FFmpeg? Is it available in the latest version of 2.5? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because the input videos for BigBlueButton recording processing switch resolution and aspect ratio, the filter chain gets re-initialized, and any state in the filters is lost. This causes problems with the following filters.
The end result of all of these issues is that a lot of duplicate frames were created with invalid timestamps, which eventually get discarded by ffmpeg. But a lot of time is wasted, causing recordings to sometimes take hours to process when they should be ready in minutes.
This is a backport of #16047 to BigBlueButton 2.5, see that PR for details on the problem and fix.
Fixes #16044