Merging formats takes a long time

[yonas]

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

• I'm requesting a feature unrelated to a specific site
• I've looked through the README
• I've verified that I have updated yt-dlp to nightly or master (update instructions)
• I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

The "Merging formats" stage takes a long time if the video is large. For example 20GB.

Is there a better, faster way to dump the audio file at the end of the video file into a container format. Perhaps MKV does this better than MP4? I'm not sure, just guessing here.

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• If using API, add 'verbose': True to YoutubeDL params instead
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[youtube] Extracting URL: https://www.youtube.com/watch?v=yl_ZLBwa4cA
[youtube] yl_ZLBwa4cA: Downloading webpage
[youtube] yl_ZLBwa4cA: Downloading ios player API JSON
[youtube] yl_ZLBwa4cA: Downloading android player API JSON
[youtube] yl_ZLBwa4cA: Downloading m3u8 information
[info] yl_ZLBwa4cA: Downloading 1 format(s): 571+251
[download] Resuming download at byte 2501693440
[download] Destination: Space Exploration in 8K Ultra HD ｜ Interstellar (Space Sounds) [yl_ZLBwa4cA].f571.mp4
[download] 100% of   20.44GiB in 00:13:15 at 26.32MiB/s
[download] Destination: Space Exploration in 8K Ultra HD ｜ Interstellar (Space Sounds) [yl_ZLBwa4cA].f251.webm
[download] 100% of  135.91MiB in 00:00:13 at 10.08MiB/s
[Merger] Merging formats into "Space Exploration in 8K Ultra HD ｜ Interstellar (Space Sounds) [yl_ZLBwa4cA].webm"

version: 2024.04.09
Ubuntu 23.10

[dirkf]

How does the merge time compare with the time taken to write a 20GB file on the same storage medium, say with dd if=/dev/random bs=1M count=20K of=20GB_test.dat?

[yonas]

@dirkf I would use the size of the audio file, which is 136MB, and takes 6.4 seconds at 22 MB/s.

$ time dd if=/dev/random bs=1M count=136 of=136MB_test.dat
136+0 records in
136+0 records out
142606336 bytes (143 MB, 136 MiB) copied, 6.42177 s, 22.2 MB/s

________________________________________________________
Executed in    6.43 secs      fish           external
   usr time    0.00 millis    0.00 micros    0.00 millis
   sys time  677.36 millis  655.00 micros  676.71 millis

[bashonly]

the combined filesize of both video and audio formats is what is written to disk during merge, not just the audio

[yonas]

@bashonly

the combined filesize of both video and audio formats is what is written to disk during merge, not just the audio

Is there a way to append the audio to the end of the video. Some kind of fseek with SEEK_END, or opening a file in APPEND mode.

[bashonly]

No, it's not possible with ffmpeg

[dirkf]

When the formats are combined into an MP4 container, the audio and video streams are multiplexed, like doing up a zipper. This means that during playback (when it's important that data is made available in real time) a read operation can get the data for the next frame(s), instead of the I/O subsystem having to maintain two input streams. But as a result the merge has to re-write the whole file.

I assume that this applies to MKV as well.

[yonas]

@bashonly Thanks

@dirkf Thanks. It seems MKV might not require multiplexing:

"It can add a single or multiple audio tracks to MKV by rewrapping the audio into the .mkv container without re-encoding the video. Thus, it can export the video fast and won’t affect the video quality. By the way, this tool can be used to extract audio as well as remove audio tracks from MKV."

-- https://www.videoconverterfactory.com/tips/add-audio-track-to-mkv.html

In which case, passing --merge-output-format mkv might work.

[bashonly]

multiplex == mux

re-encoding != remuxing

[bashonly]

Closing since there's nothing we can do. The optimization would need to be made to ffmpeg