Native HLS downloader results in truncated audio stream

[pdball]

Checklist

• [x ] I'm reporting a broken site support
• [x ] I've verified that I'm running yt-dlp version 2021.09.02
• [x ] I've checked that all provided URLs are alive and playable in a browser
• [x ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
• [x ] I've searched the bugtracker for similar issues including closed ones

Verbose log

yt-dlp --verbose -f hls-multiple_audio_tracks-English https://video.pbs.org/video/episode-1-o8ybej/
[debug] Command-line config: ['--verbose', '-f', 'hls-multiple_audio_tracks-English', 'https://video.pbs.org/video/episode-1-o8ybej/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] yt-dlp version 2021.09.02 
[debug] Python version 3.6.9 (CPython 64bit) - Linux-5.4.0-84-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.8, ffprobe 3.4.8, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: keyring, mutagen, pycryptodome, sqlite, websockets
[debug] Proxy map: {}
[debug] [generic] Extracting URL: https://video.pbs.org/video/episode-1-o8ybej/
[generic] episode-1-o8ybej: Requesting header
[redirect] Following redirect to https://www.pbs.org/video/episode-1-o8ybej/
[debug] Using fake IP 6.151.228.73 (US) as X-Forwarded-For
[pbs] Downloading JSON metadata
[debug] [pbs] Extracting URL: https://www.pbs.org/video/episode-1-o8ybej/
[pbs] episode-1-o8ybej: Downloading webpage
[pbs] episode-1-o8ybej: Downloading widget/partnerplayer page
[pbs] episode-1-o8ybej: Downloading portalplayer page
[pbs] episode-1-o8ybej: Downloading 0 video url info
[pbs] episode-1-o8ybej: Downloading m3u8 information
WARNING: [pbs] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[pbs] episode-1-o8ybej: Downloading 1 video url info
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] 3058097753: Downloading 1 format(s): hls-multiple_audio_tracks-English
[debug] Invoking downloader on "https://ga.video.cdn.pbs.org/videos/guilt/17d8c436-89db-41d3-843c-f22e096ebb21/2000249857/hd-16x9-mezzanine-1080p/mast5135_r-hls-16x9-1080pAudio%20Selector%201.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 519
[download] Destination: Guilt - Episode 1 [3058097753].mp4
[download] 100% of 59.56MiB in 00:54
[debug] ffmpeg command line: ffprobe -show_streams 'file:Guilt - Episode 1 [3058097753].mp4'
[FixupM3u8] Fixing malformed AAC bitstream of "Guilt - Episode 1 [3058097753].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Guilt - Episode 1 [3058097753].mp4' -c copy -map 0 -dn -f mp4 -bsf:a aac_adtstoasc 'file:Guilt - Episode 1 [3058097753].temp.mp4'
 $ 

Description

The resulting file duration is 04:12 minutes - should be 52 minutes.
Running with --downloader=ffmpeg fixes the problem

git bisect points to this being broken from the start

Bisecting: 0 revisions left to test after this (roughly 0 steps)
[177877c] [extractor] Always prefer native hls downloader by default

[pukkandan]

Try using --fixup never. If it solves your issue, the problem is probably the same as #618 (comment). It is pervasive enough that I have considered changing the default downloader back (or setting --fixup never as default). But in the end, the proper solution is to have this issue be addressed by ffmpeg and not us.

[pdball]

'--fixup never' results in an almost complete download.
ffprobe for the downloaded file with the above:
[aac @ 0x55eccb85c080] Estimating duration from bitrate, this may be inaccurate
Input #0, aac, from 'Guilt - Episode 1 [3058097753].mp4':
Duration: 00:51:52.70, bitrate: 160 kb/s

ffprobe when using '--downloader ffmpeg':
Duration: 00:51:58.46, start: 0.000000, bitrate: 158 kb/s
without the 'estimating duration' warning

[pukkandan]

The timestamps in the file provided by server needs to be corrected (hence the wrong duration without fixup). ffmpeg does it correctly when using it as a downloader. But when asked to do it on an already downloaded file, it totally breaks the file at the point where the timestamp correction is needed. This issue does not exist in ffmpeg 3.0.1, only in later versions. So it is a regression afaik.

I do not know enough about the internal workings of ffmpeg to know what caused this to break or whether there is any workarounds. But it could be possibly related to ytdl-org/youtube-dl#28042 (see #871 (comment)) but the patch for that issue doesn't fix this. So I am not sure

[pdball]

Actually, 'fixup never' leaves a .mp4 container with the 00:51:52.70 duration as previously shown.
If I extract the audio in that .mp4 download, the resulting is Duration: 00:04:12.29.
With '--downloader=ffmpeg' it is Duration: 00:51:58.46

[pukkandan]

Let me try downloading it. I'll get back to you

[pukkandan]

You are correct that video players are unable to play the audio after the 4min mark. However, performing fixup with ffmpeg3.0.1 gave me the corrected file with a full 52:09 duration. So this is the exact same issue as #618

[fstirlitz]

Have you tried comparing the outputs of ffprobe -show_packets -show_frames -show_format -show_data between FFmpeg downloader and native downloader pre- and post-fixup?

I suspect the whole brokenness might as well be a bug in the native downloader. (Compare #637.)

[pdball]

Good news is that native hls downloader worked when using latest static build of ffmpeg I downloaded from https://johnvansickle.com/ffmpeg/
ffmpeg N-58594-g715f63232f-static, ffprobe N-58594-g715f63232f-static

As 'latest' is the only ffmpeg supported by the ffmpeg community I vote for closing this issue.

[pdball]

It is used as a final step - look at the last three lines of the verbose log

[0xallie]

Yeah, I already deleted my comment because I realized that.

[pukkandan]

Great, I can confirm that the latest builds do in fact fix this. My ffmpeg build was just a few weeks old, so it is probably a recent fix.

I'm not interested enough to go though the ffmpeg commits and figure out which one fixed it, but if anyone knows, let me know (just out of curiosity)

---

@fstirlitz You could argue that it is the native downloader's fault - in the sense that the native downloader doesn't have the capability to fix the timestamps. But we know this is a potential issue when downloading hls streams. This is why we pass it to ffmpeg for fixup

[pukkandan]

The root cause of this issue is: https://trac.ffmpeg.org/ticket/9433
A patch has been submitted to ffmpeg devs: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210927213133.28258-1-jeebjp@gmail.com
and this patch has been added to our custom ffmpeg builds: https://github.com/yt-dlp/FFmpeg-Builds

250a938 also implements a workaround for the issue which should work even without the above mentioned patch

[Jules-A]

Just wanted to point out that 250a938 doesn't fix it for all sites (still get issues with Funimation) but using the custom ffmpeg builds does.

[Zerogoki00]

this is NOT fixed, I just encountered this problem while downloading a long stream from youtube

[bashonly]

@Zerogoki00 If you're talking about a livestream, it would be an unrelated issue since the native HLS downloader is not used for livestreams. Either way, you should open a new issue with a verbose log