stream.hls: Support for EXT-X-DISCONTINUITY tag

[Grabien]

Checklist

• This is a feature request and not a different kind of issue
• I have read the contribution guidelines
• I have checked the list of open and recently closed plugin requests

Description

I'm trying to record an M3U8 stream from this page: https://www.wthitv.com/livestream/

Streamlink records the stream for some time, but then stops recording. When I look at the M3U8 source, I see in it EXT-X-DISCONTINUITY tags, and after each such tag, the TS fragments numeration starts again from "1". It looks like Streamlink stops recordings because of this.

Is it possible to add proper processing of this tag?

It's an example of M3U8:

#EXTM3U
#EXT-X-TARGETDURATION:12
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:54
#EXT-X-START:TIME-OFFSET=0
## 2022-05-19 09:09:52.619518339 +0000 UTC
#EXT-X-DISCONTINUITY-SEQUENCE:1
#EXTINF:6.00,
https://wthi-cdn.bitcentral.com/sef/c_bdb309a0-87cf-402a-b8e3-d54922e91ec2_v_3500K720AWS_s_00085.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=bdb309a0-87cf-402a-b8e3-d54922e91ec2&i=84&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=53&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=6.00
#EXTINF:6.00,
https://wthi-cdn.bitcentral.com/sef/c_bdb309a0-87cf-402a-b8e3-d54922e91ec2_v_3500K720AWS_s_00086.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=bdb309a0-87cf-402a-b8e3-d54922e91ec2&i=85&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=54&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=6.00
#EXTINF:6.00,
https://wthi-cdn.bitcentral.com/sef/c_bdb309a0-87cf-402a-b8e3-d54922e91ec2_v_3500K720AWS_s_00087.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=bdb309a0-87cf-402a-b8e3-d54922e91ec2&i=86&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=55&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=6.00
#EXTINF:6.00,
https://wthi-cdn.bitcentral.com/sef/c_bdb309a0-87cf-402a-b8e3-d54922e91ec2_v_3500K720AWS_s_00088.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=bdb309a0-87cf-402a-b8e3-d54922e91ec2&i=87&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=56&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=6.00
#EXTINF:2.97,
https://wthi-cdn.bitcentral.com/sef/c_bdb309a0-87cf-402a-b8e3-d54922e91ec2_v_3500K720AWS_s_00089.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=bdb309a0-87cf-402a-b8e3-d54922e91ec2&i=88&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=57&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=2.97
#EXT-X-DISCONTINUITY
#EXTINF:12.00,
https://wthi-cdn.bitcentral.com/sef/c_9cd2ffa1-037c-49fe-8387-9e9af9b109ea_v_3500K720AWS_s_00001.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=9cd2ffa1-037c-49fe-8387-9e9af9b109ea&i=0&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=58&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=12.00
#EXTINF:12.00,
https://wthi-cdn.bitcentral.com/sef/c_9cd2ffa1-037c-49fe-8387-9e9af9b109ea_v_3500K720AWS_s_00002.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=9cd2ffa1-037c-49fe-8387-9e9af9b109ea&i=1&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=59&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=12.00
#EXTINF:6.03,
https://wthi-cdn.bitcentral.com/sef/c_9cd2ffa1-037c-49fe-8387-9e9af9b109ea_v_3500K720AWS_s_00003.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=9cd2ffa1-037c-49fe-8387-9e9af9b109ea&i=2&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=60&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=6.03
#EXT-X-DISCONTINUITY
#EXTINF:12.00,
https://wthi-cdn.bitcentral.com/sef/c_a8217637-7a7c-461e-9efd-37f44b680497_v_3500K720AWS_s_00001.ts?s=7b0d8466-3194-4409-85eb-b4dd8a0763d1&c=a8217637-7a7c-461e-9efd-37f44b680497&i=0&v=3500K720AWS&u=https:%2F%2Ffuel-streaming-prod01.fuelmedia.io&m=8&ch=3a9b6ddf-a8ad-422d-adf7-0a26f09c685d&d=12.00

[bastimeyer]

It's unlikely that EXT-X-DISCONTINUITY and EXT-X-DISCONTINUITY-SEQUENCE are ever going to get properly implemented in Streamlink due to how the stream output works, both via Streamlink's CLI and the Streamlink API. These tags get parsed, but they are ignored by the HLS worker+writer.

EXT-X-DISCONTINUITY indicates a complete reset of the stream, with a new stream following afterwards. Streamlink however always outputs a single progressive stream (in case of HLS a stream of concatenated HLS segments), which makes it impossible to output two (or more) separate streams after another unless some kind of video/audio container gets used which supports this and which players can read. Even then this would be challenging considering that the HLS data is not known ahead of time in live streams (number of discontinuities), and it's also problematic because of how FFmpeg's CLI works when multiple unknown numbers of streams would need to get remuxed.

HLS implementations which are directly tied to a video/audio renderer in a single application can control/handle discontinuities, because the renderer's state can be reset, which is the intention of this HLS tag. Streamlink can't do this, as it's a standalone HLS client, and its only purpose is to output a single stream.

A different solution on the CLI side could be writing streams to a variable number of (lazily created) named pipes and making players read from them. The problem here however would be telling the player to switch to a different input. This could be done via IPC interfaces of known players like MPV for example, or by passing the player a playlist of a fixed number of non-lazily created named pipes as input. Another problem would also be that Streamlink doesn't know when the player is finished displaying the content, so if the stream data gets read too fast by the player, a switch to the next stream via IPC commands would happen too soon. This solution is way too specific and error-prone, and is thus not useful.

In case of recording, stream discontinuities are (technically) less of a problem, and something could be implemented so that multiple files get written to the filesystem whenever a discontinuity occurs, but since Streamlink's focus is on live streaming and not recording, a special case exception for this most likely won't get implemented.

The only exceptions implemented for HLS discontinuities are in terms of "ad-blockers" for live streams, so that a single "cohesive" stream can be output by Streamlink that doesn't crash the player's demuxer/renderer. See the Twitch plugin for example.

If your goal is to record an HLS stream with discontinuities, then Streamlink is not the right tool for you.

[ischmidt20]

Just want to chime in here, especially about this bit:

Streamlink's focus is on live streaming and not recording

I know this is the stated purpose of the software, but many people (myself included) use streamlink for its recording features. AFAIK, the only other software that can handle such a wide set of live streams is youtube-dl/yt-dlp, but I personally find its live recording functionality unintuitive and easy to screw up. Therefore, if I want to record live, streamlink is the preferred tool.

As a side note, I actually find that streamlink does handle HLS discontinuities quite well when sending the stream to a player, such as VLC. The only problem is that when casting from VLC, the stream stalls whenever a discontinuity is reached, but this is clearly an issue on the VLC side, not streamlink.

If your goal is to record an HLS stream with discontinuities, then Streamlink is not the right tool for you.

Again, I understand this sentiment by the dev team. However, I don't know of any tool that would be the "right" one for such a purpose, and I believe streamlink is the best home for such functionality out of all currently existing software.

About the original issue, a platform I commonly record from that has discontinuities still gets completely recorded with streamlink, just in a format that is difficult to work with. I wonder if OP's issue has something to do with the recurring segment names that cause them to be treated as "duplicates" and thus skipped. If so, it could be possible to implement a flag that would disable such behavior, which would allow OP's issue to be fixed without addressing discontinuities in general.

[bastimeyer]

This isn't about whether Streamlink is the preferred tool for you or other people when recording streams. This is about technical difficulties which can't be overcome because of how Streamlink works as a simple, standalone HLS client and how external players have to read its output, as explained.

Streamlink doesn't handle discontinuities "quite well", it doesn't handle them at all (because it can't), which leads to a broken output due to concatenating two or more completely different streams. Other applications then have to deal with that issue, be it a player, ffmpeg, an editing software, etc.

In case of the TwitchHLSStream implementation for example, which can filter out mid-roll ads in a live stream or VOD due to additional metadata embedded in the playlists, the result is a stream with totally messed up decoding and presentation timestamps in the following audio/video frames after the output continues/resumes. This works fine most of the time because these timestamps have a positive delta (compared to the same previous stream) and nonsensical frame references can be discarded by the demuxer. Other kinds of discontinuities don't work like this.

There is no special issue with the HLS playlist posted in the OP. If you add the #EXT-X-PLAYLIST-TYPE:VOD tag at the top (to make it start at the beginning) and #EXT-X-ENDLIST at the bottom (to indicate the end of the stream), you will see that every segment gets properly downloaded and concatenated in the output (discontinuities get ignored). This then leads to a broken output file and only the first part until the first discontinuity is a proper stream.

[ischmidt20]

This is about technical difficulties which can't be overcome because of how Streamlink works as a simple, standalone HLS client and how external players have to read its output, as explained.

I understand the limitations you described. My original comment was moreso referencing the proposed solution of having the output file be split upon each discontinuity tag, which would be immensely beneficial for many users, even though as you say, streamlink's stated primary purpose is streaming, not recording.

[Leatherface75]

Well it's same problem when just playing the stream and a lot of free TV streams are using this now.
Samsung Plus, Stirr, PlutoTV for a few examples.

[Leatherface75]

Well it's same problem when just playing the stream and a lot of free TV streams are using this now. Samsung Plus, Stirr, PlutoTV for a few examples.

Not sure how difficult it would be to integrate that into streamlink but using tsduck.io seems to work well with pluto/samsungtv streams

~ » tsp -I hls --live --highest-resolution https://i.mjh.nz/SamsungTVPlus/DEBD2700001ID.m3u8 \
            -P continuity --fix --no-replicate-duplicated -O file - | vlc -

That's because you're opening it with VLC that supports this.