plugins.twitch: mid-roll ads ("Commercial Break in Progress") filtering issue with --twitch-disable-ads #4934
Comments
This comment was marked as outdated.
This comment was marked as outdated.
bastimeyer
added
duplicate
|
Unless you can show a debug log where ads filtering doesn't work, this thread will stay closed. Also read this first: And btw, #4106 is about a bug in regards to the Twitch plugin's low latency streaming when a mid-roll ad occurs, as you can read in the thread's last comment. Everything else is irrelevant to this thread and it got locked because people didn't read the thread and started off-topic comments. |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
I can confirm that I am also getting 'commercial in progress' when the streamer run ads manually mid-stream in order to avoid pre-rolls on stream. debug output for when the ads are run: |
This comment was marked as outdated.
This comment was marked as outdated.
This is exactly the bug as described in #4106 because of the segment prefetching. Read the last comment in that thread. This requires a rewrite of the low latency implementation which would add an additional delay of two seconds, as mentioned and explained there. This thread here is therefore a duplicate. |
Nope, same happens without --twitch-low-latency flag. I just tested it. |
|
I figured out using the config file to set twitch-disable-ads rather than in Chatterino, and it kills the new preroll, but the stream still freezes on streamer-run ad breaks until you either force-close or the break ends. I did not use the low latency option. Edit: when it skips prerolls, it also opens pretty slow, making me wonder if it isnt just frozen the same way the midroll breaks are. |
|
From my testing without Currently working browser addons do the same with pausing playback until ads end, or replacing the stream with a 480p ad-free stream by proxy since certain regions don't get particular ads (which seems out of streamlink's scope?). It's likely due to this update below with an internal twitch change. |
Skipping ads is an entirely different thing which we had also implemented several times in the past, and all of those implementations/workarounds had to be restored at some point because they caused other issues later on. This is done by applying certain HTTP headers or query string parameters to the API request / Twitch's Usher service when acquiring a streaming access token for the HLS playlist. See the
Prefetch segments get parsed regardless whether the option is set or not. The way prefetch segment handling is implemented could still cause issues with how the ads filtering works. However, it's also possible that Twitch annotates mid-roll ads differently, which incorrectly causes the last segment to be included in the output, which we don't want. The ads-filtering implementation works like this: In addition to that, segments each have a duration and optional title attribute. The segment titles are currently not being used when filtering out ads, even though they seem to contain specific names which are unlikely to be used by regular segments. For example, pre-roll ad segments at least contain the "Amazon" string in their title, which could be used as an additional indicator in addition to the date range checks. If there's something wrong with the date ranges of mid-roll ads, then this could be used as a fallback. However, I don't know the contents of HLS playlists yet which contain mid-roll ads, so I could use some help. Please apply the following diff and help debugging this, as we need to know the exact HLS playlist contents of the moment when a mid-roll ad gets embedded. This adds diff --git a/src/streamlink/stream/hls_playlist.py b/src/streamlink/stream/hls_playlist.py
index 9f193d80..633bec54 100644
--- a/src/streamlink/stream/hls_playlist.py
+++ b/src/streamlink/stream/hls_playlist.py
@@ -550,6 +550,7 @@ class M3U8Parser:
parse_line = self.parse_line
for line in lines:
+ log.trace(line) # type: ignore[attr-defined]
parse_line(line)
# Associate Media entries with each Playlist$ streamlink \
--loglevel=trace \
--logfile=/path/to/tracelog.txt \
twitch.tv/CHANNEL-WHICH-WILL-MOST-LIKELY-RUN-MIDROLL-ADS \
bestMake sure to remove the existing log file before running Streamlink, or use a log file path with a dynamic name, like a timestamp. Once you've found mid-roll ads, please upload the log file in its entirety to github gist and post the link. Be careful though, there might be personal information included in the HLS playlist contents. I'm going to reopen this thread now and change its title. PLEASE DO NOT COMMENT UNLESS YOU HAVE SOMETHING USEFUL TO SHARE WHICH WILL HELP FIXING THE ADS FILTERING (NOT SKIPPING/PREVENTING/ETC). |
bastimeyer
changed the title
|
So I just had a mid-roll ad, and every single ad segment was properly filtered out by Streamlink. Prefetch segments were not included in the HLS playlist though, which means that prefetch segments are very likely the issue here, as described in #4106. The only thing stream discontinuities cause, even if it's a discontinuity in the same stream data and not just between the streams and an ad, are gaps in the program timestamps of the MPEG-TS bitstream, so players will handle this differently. MPV for example shows incorrect timings in the seek bar. Other players might still crash, eg. VLC, but if that's the case, then there is literally nothing that Streamlink can do, and users will need to use a different player. Here's the playlist from the time when the first ad segments appeared And here's the continuation of the regular stream: |
|
Quick correction of my previous post. Prefetch segments were involved in the playlist request prior to the start of the filtering, and ad date ranges for that are included: |
|
Found a stream where the mid-roll ads were just included in the second playlist refresh, and the ad segment was not filtered out correctly and the "purple screen" was incorrectly included in the output. |
|
I just had a, I think 3 minute long, |
|
Thanks. I think I already have enough data with stuff mentioned in my last post. All I needed was a case where the filtering fails. My first run from the previous posts had ads filtered out correctly despite the prefetch segments, but my last one did not, so this should be enough for figuring out the issue. I will need to see how the cloning of the last regular segment affects the dateranges included for the prefetch data, where the ads already start. AFAIA this is not taken into consideration in the ad filtering implementation. I had already mentioned this in the last post of #4106. You can keep running the trace logs, but it will most likely not be necessary anymore. Once I need new/more data, I will post an update. |
|
I have pushed a branch to my fork repo which you can test: pip install -U git+https://github.com/bastimeyer/streamlink.git@plugins/twitch/fix-prefetch-adsI won't be submitting a PR for this just yet, because I have not fully tested it, so this could also be complete garbage. Let's see... The added test you can see in the diff which is built from the data I have gathered earlier today however fails without the modification of the custom A bit more as a clarification on why I think mid-roll ads didn't get filtered out correctly: Once I'm confident with my changes, I will open a PR for proper review. The branch is currently based on a different commit which I have opened a separate PR for, and this will need to be changed anyway. We also had already scheduled and then delayed the 5.1.0 release, which means a fix will definitely be included before we publish the next release. Please test. Finding mid-roll ads is a bit tedious. And as already explained, this does only filter out ads, and even if it works properly without the "purple screen", there will still be a stream discontinuity, which means bad players like old versions of VLC might still crash. This is a known issue which we can't do anything about. As always, use recent software versions or better, switch to a different player like MPV if this keeps happening on recent versions of VLC as well. Enough with off-topic stuff now, though. edit: |
|
Hi, i guess my issue is somehow related to this one. I've been using streamlink to fetch short (~8s) fragments of a stream, but today at midnight something happened and it broke functionality. It seems that twitch injected new preroll ad (it's not even an ad actually, its splash screen saying "Preparing your stream...") like one @donnydark0 posted. |
|
@Urbinholt
As a general notice: |
|
I've been getting this issue too, and I thought I'd point out easy twitch channels to test the ad-freezes on. Channels that stream Tarkov often manually play ads during the load-in phase before a game to prevent mid-rolls from interrupting the game. A big streamer called /WillerZ does this very reliably and is live right now. After 5 raids, the MPC-HC player froze when he manually ran an ad 100% of the time . Thanks for working on this bug, and sorry if this isn't helpful. |
I've not been getting any mid-roll ads for the past one and a half hours or more. Not sure why. I've updated the branch that I've pushed earlier with the trace logging diff posted 8 hours ago, so it doesn't have to be applied manually. If you want to help getting a trace log output which can be used to diagnose the resuming issue (and potentially other issues as well which I didn't catch), then please follow this after installing Streamlink from my dev branch as mentioned above, thanks: edit: |
|
Log from a run where the segment didn't appear, while the player paused for as long as it was supposed to be running. |
What do you mean with this? I'm seeing after it encountered live prefetch segments again after the ad break which started at so everything looks like it's okay. If you take a look at the playlist reload at this time where it resumes and compare the segment/prefetch URLs with the contents of the playlist before and after, this is all fine. |
|
@bastimeyer Yeah, I only just reflected that my phrasing may have been ambiguous, as it seemed to be working fine™, based on last time this has become a developing topic. |
|
@bastimeyer I tested your pull request, it doesn't work, it still show the ads placeholder (I will enable the debug flag now because i forgot that) |
|
@Fijxu Post a trace log with the patch applied. As said, the only issue I will have to look into later today is that it doesn't stop filtering in some cases. If you're seeing what you have posted (and I have removed because it's unnecessary spam ITT), then you either have not set the |
|
Since I didn't post this yesterday, a PR has been opened for anyone to test and submit feedback: |
|
I'd like to ask for some more feedback. As said, I am not 100% confident with the current state of my fixes which I have submitted in PR #4942, because I had two ad blocks over the past 24h now which caused the filtering to keep running, effectively pausing the stream forever. Other mid-roll ads were all filtered out fine, dozens of them. Unfortunately, I didn't capture any logs during these two bad tests, so I don't have any data for that and can't reproduce the issue. I have once again updated the branch I've commented about earlier (not the PR branch). The data I'm looking for is a full trace log without any truncation where mid-roll ads occurred and where the stream did not resume playing after the ads ended. Ad blocks can take several minutes now, so patience is needed. The branch doesn't require any additional changes. Once again, in order to install from my dev branch, use pip. The branch is this, so basically the commits of the PR and another one for temporary trace logging support: pip install -U git+https://github.com/bastimeyer/streamlink.git@plugins/twitch/fix-prefetch-adsThen run rm -f tracelog.txt
streamlink --loglevel=trace --logfile=tracelog.txt --twitch-disable-ads twitch.tv/CHANNEL bestIf you want to see the log while the stream is running, run Please post and upload the compressed log here without removing any parts, especially not the log header, so that I know that you were actually running the right thing with the right arguments. If the log is too large and mid-rolls didn't happen for a long time, then it's probably better to use a different log and try again. As said, I'm only looking for those cases where mid-roll ads caused the filtering to not end, even after 10 minutes or so. I need to know which data was included in the HLS playlist which caused this behavior. Otherwise, please follow #4942 for any updates. Thanks. |
|
Ok, i tested your branch, i still get mid-roll ads |
|
@Fijxu You are not running the version from my branch, as you can see in your log file. I've already told you this in my other comment that this is very likely the reason. Now you know why.
|
|
Ok, i am dumb, it was using my preinstalled streamlink executable. For some reason it shows this version of streamlink Now it shows the traces of the streams, my bad, srry :p Also i can talk to you in matrix if you want to test more things in real time |
This is not the case according to your logs (
That's because my fork doesn't have the latest tags pushed and the version gets calculated differently by |
Yes, just for some minutes and then it resumes the stream after the mid-roll ad as should be, right? |
|
It would be helpful if streamlink mentioned (on the console) a link to directions for using the OAuth token from a signed in browser cookie. I wasn't even aware it has regained the ability to utilize Twitch credentials again until this issue happened. The OAuth Twitch subscription benefits / authentication directions mentioned in #4949 are a good overview and would be great on some landing page at a permanent URL. |
|
As I said, it would be helpful if how to setup ad-skipping were linked from the console messages, when an ad happened. Others may be like I had been until this bug thread and not realize there had been a change in functionality; particularly when it hadn't mattered that much since ads were being skipped before anyway. |
|
@bastimeyer @Fijxu I've noticed similar behavior specifically in cases where a streamer slaps an ad roll as they're winding down the stream. Would it be possible that the roll itself extends beyond the remaining length of the actual stream? I faintly remember seeing (and promptly discarding without much evaluation) a log where Streamlink keeps trying to fetch segments from a stream I've left on overnight, long after it has ended. |
This is issue #2198 due to the missing However, it is possible that if the output is paused it is interfering with the timeout, because the timeout gets ignored during pauses. So if a stream goes offline during a pause, this is an issue. I didn't run into this specific issue though. But thanks for the hint, this needs to be fixed at some point. The solution I proposed in #2198 (comment) should be good enough.
No, that would be way too much and unnecessary noise. And storing cached message checks for only displaying a message once in a while is also not a great idea. We can improve the notes of the plugin file though which is shown in the plugins list, and we can also improve the description of the |
|
Just to provide the context for the mentioned event: I've increased that timeout to 180 seconds, but the stream (left running, mind you), at the point I've realized it has ended, had been off for hours. Thanks for clarifying that particular case, tho. |
|
Just merged #4942, so that the main issue is fixed. The next nightly builds for users of Windows will be available in a couple of hours. Otherwise, install from the master branch via pip: Streamlink 5.1.0 will be published soon. I will also delete my secondary dev branch now, as it's not needed anymore. As said in #4942, I haven't been able to find a mid-roll ad block again where the |
Checklist
Streamlink version
Latest stable release
Description
Basically the same as this issue, but its marked as resolved, so I can't comment to say its back, apologies 🙏
#4106
Debug log
The text was updated successfully, but these errors were encountered: