-
-
Notifications
You must be signed in to change notification settings - Fork 381
Description
SponsorBlock is invaluable when a portion of a video is entirely unwanted. However, some creators prefer to sprinkle subscribe buttons and social media icons throughout their video, during otherwise unrelated speech or visual activity.
Trinkets like those prefer to collect in the most inconvenient places, in the middle of useful or engaging information. Viewers generally still want that information, so skipping the section completely would not be very helpful. Example of a video utilizing trinkets: the server advertisement is a dedicated section that can be cut out with a self-promotion segment, which I have already submitted. However, the twitter trinket, which is followed by an instagram trinket, occurs in the middle of speech/gameplay that should not just be cut out.
I would like to propose a new feature for SponsorBlock: visual segments. Visual segments are not to be skipped like other segments are, but they represent a time window for a particular trinket - defined as a piece of intrusive visual content, like a subscribe button or social media icon, present during an otherwise engaging or important part of the video.
Each visual segment would contain exactly one obstruction, which would be overlaid on top of the video content for the duration of the segment. Of course, it's not nearly as seamless as completely skipping an obnoxious section, and it's pretty obvious that there's something beneath the obstruction - but viewers won't have to see what's behind it if they don't want to, and they will still not miss out on an important section of the video.
Obstructions are solid shapes (consisting of a bounding rectangle and the name of the primitive to draw, such as 'box' or 'circle'). The submitter of the obstruction can define a solid color, which defaults to black, but that's it. Obstruction bounds would be defined using percentages.
Viewers can enable or disable obstructions just like annotations. Viewers can also, in the extension settings, select if they would like to overlay the defined obstruction color, or blur obstructed parts instead (achievable with backdrop-filter).
The only obstruction shapes to start would be box
(square/rectangle) and circle
(circle/ellipse). There's no need for fancy shapes like stars or arbitrary polygons or even rounded rectangles.
To define a visual segment, you would first select its type as 'visual segment', then you are given a basic plotting UI to define one (and only one) obstruction. It's probably up for debate how to plot multiple overlapping visual segments at once; you'd likely only do one at a time by selecting which one the plotter is using. It sounds like it would be rare to see multiple social media icons on different parts of the screen at the same time... but it's actually not so rare to see, for example, one icon in each corner, during outros and such.
Resolved problems:
- Little trinket problem - Little subscribe buttons or social media icons can be blocked out
Unresolved problems:
- Moderation - what's to stop someone from using these to draw out text? Or completely block out a video? The voting system, but someone will have to suffer and know how to downvote the segment. There is no neural network to determine if there is currently a social network icon on the screen or not. It's not even that simple because a trinket could just be text that says "remember to subscribe".
- Large trinkets - Skeppy is known for this. They have a gigantic, elaborate subscribe animation that they paste smack dab in the center of their video content while doing other things. Obstructing this would be hugely detrimental because it would actually completely ruin visibility. Usually these sections can be cut out but not always. They can be pretty long and cut deeply into the backstory of the video.
- Many aspects of the plotter - it is mostly undefined
- What will this mean for SponsorBlock on non-Chrome platforms? YouTube Vanced?
- If SponsorBlock should allow segments to block out a portion of the video but not the audio, what about vice versa? I don't think there should be audio blocking segments... but that is definitely a debate that might happen (also would be nice to block out obnoxious music during timelapses and such without having to manually operate the mute button - multiple categories of audio blocking? heck)
- This extends SponsorBlock in a potentially non-backwards-compatible way and violates its current guarantee of only providing occasional skips (by adding visual obstructions to that as well)
Right now the questions largely outweigh the answers, so this is mostly a request for comment at the moment. I'd love to work out a more complete architecture for this in a way that most benefits the SponsorBlock project. I'd love for the extension developer(s) to weigh in on this and provide feedback, and I'd love to see a feature like this added to SponsorBlock eventually. Thank you for reading :D
Activity
ajayyy commentedon Nov 9, 2020
Semi-related: #96
LoganDark commentedon Nov 9, 2020
Yeah, probably. This is a bit more fleshed out and dedicated to one feature. A good format for a GitHub issue, I'd imagine.
ajayyy commentedon Nov 9, 2020
Yep, just want to link previous discussion
LoganDark commentedon Nov 9, 2020
Well, what do you think?
LoganDark commentedon Nov 23, 2020
:(
ajayyy commentedon Nov 24, 2020
I definitely think this is a cool idea, but it may probably is a little out of scope right now. Maybe as a long term goal.
Now that SponsorBlock is old, it does have many people with high reputation, so "dangerous" features like this and #409 can be limited to only those with very high reputation
ajayyy commentedon Nov 24, 2020
To work best, they would probably have to be replaced with an image, not just a shape, but that might be too much work to be viable.
Some way to have the "annotations" move smoothly across the screen over time might be needed as well.
LoganDark commentedon Nov 24, 2020
So far I've submitted 156 segments and saved others about 60 hours. Does that count as high reputation? "Very high reputation" worries me, I wouldn't set the bar too high. I've been submitting segments for about a month so far, on pretty much every video I watch.
Instead of specifying a solid color, you could have the extension take a screenshot of that portion of the video, and display it for the duration of the visual segment. It's easily something that could happen in an update, rather than on first release.
Careful with this. I intentionally kept this as primitive as possible, to hopefully prevent revealing more than necessary about the underlying distraction.
This is one of the only extensions to the idea I would theoretically approve of (if I were the one calling the shots, which I'm not), but again like other aspects of the 'plotter' the way this would be presented to the editor is completely undefined. You could implement an entire keyframe system, or just have a start and end point and an animation curve (probably the way I would do it). Allow the curve to be edited manually, but provide a couple presets. This adds additional complexity, beware.
Of course, this obstruction system doesn't even touch things like sound effects... but it would be much more trouble than it's worth to try to hammer out a solution to somehow lessen that without affecting the surrounding audio in a negative way. It would also require SponsorBlock to start processing all audio which comes out of the player which is probably even more complicated than visual segments would be. Mute segments are mildly less complex than that, but still not optimal.
Finding the optimal solution is most likely not possible, and like the rest of SponsorBlock, a compromise needs to be found.
LoganDark commentedon Nov 24, 2020
Keep in mind that once most of the concerns are worked out, since I'm a programmer myself I could look into implementing this all as a pull request. I just don't want to put work into a plotter that you don't like, or an animation system that you don't like.
ajayyy commentedon Nov 24, 2020
I agree with your comment about audio, I don't think that is something feasible, especially with just a browser extension.
ajayyy commentedon Nov 25, 2020
I feel like this may be useful getting the opinions of others on this. Should I make a channel/room on Discord/Matrix to discuss this?
ajayyy commentedon Nov 25, 2020
Also, on the example link you showed in the first post, I'm not sure just covering it with a screenshot of the video will help much.
LoganDark commentedon Nov 25, 2020
Sure.
It also won't help much to ask users to invent their own image to overlay. Not to mention the abuse that could bring.
You'd also either have to store every image that they upload, or have their computer make requests to an arbitrary url (BAD BAD BAD BAD), or proxy it through your backend (less bad but still dangerous)
ajayyy commentedon Nov 25, 2020
While anonymous image upload is scary, I'm not sure if there is another viable way. We've already experienced severe ratelimiting just downloading captions for use with https://github.com/andrewzlee/NeuralBlock and had to shut it down until we find another solution.
I doubt we would be able to download videos on the server to be able to take screenshots server-side, they would have to be done client side.
Could you send an example of what you think the example video should look like to hide the obstructions?
LoganDark commentedon Nov 25, 2020
You could not have users provide custom images? That's kind of what I was hinting at in my comment.
That's not at all what I mean. The client is already playing the video, so when a visual segment is encountered they could just display a portion of the frame that it started on, on top of the video until the segment is over.
It's hard to explain over text, but let me try to show what I mean.
Control
Solid color
Persist
This is not a job for the server anyway. The server should only store data about the shape of the obstruction.
This could be what you were thinking of when you said "custom images". As a plus, the server does not have to do any extra work, and you don't have to think about image uploads. You do have to work out how to display that, though... possibly problems when you skip to the middle of a visual segment and the client does not have that frame in memory. You might just have to use black in that case.
19 remaining items
jac0b-w commentedon Apr 12, 2022
Sorry I'm not familiar with effect of a tunnel filter. Instead of sampling the pixels directly behind the blur filter like you would typically expect you sample the pixels around the edge of the bounding box to be blurred.
A very crude implementation could be as simple as this: say you have the bounding box of a trinket which is 100px vertically. You take a 50px vertically above and below the trinket then append the images together then blur that new image and apply it over the trinket.
Hope that makes the basic idea clear.
LoganDark commentedon Apr 12, 2022
Photo Booth (Apple program) has a filter that basically only reveals a circle, and then stretches out the edges of the circle to fill the entire rest of the image. Reversed, that would be filling in a circle by stretching its edges inward.
jac0b-w commentedon Apr 12, 2022
Yes this sounds right!
ajayyy commentedon Jul 26, 2022
A priority list to make the first implementation less over-complex:
Priorities: Focus on fixing transitions > covering logos/icons. Simpler problem with a more clear goal, can easily be extended to logo covering.
More out there stuff reserved for a later time:
mchangrh commentedon Oct 20, 2022
I have a working demo with a copied 2D canvas in the discord
latency
unfortunately there is latency associated with it, the refresh rate is tied to the draw interval. The lower the draw interval, the higher the memory/cpu usage, naturally. Ideally it would be synced to the video but when set up to integers, there can be desync up to 3 frames just because of frame timings. At best there is 1 process time of delay, which might be mitigated at running at 2x refresh rate
ajayyy commentedon Oct 20, 2022
I remember seeing a demo in one of the old links I posted where you can grab the video frame early to defeat the latency. double buffering
shodanx2 commentedon Jan 3, 2025
Hello,
I came here today to ask the option to draw a black square over the screen at an offending location for a certain duration (a banner ad)
This thread appears to be about much more than that, but that is what I would like to see, blacking out or blurring out the visual pollution that infects human art.