Skip to content

Visual Segments: Blocking out social media icons and other intrusive visual content #538

@LoganDark

Description

@LoganDark

SponsorBlock is invaluable when a portion of a video is entirely unwanted. However, some creators prefer to sprinkle subscribe buttons and social media icons throughout their video, during otherwise unrelated speech or visual activity.

Trinkets like those prefer to collect in the most inconvenient places, in the middle of useful or engaging information. Viewers generally still want that information, so skipping the section completely would not be very helpful. Example of a video utilizing trinkets: the server advertisement is a dedicated section that can be cut out with a self-promotion segment, which I have already submitted. However, the twitter trinket, which is followed by an instagram trinket, occurs in the middle of speech/gameplay that should not just be cut out.

I would like to propose a new feature for SponsorBlock: visual segments. Visual segments are not to be skipped like other segments are, but they represent a time window for a particular trinket - defined as a piece of intrusive visual content, like a subscribe button or social media icon, present during an otherwise engaging or important part of the video.

Each visual segment would contain exactly one obstruction, which would be overlaid on top of the video content for the duration of the segment. Of course, it's not nearly as seamless as completely skipping an obnoxious section, and it's pretty obvious that there's something beneath the obstruction - but viewers won't have to see what's behind it if they don't want to, and they will still not miss out on an important section of the video.

Obstructions are solid shapes (consisting of a bounding rectangle and the name of the primitive to draw, such as 'box' or 'circle'). The submitter of the obstruction can define a solid color, which defaults to black, but that's it. Obstruction bounds would be defined using percentages.

Viewers can enable or disable obstructions just like annotations. Viewers can also, in the extension settings, select if they would like to overlay the defined obstruction color, or blur obstructed parts instead (achievable with backdrop-filter).

The only obstruction shapes to start would be box (square/rectangle) and circle (circle/ellipse). There's no need for fancy shapes like stars or arbitrary polygons or even rounded rectangles.

To define a visual segment, you would first select its type as 'visual segment', then you are given a basic plotting UI to define one (and only one) obstruction. It's probably up for debate how to plot multiple overlapping visual segments at once; you'd likely only do one at a time by selecting which one the plotter is using. It sounds like it would be rare to see multiple social media icons on different parts of the screen at the same time... but it's actually not so rare to see, for example, one icon in each corner, during outros and such.

Resolved problems:

  • Little trinket problem - Little subscribe buttons or social media icons can be blocked out

Unresolved problems:

  • Moderation - what's to stop someone from using these to draw out text? Or completely block out a video? The voting system, but someone will have to suffer and know how to downvote the segment. There is no neural network to determine if there is currently a social network icon on the screen or not. It's not even that simple because a trinket could just be text that says "remember to subscribe".
  • Large trinkets - Skeppy is known for this. They have a gigantic, elaborate subscribe animation that they paste smack dab in the center of their video content while doing other things. Obstructing this would be hugely detrimental because it would actually completely ruin visibility. Usually these sections can be cut out but not always. They can be pretty long and cut deeply into the backstory of the video.
  • Many aspects of the plotter - it is mostly undefined
  • What will this mean for SponsorBlock on non-Chrome platforms? YouTube Vanced?
  • If SponsorBlock should allow segments to block out a portion of the video but not the audio, what about vice versa? I don't think there should be audio blocking segments... but that is definitely a debate that might happen (also would be nice to block out obnoxious music during timelapses and such without having to manually operate the mute button - multiple categories of audio blocking? heck)
  • This extends SponsorBlock in a potentially non-backwards-compatible way and violates its current guarantee of only providing occasional skips (by adding visual obstructions to that as well)

Right now the questions largely outweigh the answers, so this is mostly a request for comment at the moment. I'd love to work out a more complete architecture for this in a way that most benefits the SponsorBlock project. I'd love for the extension developer(s) to weigh in on this and provide feedback, and I'd love to see a feature like this added to SponsorBlock eventually. Thank you for reading :D

Activity

ajayyy

ajayyy commented on Nov 9, 2020

@ajayyy
Owner

Semi-related: #96

LoganDark

LoganDark commented on Nov 9, 2020

@LoganDark
Author

Yeah, probably. This is a bit more fleshed out and dedicated to one feature. A good format for a GitHub issue, I'd imagine.

ajayyy

ajayyy commented on Nov 9, 2020

@ajayyy
Owner

Yep, just want to link previous discussion

LoganDark

LoganDark commented on Nov 9, 2020

@LoganDark
Author

Well, what do you think?

LoganDark

LoganDark commented on Nov 23, 2020

@LoganDark
Author

image

:(

ajayyy

ajayyy commented on Nov 24, 2020

@ajayyy
Owner

I definitely think this is a cool idea, but it may probably is a little out of scope right now. Maybe as a long term goal.

Now that SponsorBlock is old, it does have many people with high reputation, so "dangerous" features like this and #409 can be limited to only those with very high reputation

ajayyy

ajayyy commented on Nov 24, 2020

@ajayyy
Owner

To work best, they would probably have to be replaced with an image, not just a shape, but that might be too much work to be viable.

Some way to have the "annotations" move smoothly across the screen over time might be needed as well.

LoganDark

LoganDark commented on Nov 24, 2020

@LoganDark
Author

So far I've submitted 156 segments and saved others about 60 hours. Does that count as high reputation? "Very high reputation" worries me, I wouldn't set the bar too high. I've been submitting segments for about a month so far, on pretty much every video I watch.

To work best, they would probably have to be replaced with an image, not just a shape, but that might be too much work to be viable.

Instead of specifying a solid color, you could have the extension take a screenshot of that portion of the video, and display it for the duration of the visual segment. It's easily something that could happen in an update, rather than on first release.

Some way to have the "annotations" move smoothly across the screen over time might be needed as well.

Careful with this. I intentionally kept this as primitive as possible, to hopefully prevent revealing more than necessary about the underlying distraction.

This is one of the only extensions to the idea I would theoretically approve of (if I were the one calling the shots, which I'm not), but again like other aspects of the 'plotter' the way this would be presented to the editor is completely undefined. You could implement an entire keyframe system, or just have a start and end point and an animation curve (probably the way I would do it). Allow the curve to be edited manually, but provide a couple presets. This adds additional complexity, beware.

Of course, this obstruction system doesn't even touch things like sound effects... but it would be much more trouble than it's worth to try to hammer out a solution to somehow lessen that without affecting the surrounding audio in a negative way. It would also require SponsorBlock to start processing all audio which comes out of the player which is probably even more complicated than visual segments would be. Mute segments are mildly less complex than that, but still not optimal.

Finding the optimal solution is most likely not possible, and like the rest of SponsorBlock, a compromise needs to be found.

LoganDark

LoganDark commented on Nov 24, 2020

@LoganDark
Author

Keep in mind that once most of the concerns are worked out, since I'm a programmer myself I could look into implementing this all as a pull request. I just don't want to put work into a plotter that you don't like, or an animation system that you don't like.

ajayyy

ajayyy commented on Nov 24, 2020

@ajayyy
Owner

I agree with your comment about audio, I don't think that is something feasible, especially with just a browser extension.

ajayyy

ajayyy commented on Nov 25, 2020

@ajayyy
Owner

I feel like this may be useful getting the opinions of others on this. Should I make a channel/room on Discord/Matrix to discuss this?

ajayyy

ajayyy commented on Nov 25, 2020

@ajayyy
Owner

Also, on the example link you showed in the first post, I'm not sure just covering it with a screenshot of the video will help much.

LoganDark

LoganDark commented on Nov 25, 2020

@LoganDark
Author

I feel like this may be useful getting the opinions of others on this. Should I make a channel/room on Discord/Matrix to discuss this?

Sure.

Also, on the example link you showed in the first post, I'm not sure just covering it with a screenshot of the video will help much.

It also won't help much to ask users to invent their own image to overlay. Not to mention the abuse that could bring.

You'd also either have to store every image that they upload, or have their computer make requests to an arbitrary url (BAD BAD BAD BAD), or proxy it through your backend (less bad but still dangerous)

ajayyy

ajayyy commented on Nov 25, 2020

@ajayyy
Owner

You'd also either have to store every image that they upload, or have their computer make requests to an arbitrary url (BAD BAD BAD BAD), or proxy it through your backend (less bad but still dangerous)

While anonymous image upload is scary, I'm not sure if there is another viable way. We've already experienced severe ratelimiting just downloading captions for use with https://github.com/andrewzlee/NeuralBlock and had to shut it down until we find another solution.

I doubt we would be able to download videos on the server to be able to take screenshots server-side, they would have to be done client side.

Could you send an example of what you think the example video should look like to hide the obstructions?

LoganDark

LoganDark commented on Nov 25, 2020

@LoganDark
Author

While anonymous image upload is scary, I'm not sure if another viable way.

You could not have users provide custom images? That's kind of what I was hinting at in my comment.

I doubt we would be able to download videos on the server to be able to take screenshots server-side, they would have to be done client side.

That's not at all what I mean. The client is already playing the video, so when a visual segment is encountered they could just display a portion of the frame that it started on, on top of the video until the segment is over.

It's hard to explain over text, but let me try to show what I mean.

Control
Solid color
Persist

This is not a job for the server anyway. The server should only store data about the shape of the obstruction.

This could be what you were thinking of when you said "custom images". As a plus, the server does not have to do any extra work, and you don't have to think about image uploads. You do have to work out how to display that, though... possibly problems when you skip to the middle of a visual segment and the client does not have that frame in memory. You might just have to use black in that case.

19 remaining items

jac0b-w

jac0b-w commented on Apr 12, 2022

@jac0b-w

Sorry I'm not familiar with effect of a tunnel filter. Instead of sampling the pixels directly behind the blur filter like you would typically expect you sample the pixels around the edge of the bounding box to be blurred.

A very crude implementation could be as simple as this: say you have the bounding box of a trinket which is 100px vertically. You take a 50px vertically above and below the trinket then append the images together then blur that new image and apply it over the trinket.

Hope that makes the basic idea clear.

LoganDark

LoganDark commented on Apr 12, 2022

@LoganDark
Author

Sorry I'm not familiar with effect of a tunnel filter. Instead of sampling the pixels directly behind the blur filter like you would typically expect you sample the pixels around the edge of the bounding box to be blurred.

A very crude implementation could be as simple as this: say you have the bounding box of a trinket which is 100px vertically. You take a 50px vertically above and below the trinket then append the images together then blur that new image and apply it over the trinket.

Hope that makes the basic idea clear.

Photo Booth (Apple program) has a filter that basically only reveals a circle, and then stretches out the edges of the circle to fill the entire rest of the image. Reversed, that would be filling in a circle by stretching its edges inward.

jac0b-w

jac0b-w commented on Apr 12, 2022

@jac0b-w

Photo Booth (Apple program) has a filter that basically only reveals a circle, and then stretches out the edges of the circle to fill the entire rest of the image. Reversed, that would be filling in a circle by stretching its edges inward.

Yes this sounds right!

ajayyy

ajayyy commented on Jul 26, 2022

@ajayyy
Owner

A priority list to make the first implementation less over-complex:

Priorities: Focus on fixing transitions > covering logos/icons. Simpler problem with a more clear goal, can easily be extended to logo covering.

  • Full freeze frame segment (keep the frame from start to end)
  • Toggle to take the freeze frame from the start of last non skipped area (this would help fix transitions out of sponsors, keep the before sponsor frame)

  • Solid colour frame instead of freeze frame
  • Fade in and fade out toggle (for fade in + hold, two segments can be made, a solid colour fade in and solid colour non fade)

  • Rectangular select where the freeze frame should be

More out there stuff reserved for a later time:

  • Take frames from arbitrary points of the video
  • Animated shape overlays (as in, the shape slowly moves over the course of the segment)
  • Arbitrary shapes
  • Modifying frames using effects/blurs
  • Replacement images/video
mchangrh

mchangrh commented on Oct 20, 2022

@mchangrh
Contributor

I have a working demo with a copied 2D canvas in the discord

  • freeze frame
    • pull a certain frame natively and repeat that image on the canvas
    • add external overlay to combine to that canvas
  • solid colour freeze frame / rectangular select
    • just load external svg and draw it on the canvas (can also just draw 2d shapes)

latency

unfortunately there is latency associated with it, the refresh rate is tied to the draw interval. The lower the draw interval, the higher the memory/cpu usage, naturally. Ideally it would be synced to the video but when set up to integers, there can be desync up to 3 frames just because of frame timings. At best there is 1 process time of delay, which might be mitigated at running at 2x refresh rate

ajayyy

ajayyy commented on Oct 20, 2022

@ajayyy
Owner

I remember seeing a demo in one of the old links I posted where you can grab the video frame early to defeat the latency. double buffering

shodanx2

shodanx2 commented on Jan 3, 2025

@shodanx2

Hello,
I came here today to ask the option to draw a black square over the screen at an offending location for a certain duration (a banner ad)

This thread appears to be about much more than that, but that is what I would like to see, blacking out or blurring out the visual pollution that infects human art.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @basilevs@LoganDark@shodanx2@ajayyy@mchangrh

      Issue actions

        Visual Segments: Blocking out social media icons and other intrusive visual content · Issue #538 · ajayyy/SponsorBlock