FR: Auto-blurred 'frame' for media

[pbakaus]

Describe the new feature or change to an existing feature you'd like to see

Specifically for AMP Stories, but likely generally useful for all of AMP, I'd like to propose a feature that will allow you to blow up the 'box' of a media element to whatever size one desires, and have the background filled with a blurred version of it.

As potential API, I'd propose something like this:

<amp-media-backdrop blur="50px">
<amp-img src="..." width="..." height="..." layout="...">
</amp-media-backdrop>

<amp-media-backdrop blur="20px">
<amp-video src="..." width="..." height="..." layout="...">
</amp-media-backdrop>

amp-media-backdrop is essentially a supercharged amp-layout element, so it can be sized, scaled etc., but with the characteristic that AMP will always fill the box with a blown up, blurred version of the first media element it finds as child.

I would probably expose the following additional attributes:

• blur - controls the amount of Gaussian blur applied to the backdrop
• fit - subset of object-fit, can be set to either fill (default, stretches the backdrop to fill the container) or cover.

Additional context

This is what this looks like:

I've produced two CodePens for inspirational purposes (there are probably smarter ways to do this):

• Image version
• Video version

/cc @raovs @newmuis

[newmuis]

Just want to note that we've actually found this to have really bad performance using CSS filter.

We should explore what happens if we generate the image in JS, e.g. off-thread in a worker and/or server-side on the AMP cache.

@pbakaus was any consideration made for generating the blurred asset server-side on the origin?

[newmuis]

/cc @ampproject/wg-stories

[pbakaus]

Just want to note that we've actually found this to have really bad performance using CSS filter.

Can you quantify bad? I'm surprised it would be very noticeable for a single image at a time. Are we speaking > 100ms or significantly more?

We should explore what happens if we generate the image in JS, e.g. off-thread in a worker and/or server-side on the AMP cache.

Agree. Note that the video codepen uses a canvas to reproject the video, not sure if that's faster or not, but I'm guessing not..

@pbakaus was any consideration made for generating the blurred asset server-side on the origin?

We can certainly do that for images, but likely prohibitive for video. If we can get the performance to an acceptable state, it would be much more developer friendly when done on the client.

[newmuis]

Can you quantify bad? I'm surprised it would be very noticeable for a single image at a time. Are we speaking > 100ms or significantly more?

P0 bad. It caused videos to not play on some devices. My understanding is that it was in a "straw that broke the camel's back" kind of way; not necessarily that blurring images causes this, but that the format is already demanding, and this makes things worse.

See also:

• 🚮🖍🚀 Remove amp-story backgrounds #21289
• amp-story videos do not play or stutter on 3-panel desktop UI in Chrome 72 #21287
• https://twitter.com/cramforce/status/1105877648410984449

We can certainly do that for images, but likely prohibitive for video. If we can get the performance to an acceptable state, it would be much more developer friendly when done on the client.

I wonder whether it would be sufficient to blur the poster image, rather than the current frame of the video (from a product perspective, regardless of where in the tech stack it occurs)?

[gmajoulet]

Paul's feature is a bit different from what we had. I believe our performance issues were caused by (1) animating elements over a blurred image, and (2) blurring/cropping (border-radius) a video. Both were causing a lot of expensive repaints.
(I'm not fully sure about (1), it might just be the large blurred image, in which case there's nothing we could do)

Our problematic blurred backgrounds were story-level with pages animating/translating on top of it, or page-level with the bookend animating/translating on top of it.
If Paul's feature request is a static page-level background that doesn't ever need to be repainted (and not in the scope of any repaint), it might be worth building a prototype to check the performance.

[newmuis]

Yeah, makes sense to prototype, and honestly we can keep the API and change the internal implementation at any time. Just wanted to forewarn about the filter history here

[pbakaus]

I'll let you all make the call here - imho would be a pretty awesome feature if it can be done responsibly!

Re: only doing it for the poster image for videos: Afraid that that won't be enough for the desired effect, as the borders of a frame can change greatly through the course of a video.

[raovs]

Very much like the feature idea - thanks Paul! Sounds like we can spend a bit of time seeing if it can be done with good performance.

…

On Tue, Jul 7, 2020 at 10:33 AM Paul Bakaus ***@***.***> wrote: I'll let you all make the call here - imho would be a pretty awesome feature if it can be done responsibly! Re: only doing it for the poster image for videos: Afraid that that won't be enough for the desired effect, as the borders of a frame can change greatly through the course of a video. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#29168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE6JAFX4AREGKLK4PPPUPALR2NL5HANCNFSM4OS7NCOA> .

[mszylkowski]

My 3 mins of research: Seems like this can be done by reading the image once and generating a blurred version in JS using one of the many MIT licensed libraries for blurring (eg: StackBlur) or write our own blurring algorithm. Then for images it'd be easy to read the image and apply the blur, and for videos it looks like it could be done at 30fps but also we might want to run it at a lower fps if the device is not as fast.

[cramforce]

+1. Blurring into canvas and then just displaying the static canvas would be fast enough. But only if done for static images.

This could be done by generating an SVG with a filter and referencing the image in it similar to how we do blurry backgrounds. That would move the filter application into C++. See https://github.com/ampproject/amp-toolbox/blob/0c8755016ae825b11b63b98be83271fd14cc0486/packages/optimizer/lib/transformers/AddBlurryImagePlaceholders.js#L174 for reference.

[pbakaus]

Curious if something ever came out of this? I still think this would be a killer feature :)

[newmuis]

It's not something we have on our prioritized roadmap, but I think it will help keep quality high (e.g. no black bars when letterboxing) as we proliferate fixed aspect ratio stories.

/cc @gmajoulet @raovs

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.