[css-view-transitions-1] Exposing ink overflow rect bounds to script

[khushalsagar]

The steps for capturing an element's image during View Transitions involve painting the element and all its descendants in their entirety such that the produced image's bounds are equal to the element's ink overflow rectangle bounds. The spec text is here.

The ::view-transition-group pseudo-element which displays the image has bounds equal to the captured element's border-box. Since the image's natural size is equal to the element's ink overflow area but it needs to be drawn in an element which is smaller than that, we use object-view-box to ensure the correct subset paints into the element.

So for example if you have a 100x100 box with a shadow that paints 10px outside this box:

• The ink overflow rect is [-10, -10] 120X120.
• The natural size of the produced image will be 120x120.
• The ::view-transition-group for this will be sized to 100x100.
• The object-view-box set on this group will be inset (10px, 10px, 10px, 10px).

Given that script can read the object-view-box on the group, it allow authors to query the value for ink overflow rectangle computed by the UA.

There was an additional risk of fingerprinting if object-view-box was affected by where the UA decided to clip the element's painting. But with the resolution in #8561 that is not an issue. Clipping the element's painting is an internal detail, the natural size of the image and its object-view-box value is always as described above.

It came up during the discussion in #8561 that its odd that the computed ink overflow rectangle is being exposed to authors. Since there are cases where it is technically infinite and left undefined, see spec text here. So now authors will see a value for it based on the UA's implementation.

Questions:

• Is the above ok as-is? The value is exposed to script.
• If the value shouldn't be exposed to script, how do we set that up. We can't just elide it from getComputedStyle. An author could do object-view-box: unset and learn the value based on how it affects the size of the replaced element displaying this image.

@chrishtr @fantasai FYI.

[khushalsagar]

Offline discussion, it sounds like we need to define ink overflow area in the spec precisely for IntersectionObserver anyway. So exposing that here shouldn't be an issue. @dbaron FYI.

[khushalsagar]

The IntersectionObserver discussion touching upon this is here btw.

[flackr]

• If the value shouldn't be exposed to script, how do we set that up. We can't just elide it from getComputedStyle. An author could do object-view-box: unset and learn the value based on how it affects the size of the replaced element displaying this image.

If it shouldn't be exposed to script, I think we would keep it as some internal detail associated with the specific replaced element, and combine it with any object-view-box specified by the developer (e.g. similar to an ancestor / descendant transform). There would be no ability to unset the internal object-view-box.

[khushalsagar]

combine it with any object-view-box specified by the developer

This isn't an option since object-view-box depends on the natural size of the image. This value combined with the natural size gives us the intrinsic size used by layout/paint. So the author being able to specify it by design implies they know the image's natural size which depends on the overflow rect.

So hiding it implies authors are unable to read or set it. I don't have a strong opinion on whether they should be able to, would be good to get a developer opinion on this: @jakearchibald @mirisuzanne @ydaniv.

But if the only reason to hide it is that its currently undefined in the spec, then I err on just defining it given that other features like IntersectionObserver do need it to be defined.

[fantasai]

So the author being able to specify it by design implies they know the image's natural size which depends on the overflow rect.

I think what we do here is treat the natural size of the view transition image as being the scrollable overflow rect (which is layout-based and definite). Any ink overflow is scaled and positioned and transformed and painted with the rest of the image, but it's not measured.

[fantasai]

In general, I think we want to say that object-view-box operates on the defined bounds of an image; and if the image has stuff that paints outside it, that's painted outside the measured bounds if it's not clipped. For example, for an SVG image, we want to use the viewbox of the SVG image, but an SVG image has a theoretically infinite canvas, and there can be things drawn outside the viewbox; if we're not clipping to the viewbox rectangle, then you'll see those things. But we shouldn't be measuring them when trying to position the SVG inside a replaced element's content area.

[khushalsagar]

This sounds fine to me from an implementation perspective. Especially with the resolution we had at #8561, there is already precedent for the natural image size (as exposed to authors via object-view-box) to be different from what is actually rendered by the browser.

I'm curious though, are we going with the scrollable overflow rect because ink overflow is undefined? If another feature on the platform (like IntersectionObserver) required us to define the ink overflow extent, would we still use scrollable overflow?

We could just not expose object-view-box to authors at all. The natural image size can then be the ink overflow rect and the exact value will stay an internal UA detail. The motivation for exposing it is the assumption that authors would want to customize based on that data. So we should expose the value which authors would actually use, and maybe that is scrollable overflow rect. Just want to ensure that what we choose is based on what makes sense to authors, not based on ink overflow being undefined on the platform today.

[ydaniv]

@khushalsagar yes, what @fantasai wrote and your analogy to SVG is exactly what I had in mind:
The viewBox relates to vector elements that are part of the layout and can be measured.
The ink overflow here can be thought of as (and usually is) raster effects that can't be measured and aren't part of the layout, only the paint.
It's usually enough to be able to allow overflow, and not necessary to query it.

[fantasai]

@khushalsagar I don't actually know whether it would be easier for authors to work only with the base geometry of the box or also manipulate the scrollable overflow rectangle; I haven't played enough with view transitions to have a good sense of it. But I'm pretty sure the ink overflow rectangle isn't generally something they'd want to work with.

[css-meeting-bot]

The CSS Working Group just discussed [css-view-transitions-1] Exposing ink overflow rect bounds to script, and agreed to the following:

• RESOLVED: Snapshot raster is theoretically infinite. Snapshot has a "natural view box" of the snapshot element's border box (giving the natural size). Object-view-box can change this view box.

The full IRC log of that discussion

<TabAtkins> khush: We touched on defining the bounds of the element you capture at the f2f
<TabAtkins> khush: One aspect is the natural size of the image is the ink overflow rectangle.
<TabAtkins> khush: This can be larger than the border box, so you use object-view-box to size it
<TabAtkins> khush: This means the ink overflow rectangle is exposed to authors
<TabAtkins> khush: But this is currently undefined, what to do?
<TabAtkins> khush: Two options. One questions whether we expose this at all.
<TabAtkins> khush: We can censor it in getComputedStyle() and make it !important in UA styles here.
<TabAtkins> khush: This can be bad if authors want to customize the size
<TabAtkins> khush: I haven't seen good use-cases for customizing it tho.
<TabAtkins> khush: Suggestion from fantasai is to use the scrollable overflow rect instead
<TabAtkins> khush: So the natural size authors see is scrollable overflow, but the browser can actually draw a larger ink overflow
<TabAtkins> khush: I'm fine with either option from impl.
<TabAtkins> khush: If scrollable overflow makes more sense as an author-exposed concept, that's fine
<TabAtkins> khush: But if ink overflow would be useful we shoudln't shy away from using it
<TabAtkins> q+
<Rossen_> ack fantasai
<TabAtkins> fantasai: One, I don't think authors want to control layout of stuff using ink overflow rect. Kinda unpredictable.
<TabAtkins> fantasai: And not what they're actually measuring. It's excess stuff, you don't really notice it's there. Serifs on font, blur on box-shadow, etc
<TabAtkins> fantasai: So I don't think this is something authors will want to use.
<TabAtkins> ack TabAtkins
<florian> q+
<TabAtkins> fantasai: I think we either want the scrollable overflow or the box fragments, and allow image to overflow the rect in some way
<flackr> q+
<TabAtkins> fantasai: Even if we expose ink overflow in IntersectionObserve for security reasons, I don't think it's something people want to measure against
<TabAtkins> +1 to fantasai's points
<Rossen_> ack florian
<TabAtkins> florian: Might be confsued, but thought in the gneeral case the ink overflow was not only underspecified, it wasn't necessarily finite.
<TabAtkins> florian: With blurs it can technically be infinite.
<TabAtkins> fantasai: You're right, tho we'd have a proposal to define the bounds that it clips. Tehcnically what impls do today anyway.
<Rossen_> ack flackr
<fantasai> s/we'd have/there was
<TabAtkins> flackr: I think hiding the ink overflow rect is one of my proposals
<TabAtkins> flackr: But my proposal was more that if authors set an ink overflow, we'd combine it with intrinsic ink overflow from the image, and they'd always work wrt the border box of the item.
<TabAtkins> fantasai: Not sure I'm following, more concrete example?
<TabAtkins> flackr: Say you ahve 100x100 box, there's 50px of shadow around it
<TabAtkins> flackr: The thing presented to the dev is no ink overflow.
<TabAtkins> flackr: But when we calculate the ink overflow, we'd add the 50px to whatever we got from the author's ink overflow rect
<TabAtkins> Rossen_: what do you mean by "presented to the dev"
<TabAtkins> flackr: The value of object-overflow won't expose that shadow size.
<Rossen_> q?
<TabAtkins> khush: I'm also having toruble understanding this - is this similar to what fantasai is suggesting, where you see the scorllable overflow as the exposed size?
<TabAtkins> flackr: Basically the same, probably just an impl detail about how it works.
<Rossen_> ack fantasai
<chrishtr> q+
<khush> q+
<TabAtkins> fantasai: I wasn't sure if the box we shoudl use is scrollable overflow or the border box, just pretty sure it shouldn't be the ink overflow rect
<TabAtkins> fantasai: If we have the property controlling the clipping, we could have an "auto" value that computes to itself and works from the ink overflow
<Rossen_> ack chrishtr
<TabAtkins> chrishtr: I think what elika and rob are saying makes sense
<TabAtkins> chrishtr: But khushal, is there any case you've seen where authors need to know the ink overflow to position correctly?
<TabAtkins> khush: No, it is handled automatically.
<TabAtkins> khush: But if authors can set the object-view-box, they get information about the natural size.
<TabAtkins> khush: Like if you say "none", then measure the box, you'll get the size out.
<TabAtkins> khush: So I'm fine with saying the natural size is the border box or scrollable overflow, and what authors see is based on that size.
<Rossen_> ack khush
<TabAtkins> khush: From what i've heard so far, I think I like the proposal of scrollable overflow. If we decide not to expose it and there's a use-case later, it'll be hard to change.
<TabAtkins> khush: So inclined to exposing it, using the scrollable overflow.
<TabAtkins> +1
<Rossen_> ack fantasai
<TabAtkins> fantasai: So we have a replaced element. It contains an image which is snapshot.
<flackr> q+
<TabAtkins> fantasai: The size of that replaced element is the size of the snapshot element's border box.
<TabAtkins> fantasai: But like an SVG image, that extends past its viewbox.
<TabAtkins> fantasai: and we're setting a property that cause th estuff outside the stuff to actually paint
<fantasai> s/outside the stuff/outside its declared size/
<fantasai> s/declared/nominal
<TabAtkins> khush: Yes, using object-view-box to make the sizing align, and then anything outside paints as ink overflow, is clipped based on 'overflow'.
<TabAtkins> fantasai: So on an SVG currently, say I draw a bunch of circles then set viewbox to 50x50
<TabAtkins> fantasai: I'll get a 50x50 image, stuff can get drawn outside of that. If I put this in an <img> they'll get clipped by default.
<TabAtkins> fantasai: But they're still there - if I use object-position we're moving that 50x50 square.
<TabAtkins> fantasai: object-view-box refers to that viewbox declaration - I could change it and it would change what portion of the SVG is "the displayed image"
<TabAtkins> fantasai: So we're creating a snapshot whose viewbox is the border box of the snapshot element
<TabAtkins> fantasai: I think we can conceptually say the top left corner of the border box is the 0 coord for the image, for th epurpose of object-view-box?
<TabAtkins> fantasai: Then I think we ahve an analogous situation
<TabAtkins> [i don't quite understand this]
<flackr> +1 that is how i was thinking object-view-box modifications should work
<TabAtkins> fantasai: I think this gets us a mental model and an API consistent with SVG
<TabAtkins> khush: So you're right about the viewbox descriptions.
<TabAtkins> khush: But the origin is based on the natural size
<TabAtkins> khush: object-view-box says to inset from those natural bounds
<TabAtkins> khush: We have an option to say what amount of overflow is exposed to authors
<Rossen_> ack flackr
<TabAtkins> flackr: I agree with elika's thinking about how this should work, that matches what i'm imagining
<TabAtkins> flackr: the ink overflow isn't dev-exposed, just part of the object we've captured
<TabAtkins> flackr: so 0,0 is at the border box corner
<TabAtkins> flackr: Do we capture scrollable overflow even if the source element clipped it?
<TabAtkins> flackr: Is it only for overflow:visible we capture?
<TabAtkins> fantasai: No, we capture what you actually see.
<khush> q+
<TabAtkins> fantasai: I want to change what I'm proposing from what was in th eissue
<TabAtkins> q=
<TabAtkins> q+
<TabAtkins> fantasai: The natural image should be the full size of stuff we see.
<TabAtkins> fantasai: But the origin of the image should be the border box corner.
<TabAtkins> fantasai: Similar to SVG with its viewport
<TabAtkins> fantasai: In SVG you can draw into negative coords
<TabAtkins> fantasai: So I'm proposing the origin of the image is the border box corner. Anything that overflows that is captured for the snapshot still gets captured.
<flackr> +1
<TabAtkins> fantasai: So we create a raster image whose origin is not the top-left corner of the image
<TabAtkins> fantasai: So that lets us say the natural size is the border box. If they want to change that, they can do that (so long as it's captured)
<TabAtkins> fantasai: and if they extend past what's captured they get transparent pixels
<TabAtkins> fantasai: so we don't need to define the boundary of the raster, just that it captures at least the visible scrollable overflow
<TabAtkins> fantasai: everything outside is on an infinite transparent canvas
<TabAtkins> fantasai: I think that gets us a model where author can do whatever they want to do
<Rossen_> ack fantasai
<TabAtkins> khush: +1 elika's suggestion
<TabAtkins> khush: It's funny we added object-viewbox because what is proposed now is what we thought about implementing internally, so we thought a cSS property is a better way to do it
<TabAtkins> khush: So it sounds now we conceptually do what object-view-box is doing but expose it to the author
<TabAtkins> khush: So +1
<fantasai> TabAtkins: basically what khush said
<Rossen_> ack TabAtkins
<Rossen_> ack khush
<fantasai> TabAtkins: original purpose of ovb was to have natural size image and cutout the border box part
<fantasai> TabAtkins: but if we're dealing withi ink ...
<fantasai> TabAtkins: So default case doesn't need 'object-view-box' at all, its natural viewbox matches border-box, sounds find to me
<fantasai> TabAtkins: requires minor spec edits, but good change overal
<fantasai> TabAtkins: and if you want the image to look larger, then you would use negative coords
<fantasai> TabAtkins: rather than smaller positive coords
<fantasai> TabAtkins: so sounds fine, good overall
<TabAtkins> Rossen_: Can you summarize resolution?
<TabAtkins> fantasai: proposed resolution is snapshot rasterization is theoretically infinite. Origin corresponds to top-left of border box of snapshotted element, natural size is width+height of border box. Author can use object-view-box to shift this "natural view box"
<flackr> +1
<ydaniv> +1
<TabAtkins> Rossen_: comments or objections?
<TabAtkins> RESOLVED: Snapshot raster is theoretically infinite. Snapshot has a "natural view box" of the snapshot element's border box (giving the natural size). Object-view-box can change this view box.
<chrishtr> https://github.com//issues/8637

[noamr]

Sorry but I'm missing how this resolution would work in practice. The SVG analogy doesn't hold (or I'm misunderstanding part of it). Let me explain and correct me if I got some of this wrong (apologies, I'm new to this spec).

Let's say we're transitioning a 100x100 element that's 10px blurry from all sides. This would snapshot a 120x120 image, that according to this resolution would appear to the web developer as a 100x100 (natural size) image. Let's say that the developer changes the object-view-box of the pseudo-element to be the top left quarter (0,0 50x50). What happens to the ink overflow/the blurry bits?

Given that * is the blurry part and 0 is the content, which one is it?

***
*00

or

****
*00*
****

?

The former doesn't seem like what the developer intended (though perhaps that's what this resolution is saying), and the latter would require keeping the filter information as part of the transition so that we could re-apply the blur, which seems like a big change.

In SVG this is not a problem since the ink-overflow bit can be derived from the SVG and CSS. But here all we have is a previously snapshotted image.

If we're saying that changing the object-view-box of the pseudo-element can crop the ink-overflow sides if we don't reach the edges, I'm fine with it, but we should probably clarify in the spec.

[noamr]

Ok I think I found my mistake, ink overflow is used at time of capture rather than at time of crop. I can work with that, ignore my previous comment