[css-conditional] [css-contain] `srcset` and `sizes` interaction with container queries

[una]

With container queries, the way developers write responsive styles changes from querying the global page for information to querying for a parent with containment for information. This shifts responsive paradigms. One case I can forsee is the need to query the parent for sizing and placement of its children. This includes srcset and sizes for responsive images

See also, the need for cw and ch units: #5888

Would the following be able to work with container queries instead of global queries? (ignore syntax, just trying to show the idea):

<img srcset="elva-fairy-480w.jpg 480w,
             elva-fairy-800w.jpg 800w"
     sizes="container(max-width: 600px) 480px,
            800px"
     src="elva-fairy-800w.jpg"
     alt="Elva dressed as a fairy">

<picture>
  <source container="(max-width: 799px)" srcset="elva-480w-close-portrait.jpg">
  <source container="(min-width: 800px)" srcset="elva-800w.jpg">
  <img src="elva-800w.jpg" alt="Chris standing up holding his daughter Elva">
</picture>

[chrishtr]

@yoavweiss @cramforce WDYT?

[chrishtr]

@mirisuzanne

[cramforce]

@chrishtr The use case I was mentioning earlier basically the opposite. Containers often determine the width of their child. So what I want is:

<main sizes="50vw">
  <img srcset="1.avif 1920w, 2.avif 640w"><!-- no redundant sizes attribute on children of main -->
  <img srcset="1.avif 1920w, 2.avif 640w">
  <img srcset="1.avif 1920w, 2.avif 640w">
</main>

[mirisuzanne]

I completely agree that this would be a very powerful and useful feature. My understanding was that responsive image sources are resolved before styles – making this reference impossible. But I'm not a browser engineer, and would love to discover that I'm wrong about this. :)

[css-meeting-bot]

The CSS Working Group just discussed [css-conditional] [css-contain] `srcset` and `sizes` interaction with container queries.

The full IRC log of that discussion

<astearns> topic: [css-conditional] [css-contain] `srcset` and `sizes` interaction with container queries
<astearns> github: https://github.com//issues/5889
<dlibby> una: next issue is related - how to deal with responsive images in container queries
<dlibby> una: would like to explore the solution space, since it seems like it'll be pretty common
<dlibby> una: adding a ??? to srcset, setting images on each compute, since they come from server
<bkardell_> q+
<dlibby> una: how should container queries interact, and background images are also a consideration
<bkardell_> how would this work with preloading? https://web.dev/preload-responsive-images/
<dlibby> s/???/new syntax/
<dlibby> una: do something simliar to media queries but seems important to consider for container queries
<astearns> ack bkardell_
<dlibby> bkardell: preloading - see linked article. you can't know what to load until later
<dlibby> una: yes, tricky because working at compute time, not sure how this works for viewport size
<dlibby> emilio: you can estimate what your viewport is
<dlibby> emilio: you can make a decent guess in iframes
<dlibby> emilio: you can't really do this in containers
<dlibby> iank: relatively sure that gecko and chromium reference image via background: url(foo), but can have multiple decls
<dlibby> iank: UA load images after computed value time so you don't load lots of things in the background
<dlibby> iank: it's a performance tradeoff
<dlibby> emilio: <img> tags are a little more important as user wants to immediatly see it
<dlibby> florian: worried that this could make slow connections slower
<dlibby> florian: you normally don't get all resources at the first request, progressive modification, which happens more on slow networks. as this happens container can change, possible to congest/thrash network
<dlibby> florian: that sounds unfortunate side effect
<dlibby> una: if youre using container queries and there's a hero, or in a card, or sidebar, those are different
<dlibby> una: if container queries allow for composable styles in components, I can't imagine that we can't have different image sizes based on the size
<bkardell_> not just image sizes maybe different art direction too?
<dlibby> una: not sure if it is slower to load a bunch of images first, or layout first then download other images
<bkardell_> I suggest we send this to RICG :)
<dlibby> florian: use case makes sense - should and can are different questions
<dlibby> florian: if you have multiple small parts, many changes could be happening, and as the stylesheet comes in over network, you might end up loading all the images
<dlibby> florian: ending up there sounds bad, but perhaps there's another approach
<dlibby> myles: problem appears endemic to container queries
<dlibby> florian: most thrashing doesn't hit the network for each change
<dlibby> una: perhaps we can scope to initial load to avoid this problem - maybe solves the majority use cases
<dlibby> emilio: not sure - if you resize the window, you'd bet stuck in possible suboptimal state
<fantasai> I was about to say the same thing also
<dlibby> myles: I don't think visual layout should be permanently affected based on network speed
<dlibby> myles: CSS shouldn't be sensitive to timing of bytes on the wire
<dlibby> una: might be misunderstanding. when we figure out the layout size keep that image
<dlibby> una: refresh page would get different images
<dlibby> emilio: layout while stylesheets are outstanding is common. stylesheet in body can affect the page layout, this should be taken into account
<dlibby> iank: the place where this functionality may make sense is html layout. some components are already stateful, but this should be carefully considered
<dlibby> e.g. a oneshot load - how does this affect the html layout, something to consider
<nicole> a balance btw not requesting unnecessary images and not waiting to load images until css is done parsing
<dlibby> myles: maybe specify behavior, UAs can use different heuristics
<dlibby> myles: CSSWG doesn't have to specify when loads are triggered
<dlibby> iank: option for this would be adding to Image specificiation (perhaps additional data on <img>) once this has been laid out and it reaches 'stable' state, then UA can fire off request
<dlibby> ???: problem with containment in general?
<fantasai> s/????/nicole/
<fantasai> s/???/nicole/
<dlibby> emilio: perhaps one approach is to define what the behavior should be (lazy or something). just a gneeral problem with containment.
<dlibby> s/containment/container queries/
<dlibby> iank: reasonably easy to polyfill, prototype, iterate to discover 'good behavior'
<dlibby> iank: don't have to nail down spec text right now, we can try to find optimal via prototyping
<dlibby> bkardell: might be good to ask folks on ??? CG
<jensimmo_> yes, do ask Mat.
<dlibby> bkardell: to the point of wanting different size, you might want different art directives(?)
<dlibby> astearns: is it possible to load different images?
<dlibby> iank: yes this is possible to get different urls
<dlibby> astearns: that is for separate declarations, but this is for a single declaration
<dlibby> iank: i was referring to img tag attribute, there's nothing in CSS currently that allows for this
<dlibby> nicole: having it in CSS should be the same problem as having this specified in the html itself. likely has same characteristics
<dlibby> una: i recognize that this is probably for specification in HTML
<dlibby> una: glad bkardell brought up different images - third party service could provide different crops
<dlibby> bkardell: precisely, that's what I wanted to be recognized
<dlibby> astearns: what else would you like to get from the group?
<dlibby> una: this conversation basically
<dlibby> astearns: perhaps prototyping per iank is the way forward, thanks for bringing it up

[scottjehl]

Can't speak to the implementation side, but I like this idea, @una. Just like with container queries, I think the container size is what we typically want to know for specifying image sources. Also, in the off chance that media comes back to video, it'd be nice to see this working there as well.

[eeeps]

This seems really exciting for art direction (e.g., <picture><source container> or whatever).

However, part of the goal of <source media> and srcset + sizes is to allow UAs to load images immediately, using information contained within markup alone, and I don't see how you resolve a cw or a container() or whatever, without waiting for style and layout.

For art direction... maybe that's a high performance price we have to pay to enable art direction within responsive components, which seems important.

But for sizes and srcset selection, once you've waited for layout, you don't need sizes at all. (See also: whatwg/html#4654)

I can maybe see a path forward to instant loading here if the queried container, itself, has a sizes attribute, which can be resolved using just viewport size... e.g.

<div
  class="container"
  sizes="(min-width: 800px) 800px, 100vw" />
  <img
    srcset="small.jpg 400w, medium.jpg 800w, large.jpg 1200w"
    sizes="container(min-width: 600px) 33cw, 100cw"
  />
  [more images with the same sizes go here]
</div>

But reading, writing, and maintaining sizes is already extremely difficult; this figures to make it worse. And it doesn't play nicely with container query use cases where the author doesn't know the container size (e.g. user-resizable components)

[chrishtr]

I don't see how you resolve a cw or a container() or whatever, without waiting for style and layout.

Under this proposal, the selection of the size would indeed depend on style and layout. Which would mean the loading of these images would be significantly delayed. The same problem occurs with loading=lazy, because the only way to determine if the image intersects the viewport is to do layout.

However, there are real advantages to this pattern, and there may be many good use cases where the image is not critical for first-load rendering time.

[joemcgill]

The natural conflict between prioritizing the ability to preload images before scripts/styles load and the desire to optimize the image loaded based on layout was always the trickiest thing to solve for responsive images. What I see above are basically two different use cases, each of which (as far as I can tell) would need to carry different assumptions about how that prioritization works.

1. Preload images based on a container width

To make this work, you would need to let the browser know what the container width is intended to be before layout is calculated, by applying something like sizes attribute to the container. Any children declaring a srcset with w descriptors that don't supply a sizes attribute would default to the closest ancestor's sizes rather than the current default of 100vw. This would satisfy the use case @cramforce mentioned above.

If the image does include a sizes attribute, it could use a container query to declare relative layout intent based on the closest ancestor's sizes attribute as @eepps example shows. There's some developer overhead that seems redundant with this syntax, but I think the possibilities would be pretty powerful.

2. Lazy load images based on container width

In the event that image markup relies on a container query or container relative measurements in it's sizes attribute, and there is no ancestor supplying a sizes context, then image loading would either need to be deferred until after layout is calculated (i.e., treated similarly to lazy loading) or, in cases where loading="eager" is intended, the file would be requested using the 100vw as the default sizes context when calculating container-relative sizes for image srcset parsing. This isn't ideal, but no different that then current assumption for images supplying a srcset without a sizes attribute.

I think those are acceptable constraints/tradeoffs for the benefits that come from being able to use these new tools in building reusable interface elements that respond to their layout context.

[Wilto]

srcset and sizes interaction with container queries

One loves to see it.

I think @joemcgill nailed my thoughts on this, but one thing worries me a little: “lazy load images based on container width” feels like both a sensible default for omitted sizes and a powerful feature in the right hands—like, “I want lazyload on this img in this CQ, so I’ll skip writing sizes” sounds great on a couple of levels.

What worries me is that sizes attributes (and please know I mean this with all love for the feature) are a huge pain to write by hand. Understanding the difference between preparse and layout calculation timings is a lot to ask of a junior developer that just wants to put an image on a page, and asking devs to calculate an accurate sizes attribute in order to shave off imperceptible-in-the-moment differences in load time could be a tough sell. “You can go to all the trouble of writing a sizes attribute and the image shows up quick and looks fine, or you can not write sizes and the image still shows up quick and looks fine,” y’know?

I’m a little concerned that the lazyloading pattern would end up telephone-gamed into “we don’t need sizes in CQ,” and we’d be effectively giving up preload on srcset-powered images inside CQ. That said, srcset has always felt a little like a stopgap. So, hm.

[MattWilcox]

That said, srcset has always felt a little like a stopgap

I've thought the same about sizes tbh. When we have aspect-ratio on hand and available, plus loading=lazy, then we can safely let the browser decide which asset to pull because it already knows the width it actually needs all by itself without hinting.