Add height-based selection to srcset/sizes

[zcorpan]

See https://bugs.chromium.org/p/chromium/issues/detail?id=421909#c19

As part of the the photo page redesign on unsplash.com, we want to constrain images by viewport height. We also want to use img srcset and sizes to deliver responsive images.

In our sizes attribute, it's possible to define the width of the image when constrained by viewport height using media queries—for example, sizes="(min-aspect-ratio: 1/2) 80vh". However, if we want to add vertical padding around the image, there appears to be no way to exclude that padding from the calculated aspect ratio.

If calculations in media queries were possible, we could achieve this using (min-width: calc(100vh - var(--vertical-padding))). For the time being we are having to rely on JavaScript to perform these calculations, with necessary fallbacks.

Here is a full example of the image behaviour we are trying to achieve: http://jsbin.com/melewe/edit?html,css,js,output

This pattern we're pursuing seems to be increasingly common, so it would be great if we could make this easier for authors.

calc() in MQ would be nice, but better still is probably to allow specifying the image heights directly (in srcset and sizes). We excluded this use case originally to reduce complexity, but since this appears to be a recurring issue for web developers, it seems worthwhile to address.

Earlier issue for this: ResponsiveImagesCG/picture-element#86

[zcorpan]

Examples of width- and height-constrained images:

• https://www.flickr.com/photos/zipco-and-cal/8741776058/
• https://vasilis.nl/voto/terreur/140928151042/

Examples of only height-constrained images:

• https://unsplash.com/?photo=6VjPmyMj5KM (for the wide page layout; the narrow layout has width-constrained images instead)

[OliverJAsh]

Thanks for filing @zcorpan.

I'm not sure I see how height-based selection in srcset/sizes would help with my example. The problem remains of how to define the media query part in sizes:

• when the image is constrained by viewport height, height is viewport height - vertical padding (easily expressed as width using aspect ratio calculation)
• otherwise the image width is viewport width - horizontal padding

I need a media query for the part in bold. In my example I'm using JavaScript to calculate this—I really want to be able to express this in plain CSS, which is where calc would come in.

[zcorpan]

I was looking more at the unsplash page, which appears to have a width-based layout breakpoint. So for that page, the media condition in sizes would reflect that breakpoint, and the specified size for the narrow layout would be 100vw and the specified size for the wide layout would be height 100vh (or whatever).

The jsbin example appears to be both width and height constrained, with some padding around the image, and the aspect ratio of the image is known. Correct? Maybe calc() in the media condition is enough to make it possible, but I'd also like to explore possibilities to make these things easier (maybe a contain keyword could help?).

[OliverJAsh]

The JSBin example is what we're moving towards, with width and height constrained images—exactly as you said.

(maybe a contain keyword could help?).

I saw some discussion about this in ResponsiveImagesCG/picture-element#86 and couldn't quite see how it would help in my example. I did originally try to achieve my layout using object-fit: contain, however I don't want to stretch the image to fill the viewport height—I only want to constrain it by the viewport height. (This way there is no extraneous white space.)

but I'd also like to explore possibilities to make these things easier

I have found it extremely difficult to express the layout you see in the JSBin example. My requirements are:

1. Reserve space for the image whilst it loads.
2. Contain image (including padding) in viewport (fill width or height, whichever is smallest), whilst only taking up necessary space.
3. Minimum height
4. Responsive images

For 1 we can use the padding-bottom trick.

Because all elements are constrained on the X axis by default, we have to express constraints along the Y axis as constraints along the X axis. For 2 we have to define the maximum height as a max-width (calculated using viewport heights and the aspect ratio). For 3 we have to define the minimum height as a min-width (calculated using the aspect ratio).

For 4 we want to repeat the layout described in 2 and 3 for the sizes attribute. My current solution requires JavaScript due to the lack of calc in media queries, which unfortunately means we lose benefits of the preloader, etc.

If you have any suggestions, for how to improve this now or in the future, I would love to hear them.

[OliverJAsh]

@zcorpan FYI, you can opt-in to the (WIP) new photo page on Unsplash with this link: https://unsplash.com/?xp=new-photos-page:experiment (temporary link only). If you then click through to a photo, you will see the new photo page with the behaviour described above. The layout is identical to the JSBin example I posted.

[zcorpan]

Thank you! This is extremely useful info for evaluating solutions. (Note that whatever we come up with here won't be usable immediately, it will have to be implemented and shipped in multiple browsers first, so in a few years or so...)

1. Reserve space for the image whilst it loads.

This is what ResponsiveImagesCG/picture-element#85 is about, and I think we should fix that together with this issue.

2. Contain image (including padding) in viewport (fill width or height, whichever is smallest), whilst only taking up necessary space.

OK, so that is what contain means. If we consider the proposal in ResponsiveImagesCG/picture-element#86 (comment) we get something like

sizes="contain calc(100vw - var(--horizontal-padding)) calc(100vh - var(--vertical-padding))"

3. Minimum height

OK, that adds a height-based breakpoint to sizes, and ability to specify the height would help so you don't need to map it to a width (which is not possible if the aspect ratio is unknown).

sizes="(max-height: 400px) height calc(400px - var(--vertical-padding)),
       contain calc(100vw - var(--horizontal-padding)) calc(100vh - var(--vertical-padding))"

plus h descriptors in srcset so the browser can select a candidate when sizes only gives a height size, and calculate the intrinsic size.

[OliverJAsh]

@zcorpan Thanks so much for the detailed response. We may be years away from having support for these changes, but it's good to understand the constraints of what we have today, and what is being done about that.

As I understand it, contain in sizes would tell the browser the image is contained, and either one of the specified width or height will be used depending on how the image is constrained. Is this correct? How does the browser know whether the image is constrained by width or height?

3. Minimum height

OK, that adds a height-based breakpoint to sizes

As the image is contained, I think it also adds a width based breakpoint to sizes? This is why in my example I have to specify both a max-width and max-height media query. Or is this somehow made redundant by the specified height?

I think this looks like it would fit my example perfectly, although it's hard to know for sure without trying it out.

[zcorpan]

Ah right, I missed that aspect. It would then be:

sizes="(max-height: 400px) contain calc(100vw - var(--horizontal-padding)) calc(400px - var(--vertical-padding)),
       contain calc(100vw - var(--horizontal-padding)) calc(100vh - var(--vertical-padding))"

I don't think further breakpoints are needed, or rather "contain" handles it already.

The browser would know which dimension to use because it knows the viewport size and the image aspect ratio would be provided by srcset by using both w and h descriptors.

[OliverJAsh]

If I understand correctly, wouldn't the first entry in sizes need to be:

sizes="(max-height: 400px) contain calc((var(--min-height) * var(--width-as-proportion-of-height)) - var(--horizontal-padding)) calc(400px - var(--vertical-padding)),
       contain calc(100vw - var(--horizontal-padding)) calc(100vh - var(--vertical-padding))"

That is, the contain width is (min height * width as proportion of height) - horizontal padding)?

[zcorpan]

I probably managed to confuse myself, sorry. It's difficult to reason about this without testing.

Anyway, I realize that the min-height would need to apply on both sides of the breakpoint, since shrinking the viewport width makes the image smaller on both dimensions. I need to think through how to apply the proposal to make it work as intended.

Alternatively, the proposal is still too difficult to work with, and we should come up with something else to make it simpler.

[zcorpan]

I think w3c/csswg-drafts#544 could help.

sizes="contain
       max(100vw - var(--horizontal-padding), var(--min-height) * var(--width-as-proportion-of-height))
       max(100vh - var(--vertical-padding), var(--min-height))"

[eeeps]

@OliverJAsh as for your use case/requirements – here’s the best, simplest thing I could come up with using what’s in browsers now: https://codepen.io/eeeps/pen/VMPJzK

It uses @tigt’s awesome coping-with-the-lack-of-h-descriptors technique.

Problems:

1. The sizes is not completely accurate right around the constrained-on-width/constrained-on-height boundary, because of the padding. Most of the time, this shouldn't affect resource selection.
2. There will likely be some jank when the image loads. Again I couldn't work this out, given the padding†.

h descriptors + contain would solve both problems. @zcorpan’s use of max() looks more elegant, but here’s what my not-used-to-max()-yet brain spit out:

<img srcset="https://via.placeholder.com/150x100  150w  100h,
             https://via.placeholder.com/300x200  300w  200h,
             https://via.placeholder.com/600x400  600w  400h,
             https://via.placeholder.com/1200x800 1200w 800h"
  sizes="((min-width: 342px) and (min-height: 292px)) contain calc(100vw - 12rem) calc(100vh - 12rem), 150px"
  src="https://via.placeholder.com/150x100" />

Note: 342px = min-img-width (150px) + padding (12 rem); 292px = min-img-height (100px) + padding (12rem) – consider the magic-ness of these numbers an argument for calc() in MQ.

---

†: this is the best that I could do. A closer fit to the requirements (it reserves the correct amount of space most of the time), but at some cost of complexity.

[eeeps]

As for use cases, I'll toss out a couple more.

Here's a sideways-scrolling site which is very cool and unsual.

Click on any of the images here to get thrown into a viewport-fit lightbox. Given the ~3,000 “lightbox” repos on Github, I expect this use case is much more common.

[eeeps]

Another use case which only dawned on me in a conversation with @yoavweiss about Hui Jing Chen’s (awesome) talk at You Gotta Love Front End – probably the biggest potential use case of all – images in vertically-sized blocks of vertically-flowing text (like this https://www.chenhuijing.com/slides/yglf-2017/#/8).

[tylersticka]

I ran into this organically, so I thought I'd contribute another use-case and the steps I attempted to take (in case that's helpful to folks evaluating this issue).

I wanted a big ol' image to lead off this blog post about an art project I've been doing: https://tylersticka.com/journal/drawing-every-day/

The img container is always 100% width, 50vh tall when orientation is portrait, 75vh tall when it is landscape. The img fills the available height and width and uses object-fit: cover.

Initially I wrote the img element like so:

<img src="..." alt="..."
  srcset="
    grid-05-27-960.jpg 960w, 
    grid-05-27-1920.jpg 1920w, 
    grid-05-27-2560.jpg 2560w" 
  sizes="100vw">

But this often resulted in the chosen asset being too small, since the asset will extend beyond the horizontal boundaries of its container.

So then I tried using calc to base the width on the aspect ratio multiplied by the height:

<img src="..." alt="..."
  srcset="
    grid-05-27-960.jpg 960w, 
    grid-05-27-1920.jpg 1920w, 
    grid-05-27-2560.jpg 2560w" 
  sizes="
    (orientation: portrait) calc(16 / 9 * 50vh), 
    calc(16 / 9 * 75vh)">

This kinda seemed to work in Firefox (or at least it was failing gracefully) but it seemed to cause Edge to abandon the srcset.

So, ever the optimist, I tried writing this:

<img src="..." alt="..."
  srcset="
    grid-05-27-960.jpg 538h, 
    grid-05-27-1920.jpg 1076h, 
    grid-05-27-2560.jpg 1435h" 
  sizes="
    (orientation: portrait) 50vh, 
    75vh">

When that failed, I finally Googled the issue, found ResponsiveImagesCG/picture-element#86 and finally arrived here. 🙂

(I ended up just using a sizes value that was comfortably above 100vw. It'll probably download more than the user needs sometimes, but it's better than nothing!)

[OliverJAsh]

@tylersticka We had a similar problem on https://unsplash.com for the "hero image" you see on the homepage, which also uses object-fit: cover. We ended up solving it using the picture element and providing multiple sources representing a spectrum of aspect ratios.

We sampled a rough height of the container element at regular width intervals (e.g. from 200px to 2000px, every 200px) and then provided a source for each.

[tylersticka]

@OliverJAsh That's a clever solution, thanks for sharing!

A similar technique could work for my example, though it makes my head hurt a little considering my image's visible area is based on orientation (not viewport width) and my nav shifts from the top to the side as well. It doesn't help that my personal site is static, so I'd be preparing those images and writing that markup "by hand!"

(It could also just be my end-of-workday brain being slow…)

[DanMan]

Anything still happening on this front? I'm also trying to create a lightbox and as is, I find it hard if not impossible to use portrait images on a landscape display in a responsive way. I can make it right in either direction (horizontally or vertically), but not both. Either the image is displayed too small/big, or a bad alternative is chosen. With object-fit, depending on circumstances, chances are that part of the image will be obstructed (instead), which you don't want in a lightbox.

For now I've solved it by giving the image a max-height:86vh (100vh minus margins and stuff), but that's just a workaround. The selected alternative is likely wrong.

[MALIK-0]

Hey guys, also wanted to ask if there are some news on this? We are working on an image service which works together with these native features to deliver the perfect image on the fly (also includes stuff like cropping, focus point for art direction, image compression, and much more). For some edge cases we really need to tell the browser the image height in addition to the width.

[rpeterman-gp]

The solution I came up with was to calculate the resulting width based on the width to height ratio when creating the media, srcSet, and sizes string values for my <source> and <img> elements. (This solution works in the browser or server-side rendered React application.)

//...
// This should get you the max image height, minus any margin and padding
// around the parent container.
const containerHeight = imgElm.parentElement.offsetHeight;
// For determining orientation.
const { info: { width: mainImageWidth, height: mainImageHeight } } = mainImageData;
// Used later to decide wether or not to calculate width in `size` attribute value.
const orientation = mainImageWidth >= mainImageHeight ? 'landscape' : 'portrait';
// For brevity and readability later on.
const isPortrait = orientation === 'portrait';
// Simple media query for orientation if using `object-fit: cover`.
// const media = `(orientation: ${orientation})`;
// Let browser know which images to use and their widths, as usual.
const srcSet = imageSrcs
  .map(({ src, info: { width } }) => `${src} ${width}w`)
  .join(',\n');
const sizes = `${imageSrcs.map(
  ({ info: { width, height } }) =>
    // Set media query for size as you need. I just went with the image width here.
    `(max-width: ${width}px) ${
         // Let browser know the width of image at this media query.
         // For `object-fit: contain`, calculate width for portrait, falling back to just width for landscape.
         isPortrait ? `${(width / height) * width * (height / contentHeight)}px` : `${width}px`
         // For `object-fit: cover`, do the inverse. Will crop vertical excess on portrait,
         // horizontal excess on landscape. Fine for use as background element.
         // isPortrait ? '${width}px' : `${(width / height) * width * (height / contentHeight)}px`
      }`
  )}`;
//...

Basic concept illustrated in javascript. Update/refactor for your programming language and image data model as needed.

[DanMan]

Of course, with the help of JS you can get done all kinds of things. The request is about a CSS solution.

[drewbaker]

At my agency, we build a lot of websites that use fullscreen imagery (100vh/100vw object-fit: cover). Using srcset and sizes with these sorts of layouts results in blurry images on portrait mode on a phone (because the phone this way has a small width, but a large height, and so cover has to scale up the low res imagery).

[strarsis]

@drewbaker: Though this is only applicable for edge cases (as for full page width/height images), it may be interesting for you:
https://codepen.io/tigt/post/when-responsive-images-get-ugly#height-and-width-constrained-srcset-13
The described approach makes it possible to use the sizes attributes for cover/contain sized images.

[mnpenner]

I also came up with a JS-based solution (for React).

const LazyImg: React.FC<Props> = ({srcSet,...props}) => {
    const ref = useRef<HTMLImageElement>(null);
    const rect = useBoundingBox(ref);

    const src = useMemo(() => {
        if(srcSet?.length && rect.width > 0 && rect.height > 0) {
            const aspectRatio = getAspectRatio(rect)
            const sources = sortBy(srcSet, [
                src => Math.abs(getAspectRatio(src) - aspectRatio),
                src => src.width < rect.width,
                src => src.height < rect.height,
                src => {
                    const isBigger = src.width >= rect.width && src.height >= rect.height;
                    return (isBigger ? 1 : -1) * src.width * src.height;
                },
            ])
            return sources[0].url
        }
        return undefined;
    }, [srcSet,rect])

    return <img ref={ref} src={src} {...props}/>
}

It takes as input a list of images with a url, width & height. Then it compares each of them with the actual size of the img element (which I have styled to fit neatly in a grid). It sorts them to find the image with the closest aspect ratio, then one that is bigger then the container, and if there is one bigger than the container, the smallest such image (to save on bandwidth and memory), otherwise the biggest image.

I was hoping for something like this built-in the the browser. I just feed it my list of images and it figures out the best-fit one based on the final rendered size. You could even add 'byte size' and 'quality' as parameters if the user is on a slow connection and its more important to optimize for download speed.

Full code: https://gist.github.com/mnpenner/88f5d5d2c19b7f25bc0a4dba6c3d4a55

[strarsis]

@mnpenner:
If you are interested, a (slightly) over-engineered solution that uses ResizeObserver, MutationObserver and also observes DPR changes: https://codepen.io/strarsis/pen/RwjRraM

I try to add the height-based art-direction via lazysizes plugin so I can use lazysizes than this implementation.