New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: images handling #6177

Open
azaozz opened this Issue Apr 14, 2018 · 51 comments

Comments

Projects
None yet
@azaozz
Contributor

azaozz commented Apr 14, 2018

This is a "mini-proposal" on how to handle images in the editor and the front-end (follow-up from #4914).

Currently:

  • Images are inserted in the editor at full size, i.e. we force the user to download 2MB-5MB files in most cases.
  • When the user saves the post without changing image source from the inspector, it is possible that front-end visitors will have to also download the huge files. WP adds proper srcset attribute, but the sizes attribute is pretty useless as it only refers to the full size image file: sizes="(max-width: 6000px) 100vw, 6000px".

To fix this in the editor:

  • Always insert the "large" size (1024px) images . If it doesn't exist (if the uploaded image is smaller), insert the "full" size.
  • Add srcset attribute that will also list 2x large size (so images are retina ready). This is a new size and will have to be added to the default WP sizes, see below.
  • Add another attribute to the tag for Gutenberg images so we can recognize them easily in PHP. This is necessary as we will have to calculate the front-end srcset and sizes attributes a bit differently. Ideally that should have the attachment ID, something like data-wp-attachment-id="1234". Then perhaps we will be able to drop the "old" way of passing the ID, class="wp-image-1234".
  • Special handling of the width="123" attribute. In HTML 5.0 is has to be in pixels, but since the width of the editor is different than the width of the theme, the results would often be unexpected. This impacts mostly images that are resized when the user drags the corners or when image dimensions are set directly (see #4914). We will need better solution for these cases, perhaps recalculating the number when displaying the image on the front-end according to the theme's width.

To fix this for the front-end:

  • Add new default size 2x large, 2048px max width or height.
  • Add some logic when processing <img> tags and adding srcset, etc. that will produce usable sizes attribute. It will depend on the new data-* attribute and take into account theme's width when calculating sizes. If we are going to recalculate hard-coded width attributes (see above), this can be done here and the values used in both attributes. Alternatively we can set the width in percentage (HTML 4.0 style) in the editor, then replace them with pixels at this point.

Implementing all of the above will ensure all images are always "retina ready" both in the editor and on the site. It will also improve handling on the front-end and optimize image loading there.

This will also affect other enhancements, mostly #4914. Also, as described in #6131 the themes will be able to add more precise sizes attributes for the different images.

@azaozz

This comment has been minimized.

Contributor

azaozz commented Apr 14, 2018

Pinging @noisysocks and @iseulde for the editor changes. I can add the core changes.

For the editor the changes (as I see them) would be:

  • Add a data-wp-attachment-id="1234" attribute to all <img> tags.
  • Always insert the "large" size image.
  • Add srcset attribute with the medium, large and xlarge sizes (we'll add the xlarge image size to the default sizes shortly).
  • Add sizes attribute suitable for the image width in the editor.
  • On saving, perhaps remove both the srcset and sizes as they would probably be different when displaying. Alternatively we can override them in the display filter.
  • When setting any pixel values for image dimensions, base them off the editor width. It won't make sense to have images wider than what is visible in the editor.

TODO: consider what would be the best way to pass percentage width to the front-end.

@noisysocks noisysocks added this to the Merge Proposal: Media milestone Apr 16, 2018

@noisysocks

This comment has been minimized.

Member

noisysocks commented Apr 16, 2018

Thanks for writing this up, @azaozz! Happy to help with the necessary editor changes. They sound good to me.

Alternatively we can set the width in percentage (HTML 4.0 style) in the editor, then replace them with pixels at this point.

My hesitation with this is that if a third party (e.g. plugin, RSS reader, API consumer) uses the post_content without processing it then they won't have valid HTML5 markup. "Honouring HTML" is one of Gutenberg's requirements.

Thinking out loud: could we use Gutenberg metadata to accomplish some of this, e.g. to store the attachment ID and percentage width?

<!-- wp:image {"attachmentId":123,"horizontalScale":0.25} -->
<img src="https://example.com/image.jpg" width="600" height="500" />
<!-- /wp:image -->
@azaozz

This comment has been minimized.

Contributor

azaozz commented Apr 16, 2018

Yeah, images are "dynamic blocks" (that's how they are treated in the classic editor too). They are the best kind of dynamic blocks as they have the "fallback markup" right in there, and are processed on the
front-end to enhance them.

Having that fallback markup makes images work "everywhere" even when the "display filters" haven't been run (this would actually leave the content without paragraphs, so don't think any plugin would do it).

We can certainly add the data we need to pass to the front-end to the blocks meta, however thinking we will need the same data in the <img> tags too. One reason is that parsing blocks on display is slow and we will get bigger chunks of HTML instead of the actual img tag (the figure with the caption, etc.). Another is that (perhaps) not all images will be in separate blocks, thinking about images in blockquotes, table cells, etc. Yet another reason is that if the post is edited in any other editor the blocks may be badly mangled, but the <img> tags will "survive" and stay intact (as long as we use standard HTML).

At the end this boils down to:

  • Which is easier, faster, simpler, more readable/understandable: parsing the block and then processing the chunk of HTML (that may contain several elements), or parsing the <img> tags as we currently do.
  • Can we guarantee that all images used in the editor will always be in wp:image blocks. That includes pasted images, images in table cells, quotes, and any other blocks and tags combinations plugins may come up with.

Imho we should use both the block meta and the img tag attributes, then (one day) we will be able to choose which to use when parsing the content on display.

@davisshaver

This comment has been minimized.

Contributor

davisshaver commented Apr 19, 2018

@azaozz This is a well thought out proposal. I am not fully grasping why we don't use existing theme-declared image sizes though. Is there a prior discussion around this?

Currently a percentage field is exposed but I wasn't able to determine whether this does anything at present on the frontend. https://github.com/WordPress/gutenberg/blob/master/blocks/library/image/block.js#L266

The percentage approach is nice but it breaks backwards compatibility with the theme API.

@azaozz

This comment has been minimized.

Contributor

azaozz commented Apr 19, 2018

why we don't use existing theme-declared image sizes...

We do :) We use all available sizes (that have the same w / h ratio) when adding the srcset attribute on the front-end. Then the browser decides which image file to use.

@davisshaver

This comment has been minimized.

Contributor

davisshaver commented Apr 19, 2018

@azaozz I believe that would function a bit differently than current behavior. Image sizes can also be used to deliver styles, not just the image crop to use. I also think that existing lazy load plugins expect the src value to be the "right" size so I'm not sure we can rely totally on srcset.

Edit: This would be the image_size_names_choose filter I think.

@mor10

This comment has been minimized.

mor10 commented Apr 21, 2018

@davisshaver I've gone through several lazy load plugins in the past week. Many of them look for existing src and srcset attributes and simply replace them with other markup in PHP using a the_content filter. The challenge here is how to get core to generate the correct srcset and sizes attributes as well as physical images in the first place.

@mor10

This comment has been minimized.

mor10 commented Apr 21, 2018

There are several issues related to responsive images that will have overlapping solutions. I propose we consolidate everything into this issue to avoid further fracturing of the discussion. Here's an incomplete list:

Related issues:

  • #5674 Gallery images are not responsive
  • #4505 Add 'has-wide-support' classes and mechanisms
  • #4342 Content displays incorrectly when switching themes
  • #6131 Allow theme to control sizes attribute for wide/full images

Core tickets:

@ms-studio

This comment has been minimized.

ms-studio commented Apr 21, 2018

Regarding this, in @azaozz proposal:

Add new default size 2x large, 2048px max width or height

Has there been consideration of bumping up the current default pixel sizes of "thumbnail", "medium", and "large", which were originally defined in 2008? Maybe with Gutenberg it's the perfect moment of increasing them x2.

Otherwise, the terms "medium" and "large" will lose their semantic meaning.

@mor10

This comment has been minimized.

mor10 commented May 3, 2018

Also related: #5650 content_width and new block widths

Same core issue cause: new content width paradigm makes things that used to be standard not work as expected and introduces need for rethink of how core works.

azaozz added a commit that referenced this issue Jun 11, 2018

@azaozz

This comment has been minimized.

Contributor

azaozz commented Jun 11, 2018

/update/image-block is a "work in progress" :) Please test.

Changes:

  • Add srcset and sizes inside the editor.
  • Always insert the large image size, or full if large doesn't exist.
  • Show a Default image size, a Thumbnail, plus any sizes adder by plugins and themes.
  • Add data-wp-attachment-id and ata-wp-percent-width to the img tag. Will be used on the front-end to properly set srcset and sizes, and also img width and height where missing.
  • Improve setting/resetting the image dimensions.
  • Add srcset to the attachment data from the API and to the data in the media modal.

TODO:

  • Add the front-end code that will handle the new img tag attributes.
  • Fix removing focus from the image (and hiding the resize handles) when the caption is focused.
  • Try to fix resizing by dragging. Currently dragging the corners left or right doesn't do anything, only dragging up or down resizes the image (and is chunky).
@joemcgill

This comment has been minimized.

Contributor

joemcgill commented Jun 15, 2018

@azaozz this is looking really great so far. Nice work!

A few things I've noticed while testing this out:

First, editor width is probably not going to match the intended display width on the front end, so saving the sizes attribute based on the editor attribute is probably the wrong approach and causes the image to be constrained to the editor width, rather than the content container in the theme. We either need to make use of the content_width value set by the theme on the front end, or continue to set sizes on the front end, rather than saving it to the image markup (which, I'm sure you won't be surprised to hear I still think is a bad idea).

The latter would be my suggestion. Here we can now take advantage of the block parser in WordPress to implement a much better responsive image solution, rather than a filter on the_content. Currently, the block is saving duplicate attribute data to the image block's comment delimiter and to the markup. We could continue to save srcset and sizes as block attributes without adding them to the markup. We can then continue using them in the editable component as you have now, and make them available to the block parser. We could also clean up several of the other duplicates in the process by declaring a source for attributes we're saving to the markup.

I 💖 the changes to the size dropdown here. I'm curious if you've thought about how this could be extended to allow for custom crop sizes, i.e. add_image_size() options to be included here so someone could create a set of hard cropped options for an image.

A bug that I ran into was inserting an image, setting the source type to thumbnail, saving a draft, and then refreshing the page. In that case, the block parser responds with the classic "This block appears to have been modified externally" error. Seems it's expecting only src and alt attributes and is choking on the extra attributes on the markup.

There's still some weirdness with right/left aligned images. I wonder if in these cases we should explicitly resize to a percentage of the editor width and update the sizes attribute to match?

I was surprised by the choice to add a specific srcset value to each size returned by the REST API and by wp_prepare_attachment_for_js but it's a clever approach that I hadn't ever considered. We should open a ticket in core for this so we can discuss the pros/cons to explicitly adding this data to those responses rather than leaving this in a filter once Gutenberg lands. One small note, I'm seen some notices in the REST filter that sometimes $response->data['media_type'] does not exist, and is throwing notices, which is weird because it should always be defined. Something fun to dig into.

I'm going to leave this for now and continue testing. Bravo, sir.

@azaozz

This comment has been minimized.

Contributor

azaozz commented Jun 18, 2018

@joemcgill thanks for having a look!

First, editor width is probably not going to match the intended display width on the front end

Right. This is the reason to add the data-wp-percent-width attribute to the img tag. Then we will be able to recalculate the width on the front-end and "match" what the user sees in the editor. This also makes it compatible with editing on a phone, and also supports wide and full widths when set in the theme (we should get the themes to add these and start using them on the front-end).

...saving the sizes attribute based on the editor attribute is probably the wrong approach

Yep, it is intended to be overridden on the front. Was thinking if we should keep it in the image tag as a "fallback" in case something goes wrong. However thinking to leave the srcset in the tag as the chance for it to change after the post is published is very small (it will be filterable on the front-end of course). It's no good to have only srcset and no sizes, as far as I see the browser seems to download the largest image possible (probably another fallback) :)

we can now take advantage of the block parser in WordPress...

Possibly but that is going to be pretty slow. For now running the block parser on the front-end is not something we can do.

Currently, the block is saving duplicate attribute data to the image block's comment delimiter and to the markup.

That's how block properties work. Alternatively we can "extract" the image blocks with some regex and parse the HTML in them looking for the image tag. Still... parsing even a bit of HTML with regex seems like something we should avoid. So for now thinking to add all needed context from the editor in the actual img tag.

I'm curious if you've thought about how this could be extended to allow for custom crop sizes (the size dropdown)

This should work at present. Only sizes that match the original image ratio are removed from the drop-down, the rest (plus the thumbnail) are shown. So all custom crops added by themes and plugins are in it.

A bug that I ran into was inserting an image, setting the source type to thumbnail, saving a draft, and then refreshing the page.

Uh, will check that again. Thought I caught all of these.. This also prompted me to look at block validation, and why blocks fail so often (for example after the user adds another attribute). Thinking we will need some more general fixes there too, but that's for another issue.

There's still some weirdness with right/left aligned images. I wonder if in these cases we should explicitly resize to a percentage of the editor width and update the sizes attribute to match?

Yeah, thinking the previous behaviour was better there. We should probably go back to setting right/left aligned images to 50% width. Can also change sizes if we are keeping that attribute i the img tag.

I was surprised by the choice to add a specific srcset value to each size returned by the REST API and by wp_prepare_attachment_for_js

Tested several approaches there but this seems to work best. At the point when adding the srcset the image meta is already cached so it adds a negligible overhead when loading the data for the media library and the API. That way we also match the "proper" srcset that will be used on the front-end.

We should open a ticket in core for this...

Yeah, once Gutenberg is merged, this should go to the proper functions.

I'm seen some notices in the REST filter that sometimes $response->data['media_type'] does not exist

Hmm, we should probably open a core trac ticket for that. It should always be present.

I'm going to look into adding the front-end code that uses the new attributes next. Then we will have a good start to tweaking all this :)

@mor10

This comment has been minimized.

mor10 commented Jun 25, 2018

I had a chat with @azaozz at WCEU contributor day and want to share my thoughts with everyone in an effort to move this forward.

Context: Front-end (theme / plugin territory)
Assumptions: The proposal put forward by @azaozz re: setting a data-wp-percent-width attribute with the displayed width relative to the available space.

Proposal

The challenge I'm interested in, as I've stipulated before (#6131), is how theme and plugin developers can control the sizes attribute on the front-end of the site based on varying factors including but not limited to different layouts (sidebar(s) or not, other factors) without having to manually rewrite the entire sizes attribute for every condition as in this example.

Consider the following scenarios, a through i:


Historically, we've known two things about any image displayed: The physical width of the image file, and the $content_width as defined by the theme. We then used these two parameters to figure out the sizes attribute.

Gutenberg introduces not only the alignwide and alignfull widths, but also the ability to make various elements wide or full including but not limited to columns etc. As a result there are a myriad of possible display widths within any one theme in addition to the regular challenges posed by responsive web design.

These examples show there are two constants which can be easily defined by the theme developer (and by extension plugin developers):

  • $content_width - the maximum width of the content when it is not displayed as alignwide or alignfull.
  • $max_display_area - the maximum available space available to be filled if content is set to alignfull.

There is also one third assumption we can make:

  • An element set to alignwide will take up the full $content_width plus half of the available space between $content_width and $max_display_area, and can be calculated as
$content_width + ( $max_display_area - $content_width ) / 2

In other words, if WordPress knows the $content_width and $max_display_area values, it can calculate accurately the space inside which content is displayed and from there determine on the fly what the sizes attribute of a displayed image is based on how it is displayed, including the new data-wp-percent-width introduced by @azaozz.

Practical application

The theme developer defines two values:

  • $content_width declaring the maximum content width (as in the width of content if no alignwide or alignfull is applied) for any display condition.
  • $max_display_area declaring the maximum available width for any display condition.

Both these values will be variable depending on conditions so neither can be a global the way $content_width is today (unless I'm misunderstanding how $content_width currently works).

For images displayed without alignwide or alignfull, the $content_width variable (and custom width data-attribute if available) is sufficient to determine the sizes attribute: The displayed size will never be wider than $content_width or some percentage of $content-width defined by the data-attribute. This will be significantly simpler to set up for theme developers than the current methodology btw.

For images displayed in an alignwide or alignfull context, we need to use the $max_display_area value. It could make sense to define this variable as an array of some sort pairing viewport widths and available display areas. This array can, in combination with the data-attribute, be turned into sizes attributes generated on the fly.

So for the examples above a theme would declare something like:

// In this theme there is a fixed maximum content width of 720px.
$content_width = 720px;

if ( is_active_sidebar( 'sidebar-1' ) {
  $max_display_area = [
    'min-width: 48em' => 'calc( 100vw - 30em), // sidebar is 30em wide.
    'fallback' => '100vw',
  ];
} else {
  $max_display_area = [
    'fallback' => '100vw',
  ];
}

Assuming for d, e, and f, the breakpoint where the sidebar appears is at 48em and for g, h, and i, the data-wp-percent-width attribute is 50%, the resulting sizes attributes for examples at the top of this comment would be as follows :

/**
 * a: Image width is equal to $content_width area.
 */
sizes="(min-width: $content_width) $content_width, fallback"
sizes="(min-width: 720px) 720px, 100vw"

/**
 * b: Image width is 50% of the available space between $content_width and $max_display_area.
 */
sizes="(min-width: $content_width) calc(($max_display_area - $content_width) / 2), fallback"
sizes="(min-width: 720px) calc((100vw - 720px) / 2), 100vw"

/**
 * c: Image width is equal to fallback.
 */
sizes="fallback"
sizes="100vw"

/**
 * d: Image widths is equal to $content_width area.
 */
sizes="(min-width: $content_width) $content_width, fallback"
sizes="(min-width: 720px) 720px, 100vw"

/**
 *e: Image width is $content_width plus 50% of the available space between $content_width and $max_display_area.
 */
sizes="(min-width: $content_width) calc(($max_display_area - $content_width) / 2), fallback"
sizes="(min-width: 720px) calc((100vw - 30em) - 720px) / 2, 100vw"

/**
 * f: Image width is equal to $max_display_area.
 */
sizes="$max_display_area, fallback"
sizes="(min-width: 48em) calc(100vw - 30em), 100vw"

/**
 * g: Image width is 50% of $content_width. 
 */
sizes="(min-width: $content_width) calc($max_display_area * (data-wp-percent-width / 100)), calc(fallback * (data-image-width / 100))"
sizes="(min-width: 720px) calc(720px * .5), calc(100vw * .5)"

/**
 * h: Image width is 50% of $content_width plus 50% of the available space between $content_width and $max_display_area.
 */ 
sizes="(min-width: $content_width) calc((($max_display_area - $content_width) / 2) * (data-wp-percent-width / 100)), calc( fallback * (data-image-width / 100))"
sizes="(min-width: 720px) calc(((100vw - 720px) / 2) * .5), calc(100vw * .5)"

/**
 * i: Image width is equal to 50% of fallback.
 */
sizes="calc(fallback * (data-wp-percent-width / 100))"
sizes="calc(100vw * .5)"
@azaozz

This comment has been minimized.

Contributor

azaozz commented Jun 26, 2018

Thanks @mor10, very nicely put :)

Yes, to properly process and display images on the front-end we need couple of bits of data: some context from the editor on how the image was used exactly, and some data from the theme on how the image will be displayed.

I have couple of small suggestions/clarifications.

$content_width - the maximum width of the content when it is not displayed as alignwide or alignfull.
$max_display_area - the maximum available space available to be filled if content is set to alignfull.

I'm (still) thinking we should let the theme (be able to) specify all three widths: content/main column, wide and full. That way we won't be forcing anything on the themes. Some may want a bit wider or narrower alignwide images. Using half of the difference between content_width and full for the wide may be a fallback, in case the theme is outdated, perhaps?

So these should be something like:

  • $content_width
  • $alignwide_width
  • alignfull_width

(Technically they can also be in an associative array so we don't "litter" the globals space too much. Having that array will also signal that the theme supports this.)

Another important distinction is that we will expect these new widths to be "contextual". I.e. they should be set by the current template before it starts to output the HTML, (and perhaps reset at the end). This is critical for generating more precise sizes attributes for all images.

Gutenberg introduces not only the alignwide and alignfull widths, but also the ability to make various elements wide or full.

Don't think we need to be concerned about that re: images. If a block is set to full, it wouldn't make sense to have an alignfull image in it. It will be exactly the same as "normal" image. Same for wide blocks.

@mor10

This comment has been minimized.

mor10 commented Jun 26, 2018

Having all three widths specified makes sense, and the associative array would indeed make it cleaner. The only question is how to modify that with conditions... I guess the theme dev would just modify the array?

Don't think we need to be concerned about that re: images. If a block is set to full, it wouldn't make sense to have an alignfull image in it. It will be exactly the same as "normal" image. Same for wide blocks.

If you look at the examples in my comment you'll see we have to account for these. If someone has a very large screen and a block is set to alignfull with a 50% image inside, that image could be substantially larger than the content width. sizes attributes have to describe the actual display width of an image, and anything placed within a alignwide or alignfull context will have different widths than otherwise.

@alialaa

This comment has been minimized.

alialaa commented Jul 20, 2018

Hello @mor10, in your example, shouldn't the sizes for 'b' be:

sizes="(min-width: $content_width) calc($content_width + ($max_display_area - $content_width) / 2), fallback"

?

@mor10

This comment has been minimized.

mor10 commented Jul 23, 2018

@alialaa The math might well be wonky. It was provides as a prototypical example only.

@mor10

This comment has been minimized.

mor10 commented Jul 23, 2018

@azaozz Where are we with this? I just tested Gutenberg 3.3 with the official Gutenberg Starter Theme and as far as I can tell the current blocks are all using the full image size for everything. This causes a significant performance hit when the plugin is activated.

@andreiglingeanu

This comment has been minimized.

Contributor

andreiglingeanu commented Aug 2, 2018

@mor10 I'm trying to implement something that's very similar to the code provided by you (the wp_calculate_image_sizes filter), except I'm trying to achieve this for a bigger theme which has a lot more layouts variations. The biggest issue up until now is to implement a robust generator of sizes attribute for every place an image may appear in the theme. The biggest problems are:

  • fluid vs. fixed width layout
  • Grid system (in percents from the container)
  • Grid gap
  • Sidebar presence/absence
  • Multiple media query breakpoints

What I managed to implement already is some PHP logic that accepts a set of parameters and outputs an array of sizes, with width descriptors and media queries. My goal is to cover the most common bootstrap layouts, they are super widely used

For the context, here are the breakpoints (with slightly modified media queries) that are used to generate these numbers.

The valid parameters for the thing are:

  1. bootstrap_class: can be very long, accepts what bootstrap accepts, this is actually parsed and used to generate multiple media queries
  2. gutters: true | false. Put the 30px gutters into calculation or not. This amount is configurable
  3. fluid: true | false. Non-fluid layouts are the hardest, they have max-width in pixels

Here are some test cases that actually pass, in JSON format (each entry is a test case, first object is the input to my logic, and the second is the output):

[
	[
        
		{"bootstrap_class": "col-md-12", "gutters": true, "fluid": true},
		[{"media_query": false, "image_size": "calc(100vw - 30px)"}]
	],

	[
		{"bootstrap_class": "col-sm-12", "gutters": true, "fluid": true},
		[{"media_query": false, "image_size": "calc(100vw - 30px)"}]
	],

	[
		{"bootstrap_class": "col-xs-6", "gutters": true, "fluid": true},
		[{"media_query": false, "image_size": "calc(50vw - 30px)"}]
	],
	[
		{"bootstrap_class": "col-xs-8", "gutters": true, "fluid": true},
		[{"media_query": false, "image_size": "calc(66.6667vw - 30px)"}]
	],
	[
		{"bootstrap_class": "col-xs-5", "gutters": true, "fluid": true},
		[{"media_query": false, "image_size": "calc(41.6667vw - 30px)"}]
	],
	[
		{"bootstrap_class": "col-xs-9", "gutters": true, "fluid": true},
		[{"media_query": false, "image_size": "calc(75vw - 30px)"}]
	],
	[
		{
			"bootstrap_class": "col-xs-10 col-sm-3 col-md-7 col-lg-11 col-xl-5",
			"gutters": true,
			"fluid": true
		},
		[
			{"media_query": "1300px", "image_size": "calc(41.6667vw - 30px)"},
			{"media_query": "1000px", "image_size": "calc(91.6667vw - 30px)"},
			{"media_query": "690px", "image_size": "calc(58.3333vw - 30px)"},
			{"media_query": "480px", "image_size": "calc(25vw - 30px)"},
			{"media_query": false, "image_size": "calc(83.3333vw - 30px)"}
		]
	]
]

After getting the output, it can be assembled easily into a proper sizes attribute ready to be inserted in HTML.

For example, here's what the output should be for a very complicated col-xs-10 col-sm-3 col-md-7 col-lg-11 col-xl-5 image should be:

[
	{
		"bootstrap_class": "col-xs-10 col-sm-3 col-md-7 col-lg-11 col-xl-5",
		"gutters": true,
		"fluid": true
	},

	[
		{"media_query": "1300px", "image_size": "calc(41.6667vw - 30px)"},
		{"media_query": "1000px", "image_size": "calc(91.6667vw - 30px)"},
		{"media_query": "690px", "image_size": "calc(58.3333vw - 30px)"},
		{"media_query": "480px", "image_size": "calc(25vw - 30px)"},
		{"media_query": false, "image_size": "calc(83.3333vw - 30px)"}
	]
],

Which in final assembled HTML would look like that:

<div class="container-fluid">
  <div class="row">
    <div class="col-xs-10 col-sm-3 col-md-7 col-lg-11 col-xl-5">

      <img
        srcset="..."
        sizes="
          (min-width: 1300px) calc(41.6667vw - 30px),
          (min-width: 1000px) calc(91.6667vw - 30px),
          (min-width: 690px) calc(58.3333vw - 30px),
          (min-width: 480px) calc(25vw - 30px),
          calc(83.3333vw - 30px),
        "
      >

    </div>
  </div>
</div>

Now test cases for the non fluid layouts are much more complicated and harder to follow.

Also, sometimes the bootstrap_class is not enough and we need to be able to express the image width in raw percentages.

If interested, I can hop into a more detailed chat somewhere else about this thing, I'm really interested into moving it forward properly, there are still lots of problems that I face 🙂

@mor10

This comment has been minimized.

mor10 commented Aug 7, 2018

#8593 has added srcset and sizes attributes to individual gallery images, but the sizes attribute still acts as if each image takes up the full content width of the view which is rarely the case.

Related: #1450

@azaozz

This comment has been minimized.

Contributor

azaozz commented Nov 1, 2018

@joemcgill thanks for testing! :)

I'll again reiterate that I am strongly against the idea of saving srcset and sizes attributes to image markup in the database... srcset attributes can and should change whenever additional image sizes are generated for attachments (after switching themes, or adding new image sizes to an existing theme, etc.).

This is something we looked at few years ago when implementing srcset and sizes. In theory that sounds somewhat possible, however in practice that happens so rarely that it should be left for plugins to handle. It also breaks generation of srcset and sizes when the content is exported from one WP site and imported in another as attachment IDs change (but URLs don't change).

However in this case the srcset and sizes attributes are purely for the editor's sake. They are always regenerated on the front-end, see https://github.com/WordPress/gutenberg/blob/update/image-block/packages/block-library/src/image/index.php#L244.

Still think we should open a core ticket for WP 5.1+ to look into reusing the srcset attribute as that fixes imported content. The default sizes attribute is theme agnostic, but thinking it should always be regenerated and filtered on the front-end.

Gutenberg already formats images markup in a way that supports our current responsive image solution via the front end filter

Not quite :) Currently Gutenberg "forces" the full-size images both in the editor and on the front-end. It's true that a srcset attribute is added to <img> tags, but look at the default sizes attribute there. It's "borked" :)

I think the focus should be improving the filters available for themes to modify the sizes attribute...

Absolutely agree. This adds few new filters that help there, but most importantly we should pass the block_attributes to the wp_calculate_image_sizes filter (also to wp_calculate_image_srcset to match).

In addition after the last update there is quite a bit of editor-related data passed in these block attributes. That allows for proper scaling of images on the front-end, resetting of width and height (and ensuring these are always present, best practice), and generally gives a lot more options to the theme when rendering the image block.

I noticed that in the editor, Chrome is now downloading two versions of every image

This only happens for freshly uploaded images. As the image is added as "preview" while uploading, we need to change the src to point to the "large" size after there is an attachment post. Also, inside the editor the benefit of having srcset is to load and display "retina" images. There are no speed benefits as the editor content (HTML) is loaded long after the page has finished loading.

On full-width images, the max-width is getting capped at 1024px which breaks the preview. This doesn't happen in master.

Right. The alternative here is to always download 2-4MB or sometimes larger images. That is unacceptable, even if only in the editor. To fix this limitation we need to generate another image subsize by default, larger than "large". This was discussed many times in Slack, and there is a core ticket that's pretty "old". We should definitely look into doing that soon.

I'll continue testing and adding notes

Yes please! :)
Thinking of making a PR tonight, seems most edge cases in handling images are accounted for.

@joemcgill

This comment has been minimized.

Contributor

joemcgill commented Nov 1, 2018

However in this case the srcset and sizes attributes are purely for the editor's sake.

If that's the case, then it definitely doesn't make sense to save those attributes to the saved markup in the database.

Not quite :) Currently Gutenberg "forces" the full-size images both in the editor and on the front-end. It's true that a srcset attribute is added to tags, but look at the default sizes attribute there. It's "borked" :)

Ah right, helpful distinction here is that the width attribute of the image is being saved incorrectly, because we're no longer constraining the image size to content_width.

This only happens for freshly uploaded images.

I don't believe this is true. I can save a post, refresh the editor, and still see the double download happening.

Right. The alternative here is to always download 2-4MB or sometimes larger images. That is unacceptable, even if only in the editor.

Sorry, I don't think I was clear here. What I'm seeing is the image width being limited in CSS by the inline style on the wrapping div in the editor. Here's some screenshots to help:

Full size image in the editor not spanning the full width – https://cloudup.com/cHob4kcjLrN

Markup of the above in the inspector – https://cloudup.com/cIVWetqpSoF

@joemcgill

This comment has been minimized.

Contributor

joemcgill commented Nov 1, 2018

The scope of this ticket is very large, but there are a few bits that are crucial for WP 5.0. Can we break down the specific issues that would be regressions? As I see it, this includes:

  • Keep full size images from loading on the front end.
  • Save a more appropriate width to img elements in post content (best practice, but also relevant for calculating a default sizes attribute for responsive images).

High priority, enhancements:

  • Provide a way to filter sizes on the front end based on image alignments (#6131).
  • Keep full size images from being loaded in the editor.

Enhancements (nice to have):

Anything I'm missing?

@mor10

This comment has been minimized.

mor10 commented Nov 8, 2018

Provided an example over in Twenty Nineteen repo of why this is a blocker for the theme and for Gutenberg in general. The performance hit currently incurred by Gutenberg is significant: WordPress/twentynineteen#50 (comment)

@mor10

This comment has been minimized.

mor10 commented Nov 23, 2018

From my testing, responsive images are still not resolved in 5.0 RC1. wp_calculate_image_sizes does not have a block properties attribute, and render_block_core_image_tag_attributes from #11973 has not been merged.

Not to put too fine a point on it, but this means all images not manually resized within the editor and aligned alignnone, alignwide, and alignfull in RC1 are broken because their sizes attributes are incorrect.

To test, activate Twenty Nineteen, upload a large image (1200px or wider) into a post, align it none, alignwide, or alignfull, and check the output sizes attribute. You'll find that regardless of the displayed width or the intrinsic width of the image, the value is:

sizes="(max-width: 1024px) 100vw, 1024px"

This means for any situation in which the image is displayed wider than 1024px (so pretty much all instances where a laptop / desktop display is in use), the browser will load a 1024px wide source image and stretch it causing blur.

What's the plan to get this merged. What can I do to help?

@mor10

This comment has been minimized.

mor10 commented Nov 29, 2018

A request has been made for a test to show current and corrected behavior, so I've built them both:

The two posts linked below show the current output from core and a corrected version respectively. Note the detailed instructions on how to test this. Responsive images are a core browser feature and browsers work very hard to make the functionality invisible. Forcing visibility requires being rather heavy handed in your testing.

Take special note of the following: Responsive images are impacted by display density. When you do testing, make sure you test for both 1x and 2x displays. This can be done using the mobile preview in your browser dev tools.

@mtias

This comment has been minimized.

Contributor

mtias commented Nov 29, 2018

Thank you all for putting so much thought and care into this. I see two tangled concerns that should be separated to move forwards.

  • Avoid glaring breakage in 5.0 (like loading huge images).
  • Rethink how WordPress handles the media space in a world of flexible viewing experiences.

For the first, there needs to be a clear identification of any pending issues and agreement on what "broken" means. Specially around newly introduced features that don't carry any expectation from before (wide images, cover block, etc). My understanding is the main obstacle of loading too big a source is solved by defaulting to "large".

For the second, I very much believe we need to devote a whole cycle to media as we are carrying issues from a time when things were simpler. Now media exists in a world where the rudimentary assets we have in core are not enough — a plethora of devices accessing the web with different expectations, pixel densities, screen sizes, etc. This cannot be solved entirely by WordPress alone and would need participation from other groups — like hosting services and bandwidth management, browsers, web standard groups, etc. It is also unreasonable to expect to fix it all at the same time we are doing 5.0.

When we look at how WordPress manages assets by default, it's evident the variations it creates are not enough to power the wide range of devices, viewing conditions, and performance expectations. In the upper scale, we are dealing with 1024px or full-size on a regular install. This is not enough. There is a huge gap between "source" and "large" that makes almost any attempt at responsive images or hi-dpi support something of a losing battle. We need to have better segmentation of media assets without overloading servers for this to scale and match the quality expectations.

This is only further complicated by the fact these options are configurable by the user, themes, and plugins, so assumptions around what "large" would accommodate cannot go too far.

Being able to deliver the right image for the right context is crucial.

Ideally, the user doesn't have to intervene at all to make their creations look good and be performant.

However, our current standing proposal is not looking flexible enough at a technical or user experience level to fulfill this, while adding considerable overhead and complexity:

  • Scaling to other non wp:image blocks remains unsolved.
  • Unorthodox server implementation (likely needs something like #10108) that adds complexity.
  • Overhead in reproducing image markup ad hoc (server vs client), clashing with potential extensions and fidelity client side.
  • Unclarity in expected UX around percentage sizes.
  • Too much responsibility on the theme side.
  • More importantly, unclarity in overlap with phase 2 conditions and requirements.

In introducing this code and expectations (for themes, etc) we'll be creating a complicated web of requirements and missing the chance to take a more holistic approach to the problem. I believe this would be better done with time and in conjunction with phase 2, as the expectations of where a block lives and the user interactions will augment considerably.

Themes would no longer be able to merely say what widths they care about for the content, as blocks will be placed anywhere on a page. Blocks may be moved between block areas, too, so widths should always be contextual and controlled by the immediate InnerBlocks root. This includes things like columns within the main content area, but also areas outside the content itself. Any API we develop here needs to consider these expectations, as otherwise we'll be creating a web of legacy code that will be hard to untangle effectively.

What I see becoming a more robust and future-proof API is to attach media width responsibilities to each block container, of which post_content is just one. It could look like something like this:

innerBlocks( 'post-content', sizes: {
    default: 600,
    wide: 1000,
    full: 100vw,
}

This both specifies a default max-width and the availability of a wide alignment for the blocks under that root. As soon as you create another root (like columns), the context changes.

Which would allow us to also do:

innerBlocks( 'sidebar', sizes: {
    default: 300,
    wide: false,
    full: false
} )

and so on.

This should be paired with intrinsic responsive images handling, and more importantly, with better segmentation for asset creation or some sort of dynamic handling, if at all viable.

@joemcgill

This comment has been minimized.

Contributor

joemcgill commented Nov 29, 2018

Thanks @mtias,

I'm in full agreement with both the need to avoid breakages in 5.0, and the need to do a more high level focus on Media in WordPress to address these issues.

Regarding the former, I'm unsure that we've entirely addressed the issues that I outlined as necessary in #6177 (comment), specifically:

Save a more appropriate width to img elements in post content (best practice, but also relevant for calculating a default sizes attribute for responsive images).

If we can confirm that has been addressed, I would be comfortable with everything else being handled in 5.0.x releases, and beyond.

On the latter—rethinking media handling has been a hope of mine for some time, and is one of the reasons we had initial discussions about some of these high level problems during the community summit meetings during WCEU 2017 when Gutenberg was just getting kicked off. I look forward to resuming those conversations once the WP 5.0 release cycle has ended.

@getsource

This comment has been minimized.

Contributor

getsource commented Nov 29, 2018

Completely agreed with @joemcgill above ^

I would love to work together in the sort of high-level effort you're talking about, and it's something we've been talking about and wanting to work on for quite some time.

@mor10

This comment has been minimized.

mor10 commented Nov 29, 2018

Count me in. I think we (WordPress) are in a position to not only solve this for ourselves but create new models for media handling that helps the web as a whole. We could get involvement from browsers, hosts, CDNs, and standards bodies and use WP as the distribution channel for new best practices.

What we have encountered here is the absolute hard edge of the RICG responsive images spec. We now have a test case for what works and what doesn't work. This puts us in a unique position to move the work forward.

@mtias mtias modified the milestones: WordPress 5.0, WordPress 5.0.1 Nov 29, 2018

@chrisvanpatten

This comment has been minimized.

Contributor

chrisvanpatten commented Nov 29, 2018

@joemcgill Can you clarify this:

Save a more appropriate width to img elements in post content (best practice, but also relevant for calculating a default sizes attribute for responsive images).

What makes a "more appropriate" width? Just trying to understand the parameters of that particular item as I'm not sure I do now :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment