Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Images with layout information #4116

Open
litherum opened this issue Jul 14, 2019 · 26 comments

Comments

@litherum
Copy link
Contributor

commented Jul 14, 2019

There are quite a few examples of images which are associated with additional layout information. When I say "layout information" I'm talking about situations where the images are more than just simple width × height boxes.

MathML

From the minutes of the Math on Web Pages Joint Meeting at the Lyon F2F:

arno: Moving on to another challenge
arno: Baseline alignment
...
arno: Needed for lining up equations

Consider a math equation like this:

ex51

If it's presented inline in the middle of a paragraph, proper typesetting would align the baseline of the line it's on with the baseline of the 𝑓(t)dπ(t) part. However, that isn't the lowest part of the content; with naive typesetting, the bottom of the content would lie upon the baseline and the whole formula would be too high on the line of the paragraph.

Logos

Consider the Apple Pay logo:

Screen Shot 2019-07-14 at 4 13 10 PM

Similar to above, part of this logo should descend below the baseline. If you treat this logo as a simple image, the whole thing would be moved up too high, and wouldn't appear correctly on the line.

Currently, web authors often use icon fonts to work around this, as glyphs in fonts are allowed to have descenders. However, using icon fonts is an anti-pattern on the web. Instead, there should be a way to associate this baseline information with the logo itself.

Icons & Symbols

Consider the Play icon in the Music app on iOS. It is a little triangle, but when you have your finger pressed on it, there's a little circle surrounding the triangle.

IMG_0054

If you get your ruler out and measure, it turns out that the Play button is not mathematically centered in the circle. Instead, it's optically centered, using the overshoot concept from typography. Since the right point of the triangle is much more narrow than the left edge of the triangle, the right point gets to extend farther outward from the center.

The facilities for this are present in the "Symbol Images" concept natively available on macOS and iOS. An artist can associate a symbol with hand-authored baseline information. This allows the system to place these icons properly.

Kanbun

Not all CJK characters are encoded in Unicode. These characters are usually represented with images. Even though, technically, CJK doesn’t have descent, in practice, CJK characters end up having descent in real fonts because this is the only way to have proper vertical alignment on lines with mixed scripts. When a kanbun image is used, it cannot currently have a descent, meaning it cannot be placed vertically correctly. This is particularly bad in vertical writing mode (which is common in CJK), where the baseline is in the middle of the line, meaning the kanbun image would naturally be in a completely incorrect place.

Conclusion

CSS should have facilities to allow for this kind of layout information associated with images. Layout engines should allow designers to place or tweak layout information such as baselines or vertical baselines for their image artwork.

There is currently no file format for associating layout information with images. So, there are a few ways this could possibly work today:

  1. SVG is modified to allow an extra attribute on the root-most SVG element, which can indicate to the containing layout content additional layout information
  2. System assets, such as Symbols on macOS and iOS, could be exposed to the web. These assets would have layout information built-in. E.g. background-image: system-ui(play-button); background-position: auto;
  3. An additional CSS property that could modify the baseline of an image. E.g. image-baseline: 20% 50%; (for horizontal and vertical baseline). This would do more than background-position; it would actually move the layout box of the image.

Regardless of which method is chosen, CSS layout would have to be modified to include affordances for this additional layout information.

@litherum

This comment has been minimized.

Copy link
Contributor Author

commented Jul 14, 2019

@emilio

This comment has been minimized.

Copy link
Collaborator

commented Jul 14, 2019

If the main use-case for this is vertical-alignment, you can do that already (for inline images or SVGs), can't you?

data:text/html,Foo <img width=10 height=10 style='background: green; vertical-align: 75px'> bar

@emilio

This comment has been minimized.

Copy link
Collaborator

commented Jul 14, 2019

Oh though I guess you're saying the <img> should contribute its own baseline somehow?

@litherum

This comment has been minimized.

Copy link
Contributor Author

commented Jul 14, 2019

Yes. The image should be able to contribute its own baseline, and thereby be able to affect the size of the line box.

@othermaciej

This comment has been minimized.

Copy link
Member

commented Jul 14, 2019

It would be good if the baseline info could be contained in the image without needing external style info (which might put this partly beyond the realm of CSS, but CSS could describe the semantic for such a thing, and could have a style-level override).

@AmeliaBR

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2019

I do like the idea of exposing baselines as metadata on an SVG, and then using that for the default inline alignment when the SVG is a replaced object in CSS (e.g., an , or included via CSS content or list-style-image). For inline SVG text, I often make the SVG layout box only extend down to the baseline, and then use overflow: visible to get the descenders. But that's not an option for replaced content.

One concern is that this could be a new bit of information exposed from inside the file to the outside layout, and therefore we'd need to consider whether crossorigin restrictions should apply.

I'm not sure I understand how you're proposing this would interact with background position.

Do you know of any other image formats which have alignment metadata baked in? Is the Apple "symbol image" format standardized in any way that allows it to be used by other applications? (I notice they recommend importing from SVG for any customizations.)

As far as having access to system icons from CSS, I think that is an interesting idea that should be discussed in a separate issue. We'd need to look at whether there is a common set of icons available to apps in different platforms, and consider how it integrates into fallbacks (e.g., the image() function).

@AmeliaBR

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2019

CSS could describe the semantic for such a thing, and could have a style-level override

The style override already exists, in the vertical-align property. So it really comes down to CSS saying that replaced inline boxes can have baselines if the content inside them defines them to exist. Then the layout would behave like an inline-block CSS box.

Also relevant: CSS Inline 3 has a note about a possible baseline-table property that could define multiple baselines on an <img> or similar element, separate from the vertical-align property that defines which one to use. For SVG, we could use the same property on the root element, and then define that the auto behavior of baseline-table is to use the values from the embedded file if they exist.

@othermaciej

This comment has been minimized.

Copy link
Member

commented Jul 15, 2019

A property specifically for defining baselines seems like the right idea (although baseline or baselines seems like a more appropriate name than baseline-table).

@Crissov

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2019

@frivoal considered something similar for images of rare (East Asian) characters that are supposed to blend into text seamlessly. #4013 “gaiji”

@dbaron

This comment has been minimized.

Copy link
Member

commented Jul 15, 2019

This seems like reasonable thing to me. It seems like there are two things being discussed:

  • image formats with optional baseline information, which CSS would then honor by default (when the information is present) instead of assuming that the (alphabetic?) baseline is at the bottom margin edge
  • a CSS mechanism for specifying where the baseline(s) of an image are

It's not clear to me which would be preferable (or whether it would be best to have both).

@faceless2

This comment has been minimized.

Copy link

commented Jul 15, 2019

As it happens we've been considering this problem for some time as well.

Our goal was to improve layout for SVG embedded within HTML - for example, to insert text following a path in SVG as an inline element within HTML, and have the baselines match up:

<!DOCTYPE html>
<html>
<head>
<style>
p, text { font-size: 30px }
svg { display: inline; width:80px; height: 40px }
path { display: none }
</style>
<body>
<p>
Text on a
<svg>
<path id="curve" d="M 10 40 C 30 20 40 20 70 40"></path>
<text id="curveText"><textPath href="#curve">curve</textPath></text>
</svg>
and back to normal
</p>
</body>
</html>

image

Our plan was for an additional property on the SVG element which is the id of its descendant with the baseline to align on externally. So, in the example above we could have something like <svg baselineElement="curveText"> (syntax, property name etc all just for illustration). This would work for MathML or other embedded XML syntaxes where internal text is laid out with CSS, and for elements with a shadow DOM. We hadn't considered bitmaps.

(The second part of the problem was an auto-size property for SVG, something like <svg style="width:content-fit; height:content-fit>", to size the element to its visible boundaries. Not relevant for baselines though so I'll come back to that in another issue at some point).

@AmeliaBR

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2019

It's not clear to me which would be preferable (or whether it would be best to have both).

I'd say both. If it is possible to specify the alignment points inside the image file, that makes it much easier for authors to reuse the content & keep things synchronized. But not all images will have that extra metadata, so it would be nice to still be able to set the values from the outside document.

And as I mentioned above, for SVG in particular it might make sense to reuse the same property for defining the alignment points on the root element of the SVG file. (Although the proposal mentioned by @faceless2 is an interesting alternative for SVG that contains text internally.)

@upsuper

This comment has been minimized.

Copy link
Member

commented Jul 21, 2019

I think it would be the best to be provided by the image itself (mechanism to be explored, e.g. maybe new attribute in svg, and new table in png).

I'm not sure how much it makes sense to have a CSS property controlling that. This is a piece of information bound to the content rather than its style, so the value would need to be specified for each specific image, and it's probably not very likely that you can just apply a value to some random content that you cannot control otherwise.

For images that doesn't have such metadata but you can manually provide it, you may either update itself or wrap it inside for example an SVG.

If we do want to provide some control in the page, given its closer binding to the content, it's probably better being an attribute than a CSS property.

Oh, and maybe <canvas> and <video> etc. would want something alike.

@tabatkins

This comment has been minimized.

Copy link
Member

commented Jul 22, 2019

Even if we only add it to SVG, that would technically be sufficient for everything else, since you can always just do <svg><image href="..." /></svg>.

I agree that this information best belongs in the image itself.

@faceless2's suggested approach looks very similar to what we've talked about for allowing an HTML subtree to override its normal baseline calculations. Sounds reasonable to me.

Note that, for bitmaps (using an SVG image), you can always embed invisible text to use for baselines.

The nice thing about deferring to a textual element is that you get all the baselines for free, not just one.

@AmeliaBR

This comment has been minimized.

Copy link
Contributor

commented Jul 22, 2019

Note that, for bitmaps (using an SVG image), you can always embed invisible text to use for baselines.

The nice thing about deferring to a textual element is that you get all the baselines for free, not just one.

If you're creating artistic text using SVG paths, then even if you add invisible text for accessibility purposes, it's not going to have the correct set of baselines for alignment. If you're creating icons, then a random bit of invisible text isn't going to be useful for either accessibility or defining baselines. So I wouldn't want to defer to embedded text for baselines unless that's the actual visible text that we want to align.

That said: If CSSWG decides that there isn't a need for a general CSS property to define baseline tables for a replaced element, we can discuss SVG-specific ways of defining baseline tables in SVG-WG (or SVG-CG).

@css-meeting-bot

This comment has been minimized.

Copy link
Member

commented Jul 29, 2019

The SVG Working Group just discussed Images with layout information.

The full IRC log of that discussion <krit> topic: Images with layout information
<krit> GitHub: https://github.com//issues/4116
<krit> myles: there are different ways to get the effect
<krit> myles: images could have additional information inside
<krit> myles: logos are often presented in SVG.
<krit> myles: maybe we could add a attribute to the SVG root.
<krit> myles: there is another option like system artwork which have their own information that should be used.
<krit> krit: there was a proposal to the CSS WG which was called something like Content Aware Zoom or Scale. The idea was to specify an area that must be focused. Depending on the space, more or less of the image gets displayed. We should look if your proposal can be aligned with it. I do support working on it.
<krit> AmeliaBR: We should not only look at baseline alignment. And coordinate with CSS.
<krit> AmeliaBR: If we talk about text layout we have very different properties.
<krit> krit: consider bringing it up to SVG CG
<krit> AmeliaBR: we should look if there is a cSS proposal first and then we should look at SVG and where we would work on it.
<krit> trackbot, end telcos
<trackbot> Sorry, krit, I don't understand 'trackbot, end telcos'. Please refer to <http://www.w3.org/2005/06/tracker/irc> for help.
<krit> trackbot, end telcon
@progers

This comment has been minimized.

Copy link
Contributor

commented Jul 29, 2019

I'm trying to work through what this would actually look like if we could specify baseline. I made a simple example:

<span style="background: lightblue; font-family: cursive; color: darkblue;">
  Testing an integral <img src="https://pr.gg/integral.svg" style="height: 1.6em; vertical-align: middle"> symbol.
</span>

Does the containing page needs to know more than just the baseline to correctly position an integral symbol? For example, what height should the page specify for the image? (possibly height: 1lh for one line-height?) Is just the baseline sufficient for the math usecase or are other font glyph properties needed?

Edit: overall, solving this issue (images with layout information) seems useful to me, and would not be difficult to implement.

@kojiishi

This comment has been minimized.

Copy link
Contributor

commented Jul 29, 2019

vertical-align: middle requires x-height too, for Latin context.

https://drafts.csswg.org/css2/visudet.html#propdef-vertical-align

@dirkschulze

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2019

Speaking about images and layout information: There was a proposal of "content aware scale" based on the feature in Adobe Photoshop a couple of years ago. Though I don't have the link to the actual proposal, here is a description of the feature https://helpx.adobe.com/photoshop/using/content-aware-scaling.html

Maybe horizontal/vertical alignment can be targeted together with content aware scale in one proposal.

@Crissov

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2019

Wouldnʼt content-aware scaling (i. e. removing or merging pixels or lines with the least information relative to their neighbors) “just” require a new value alongside cover and contain? I think that deserves a separate issue.

@dbaron

This comment has been minimized.

Copy link
Member

commented Sep 15, 2019

One note on the initial comment's section on Kanbun:

This is particularly bad in vertical writing mode (which is common in CJK), where the baseline is in the middle of the line, meaning the kanbun image would naturally be in a completely incorrect place.

I don't think this is the case, since in vertical writing mode, the image will end up centered, which is likely to produce the correct result. (This is at least interoperable in Gecko and Chromium.)

@stantonma

This comment has been minimized.

Copy link

commented Sep 16, 2019

I don't think this is the case, since in vertical writing mode, the image will end up centered, which is likely to produce the correct result. (This is at least interoperable in Gecko and Chromium.)

We also have a similar problem in horizontal CJK, since vertical-align: baseline aligns with the alphabetic baseline. However images of CJK characters are usually designed to align with the ideographic baseline. I believe currently most authors use negative margins to manually shift to an approximation of the ideographic baseline.

alignment-baseline probably fixes this for ideographic characters, since they usually don't have descenders below the ideographic baseline. However, to fully enable alignment for images of characters with descenders I believe we need the baseline information.

@css-meeting-bot

This comment has been minimized.

Copy link
Member

commented Sep 16, 2019

The CSS Working Group just discussed Images with layout information.

The full IRC log of that discussion <emilio> Topic: Images with layout information
<emilio> GitHub: https://github.com//issues/4116
<emilio> myles: I'd like to present a problem and no suggestions about how to solve it, but I hope to leave with a sense of whether we're interested in solving it
<emilio> myles: there are images that need to be positioned similarly to text
<emilio> myles: like a math formula with a tall integral sign where most of the formula should be aligned to the baseline but the presence of the integral sign the whole image will sit on the baseline
<emilio> myles: so the formula won't align with the text
<fantasai> if we're talking about things like gaiji, which is inline images that should have a baseline and stuff... there was some proposal to add a property to specify a baseline table
<fantasai> we didn't spec it out because it wasn't a high priority
<emilio> myles: it'd be cool if the math formula could say "my baseline is in the middle of the image", and layout could align it correctly
<fantasai> but it's certainly possible to do
<emilio> myles: similarly with the apple pay logo and similar, because of the descender of the y
<emilio> myles: a similar case is for symbols like the play button in the ios music app, which is not fully horizontally centered, but visually centered
<emilio> myles: it's a triangle, so if you mathematically center it horizontally it would look wrong
<emilio> myles: so the layout moves it slightly horizontally
<emilio> myles: same for kanbun, which are cjk chars which are not in unicode and people use them using images
<emilio> myles: since they're images they behave wrong
<emilio> myles: three of the examples show the need for horizontal baselines, the other is a vertical baseline
<fantasai> s/kanbun/kaiji
<fantasai> s/kaiji/gaiji/
<tantek> There's another fairly common use-case of this, images used for decorative first/initial letters
<emilio> myles: there are various ways to do this, one option is using it on the image formats and such to provide the information
<emilio> myles: another option is to add a css properties
<emilio> myles: mac has these system assets that contain this information
<nmccully> unencoded glyphs = gaiji; Chinese poetry typeset for Japanese audiences = kanbun
<emilio> myles: another option would be to provide a css function that takes a name to these system assets, and returns the information
<emilio> myles: so at least we could use it for system assets
<emilio> myles: I wanted to get an indication of whether this is a problem worth solving, and if it is what's the solution could look like
<tantek> q+
<emilio> TabAtkins: chrome is fine with that, we have taken a look to the image formats to provide x-height and baseline
<smfr> ack
<stantonm> q+
<tantek> q+ to note use-case of decorative image first/initial letters and also prior art / related feature CSS cursors which reference the hotspot of the cursor image
<emilio> nmccully: a single baseline is fine, but multiple baselines may be needed for multiple writing systems
<heycam> q=
<astearns> ack tantek
<emilio> ack tantek
<Zakim> tantek, you wanted to note use-case of decorative image first/initial letters and also prior art / related feature CSS cursors which reference the hotspot of the cursor image
<heycam> q+
<smfr> q-
<emilio> tantek: I agree this is worth solving. Another use case is a decorative image for the first-letter which has decorations and borders alone
<emilio> s/alone/and such
<emilio> tantek: we have a similar feature / challenge in css-ui with the cursor property
<krit> q+
<emilio> tantek: cursor has a hotspot. May be something that we could reuse somehow
<emilio> TabAtkins: cursors are usually set once, but in this case you need the same information every time you set the image
<chrishtr> q+
<astearns> ack stantonm
<emilio> tantek: I think cursors can also have the same issue. Having the info in the image would be great, but maybe both
<emilio> TabAtkins: both is fine
<emilio> stantonm: we have a similar use case for @font-face / images and cjk, where adding a baseline to the font-face rule would be useful
<emilio> myles: can you link me to it?
<emilio> stantonm: we also have a use case to get rid of these images and use the font, so we wanted to use @font-face with the images, and for that the baseline would be great, but for the symbols it may not make sense
<fremy> @myles: if you want more details about the CUR format wrapper which can contain a PNG image and specify a hotspot: https://en.wikipedia.org/wiki/ICO_(file_format)#Icon_resource_structure
<emilio> stantonm: another thing we wanted to see if we can treat images more like a bitmap, to invert in dark mode
<emilio> myles: mix-blend-mode?
<emilio> stantonm: haven't looked into impl that much
<emilio> heycam: I wanted to express general agreement on the problem being worth solving
<emilio> heycam: have run into this with svg
<stantonm> creating @font-face from an CJK image (gaiji) - https://github.com//issues/4013
<astearns> q?
<astearns> ack heycam
<emilio> heycam: I think modifying image formats would be useful, though I'm concerned for compat if we run into a situation with image-orientation
<emilio> heycam: where we start reacting to image metadata and breaking pages
<emilio> myles: we still need some css integrations to define how these are read or what not
<tantek> FYI CSS UI cursors with optional hotspot x y that can override any built-in hotspot: https://www.w3.org/TR/css-ui-3/#cursor
<astearns> ack krit
<emilio> heycam: using vertical writing mode to getting centering and alignment seems a bit odd
<emilio> krit: file formats seems nice, but there are formats which we may not be able to modify
<emilio> krit: so we may want a css-only solution as well
<astearns> ack chrishtr
<emilio> chrishtr: what formats have this already, any examples?
<emilio> myles: nope, 0
<emilio> myles: the ui image in iOS apps have this, but it's not a file format
<bkardell_> q+
<emilio> chrishtr: so in an iOS app you create an image and set this and use it in your layout?
<emilio> myles: yes
<emilio> chrishtr: I'm curious about how feasible is it to modify the formats
<emilio> myles: people think it's a good idea so I'll try and come back if fail
<fantasai> myles, I think we need to make sure whatever we do can handle multiple baselines
<emilio> fremy: we may not need to change formats, the cursor format is a small wrapper file with some metadata. May be able to use a similar wrapper-format
<fantasai> e.g. alphabetic vs ideographic vs central vs mathematical
<emilio> chrishtr: you're proposing a new format?
<nmccully> Adobe's SING glyph format was one way to make an image with font metrics : https://web.archive.org/web/20080627183635/http://www.adobe.com/devnet/opentype/gdk/topic.html
<emilio> fremy: windows cursors do exist already, and include the hotspot. It already has the hotspot, but we don't want exactly that
<emilio> fremy: so we may want that instead
<astearns> ack bkardell_
<chrishtr> q+
<emilio> fremy: as changing formats is going to be a pain
<emilio> myles: only interested in jpeg and png really, and those do have optional tags
<emilio> myles: but yeah, first thing to try is changing those, second the wrapper format, third css-only solution
<myles> optional tags inside their existing formats
<emilio> bkardell_: We need to specify what do we do with these values, and people thought that a css solution would also be valuable too
<emilio> bkardell_: so it seems like we'd get into a situation where there are multiple sources of truth
<emilio> myles: image-orientation was like that until not long ago
<emilio> bkardell_: right, also aspect-ratio and such
<emilio> bkardell_: do you think a property of that sort would be valuable?
<astearns> ack chrishtr
<emilio> myles: I think the way I'd approach is the underline-{offset,thickness} where there is an auto value, then from-font (which would look and the image), then <length>
<emilio> chrishtr: we discussed something similar about mathml not long ago right?
<emilio> myles: I think the mathml proposal was to identify the baseline out of specific elements inside the formula
<emilio> myles: but that doesn't work for non-formula use-cases like the ones I've provided
<emilio> chrishtr: that makes sense
<emilio> cbiesinger: could potentially work for svg
<emilio> myles: yeah, true
<emilio> chrishtr: drott mentioned that fonts can be the wrapper format
<emilio> chrishtr: that may be a lot of overhead
<emilio> myles: I think making icon fonts is an anti-pattern
<emilio> drott: there are some ergonomic issues with them
<emilio> drott: but my point is that we don't need a new wrapper format, you could wrap an image in a font file
<emilio> myles: I'd prefer a new format, there's tons of unrelated stuff that you'd need to create a valid font
@w3c w3c deleted a comment from css-meeting-bot Sep 16, 2019
@Crissov

This comment has been minimized.

Copy link
Contributor

commented Sep 16, 2019

myles: a similar case is for symbols like the play button in the ios music app, which is not fully horizontally centered, but visually centered
myles: it's a triangle, so if you mathematically center it horizontally it would look wrong
myles: so the layout moves it slightly horizontally

Also see #1849 for visual or circular centering.

@css-meeting-bot

This comment has been minimized.

Copy link
Member

commented Sep 17, 2019

The CSS Working Group just discussed Images with layout information.

The full IRC log of that discussion <birtles> topic: Images with layout information
<birtles> github: https://github.com//issues/4116
<birtles> myles: when we discussed this, we talked about annotating images with baseline information
<birtles> ... is there an image format for subtitles?
<birtles> nigel: same image format as elsewhere
@litherum

This comment has been minimized.

Copy link
Contributor Author

commented Sep 22, 2019

UIKit models this information by using baselines. This is probably the most natural way to model this information in CSS, too.

I started discussing this in SVG in w3c/svgwg#739.

(One downside is: If we model it using baselines, it will be impossible to use both horizontal and vertical layout information at the same time)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.