GTK applications display missing glyph instead space after emoji characters

[13rac1]

I thought this was a fluke earlier, but it's definitively not. The header in Firefox displays the missing glyph character in some instances. I may need to adjust the null character or add a space. Could be related to #1, but I doubt it at this point.

[edent]

Just to say I also see this issue.

[13rac1]

@edent I have confirmed the cause and a workaround for this issue, but I'm trying to figure out an ideal solution. Can you help?

The problem is: this font does not, on purpose, have a space character. The OpenType spec recommends all fonts have a space character: https://www.microsoft.com/typography/otspec/recom.htm There isn't one is because we want to fallback to the next font for all other characters. The space character has different widths in different fonts and font styles/weights. The width of a space is drastically different between a monospace font and a sans-serif font for example. This shouldn't be a problem (AFAIK), but FreeType(or something else in the font render stack) isn't switching back to the originally selected font after loading character(s) from Emoji One Color.

The workaround is to create multiple font files changing only the width of the space character, but that's ridiculous. The actual solution is, probably, to determine what is causing that extra space to load.

The DejaVu font family, primary default fallback on many Linux distributions, has five different space widths alone:

$ grep "\"space\" w" *
DejaVuSans-BoldOblique.ttx:    <mtx name="space" width="713" lsb="0"/>
DejaVuSans-Bold.ttx:    <mtx name="space" width="713" lsb="0"/>
DejaVuSansCondensed-BoldOblique.ttx:    <mtx name="space" width="641" lsb="0"/>
DejaVuSansCondensed-Bold.ttx:    <mtx name="space" width="641" lsb="0"/>
DejaVuSansCondensed-Oblique.ttx:    <mtx name="space" width="585" lsb="0"/>
DejaVuSansCondensed.ttx:    <mtx name="space" width="585" lsb="0"/>
DejaVuSans-ExtraLight.ttx:    <mtx name="space" width="651" lsb="0"/>
DejaVuSansMono-BoldOblique.ttx:    <mtx name="space" width="1233" lsb="0"/>
DejaVuSansMono-Bold.ttx:    <mtx name="space" width="1233" lsb="0"/>
DejaVuSansMono-Oblique.ttx:    <mtx name="space" width="1233" lsb="0"/>
DejaVuSansMono.ttx:    <mtx name="space" width="1233" lsb="0"/>
DejaVuSans-Oblique.ttx:    <mtx name="space" width="651" lsb="0"/>
DejaVuSans.ttx:    <mtx name="space" width="651" lsb="0"/>
DejaVuSerif-BoldItalic.ttx:    <mtx name="space" width="713" lsb="0"/>
DejaVuSerif-Bold.ttx:    <mtx name="space" width="713" lsb="0"/>
DejaVuSerifCondensed-BoldItalic.ttx:    <mtx name="space" width="641" lsb="0"/>
DejaVuSerifCondensed-Bold.ttx:    <mtx name="space" width="641" lsb="0"/>
DejaVuSerifCondensed-Italic.ttx:    <mtx name="space" width="585" lsb="0"/>
DejaVuSerifCondensed.ttx:    <mtx name="space" width="585" lsb="0"/>
DejaVuSerif-Italic.ttx:    <mtx name="space" width="651" lsb="0"/>
DejaVuSerif.ttx:    <mtx name="space" width="651" lsb="0"/>
$ cat *.ttx | grep "\"space\" w" | sort | uniq | wc
      5      20     221

Apple sets it to zero:

$ grep "\"space\" w" Apple\ Color\ Emoji.ttx 
    <mtx name="space" width="0" lsb="0"/>

I've generated a font with a full-width space character to test. The space is amusingly obvious in the header:

And a problem when a sans-serif font is requested:

[13rac1]

Haha! It happens in more than just the application titles, tool tips and taskbar in Ubuntu 14.04:

Pluma is set to use "Monospace". The font render system shows space from the previous font until any other character appears.

[13rac1]

I've narrowed down the cause of this text display problem (other than the missing space in the font) and decided on a best case solution for now without going further down this rabbit hole.

I tried a Kubuntu 15.10 LIveCD:

No incorrect space characters in the taskbar. Hmm, is this GTK vs QT?

I tried a few QT vs GTK applications on my install, then ran a QT text field demo and a GTK text field demo:

There we go. I've already looked into this issue enough, so I'm done for now.

Best case solution given current Linux distribution defaults is to make two versions of the font. One matching the width of spaces in DejaVu Serif/Sans-Serif, and the other matching DejaVu Monospace.

I'm going to leave this issue open for a while before making these two additional fonts to see if anyone has a better solution. Either way, the ideal solution is far outside of my control in this font.

[13rac1]

According to the Fontconfig docs it seems possible to exclude a <range> of unicode characters in a font from being used. Emoji One Color doesn't need to be the default font if DejaVu can be overridden for a few unicode ranges. I'm having trouble figuring out what configuration is needed, plus this GTK bug (right? It's not a problem in OSX, Windows, or QT) would probably still cause the same problem.

Potential temporary workarounds:

1. Create two versions of the font for Linux, one using the 651 space width of DejaVu Serif & Sans-Serif and one using the 1233 space width of DejaVu Sans Mono. Modify the fontconfig fonts.config recommendation to specify the correct fallback. Doubles disk space use, but would work. Technically would have the incorrect spacing for some fonts.
2. Create DejaVu Serif, Sans-Serif and Monospace fonts without the emoji we need to override.
3. ???

@IBBoard Any other ideas?

[IBBoard]

I've had a look at the Fontconfig documents and it isn't clear. From the tags listed in the docs then having <range> with a value that is above the ASCII range seems like it should work to say "Emojione only supplies things outside this range", but there are no decent examples for using it that I've found and trying to add it in where I think it should go just breaks the standard overrides!

(Also, sorry for the dupe. I don't know how I missed this ticket!)

[13rac1]

I've had a look at the Fontconfig documents and it isn't clear.

+:100: It works really well, but the config is so complex.

(Also, sorry for the dupe. I don't know how I missed this ticket!)

No problem. I'm always careful when closing issues as dupes, because I don't want to deter contributions/help. I appreciate other people looking into this issue.

Bug description
When GTK (pango?) renders a unformatted text string with characters from multiple fonts using the font fallback system, it uses the whitespace character from the last used font until it encounters any other visible non-whitespace character. If the last selected font doesn't have that whitespace character an error character is displayed.

I've confirmed this is a problem for nbsp 0xa0 too, so the font needs all whitespace characters.

Relevant bugs on gnome.org:

• Font selection for space characters RESOLVED FIXED: https://bugzilla.gnome.org/show_bug.cgi?id=145275
• Space size problems NEW: https://bugzilla.gnome.org/show_bug.cgi?id=416526
• invisible unicode chars are visible next to hebrew chars RESOLVED FIXED: https://bugzilla.gnome.org/show_bug.cgi?id=337821
• Renders U+200B (zero width space) visibly under certain conditions RESOLVED FIXED: https://bugzilla.gnome.org/show_bug.cgi?id=472657
• spaces are half as long after a unicode symbol NEW: https://bugzilla.gnome.org/show_bug.cgi?id=757785 This looks like another effect of the same problem

[13rac1]

I've updated the spaces are half as long after a unicode symbol gnome.org bug with a screenshot and relevant details (aka nothing about this font, because it's not relevant to the specific issue.)

Workaround Plan: I'm going to make two versions of the font for Linux. One with monospace whitespace characters and one with the normal DejaVu Sans lengths. Slight change to the font.conf and the error characters will disappear.

[crepererum]

Might be related (otherwise I'll open an extra issue):

Environment

Application: Evolution (GNOME 3.16)
Global configs (GNOME):

• Font Scaling Factor = 0.8
• Window Scaling (HiDPI) = 2

Screenshots

Without the fontconfig file:

With the fontconfig file (from project README):

[13rac1]

@crepererum Yes, although I haven't seen this specific issue it looks like a manifestation of the same issue. I'll be creating a set of fonts to workaround this issue with GTK(pango). Please hold off on a separate issue until then. Thank you!

There may be a fontconfig conf solution so Emoji One Color can only be the default font for the specific unicode ranges it provides, but I cannot figure out what arcane XML magic is required. So any help on that is appreciated. Haha! There isn't see #17

[IBBoard]

@eosrei That's what I was looking at. The obvious <range><int>128</int><int>100000</int></range> just seemed to break the override and I hadn't found a schema, just the vague docs without examples!

(Unless, of course, I'm misunderstanding the documents and it has no way of specifying Unicode ranges to exclude - but then <blank> sounds like it should already ignore our bad blanks, which I guess it does everywhere but GTK/Pango)

[13rac1]

@IBBoard I'm fairly sure that, even if we can determine a correct fontconfig, it won't fix this issue due to the bug (design "feature"?) in Pango/GTK. 😞

Note to anyone who may have something to add to that bug report: Please don't mention this font, it'll complicate the resolution.

[13rac1]

Status update

There are 25 kinds of whitespace in unicode:https://en.wikipedia.org/wiki/Whitespace_character#Unicode I am creating two "starter" fonts. One using the DejaVuSans whitespace widths and the other using the DejaVuSansMono widths. I'll add a new option to SCFBuild to accept an input file, then generate two versions of the font for Linux and make a zip file with both.

[13rac1]

The OpenType Spec recommends a font have the space character, but doesn't require it. Reference: https://www.microsoft.com/typography/otspec/recom.htm IMO, all of these programs expecting a space are at fault. Haha! it's a bit amusing really, but I wish I were wrong. I am going to add all of the recommended characters to both fonts to avoid this.

[13rac1]

This issue is "fixed" by adding a space character with a width equal to DejaVu and Bitsteam Vera Sans then making Bitstream Vera or Noto the default font family. Bitstream Vera contains no emoji characters, and Noto contains very few. See soon-to-be-recommended-default fonts.conf here: https://github.com/eosrei/emojione-color-font/blob/245e659fe129eeceb1381d726c9ea717abcf6a73/fontconfig/user-bitstream-vera-fonts.conf New v1.0-beta3 release coming up soon.

[IBBoard]

So wait, let me get this right - this new config will default the text to Bitstream fonts rather than whatever we've set in (for example) Gnome Tweak?
Fixing the font issues is good, but I like my Liberation Sans! I'll have to try fiddling with the configs.

[13rac1]

@IBBoard You can change it to whatever you want, but I need to create/produce/distribute a working "standard" out-of-the-box install method. I've looked at fontconfig for hours and hours and I'm so over it. :-/ Bitstream Vera is the only solution I've found other than forking DejaVu/Noto/Liberation since they all contain emoji characters. Glyph blacklist (my initial hope in #17) has been a wishlist item for 10+ years. Ah yes, Liberation contains: ☺0x263A. So, I cannot recommend it either.

whatever we've set in (for example) Gnome Tweak?

AFAIK Gnome Tweak does not change the fontconfig. I recommend you install my provided conf, then adjust that for your specific needs. Beta3 cannot be your default font because it includes a space character and will break monospace formatting. Not having a space character is what caused this issue to exist, so it's sadly not an option either.

I'm open to any other solutions!

[IBBoard]

From a bit of poking then I don't quite know how Gnome Tweak does it. Replacing "Bitstream Vera" with "Liberation" (then replacing "Sans Mono" with "Mono") seems to work. I can managed with the white smiley face being B&W - it's the tiny one that I try to avoid anyway, and most people probably use the nearby one that's full-size.

I know what you mean about fiddling with Font Config, though. I didn't do anywhere near as much as you've gone and it was a pain. It seems like you should be able to do things (like selectively pick fonts for certain character ranges) but none of it is documented and none of it works.

Thanks for all the work!

[13rac1]

I can managed with the white smiley face being B&W - it's the tiny one that I try to avoid anyway, and most people probably use the nearby one that's full-size.

Haha, it's actually a full size emoji in the source and works correctly with Bitstream Vera. I may create/document how to subset the other fonts in the future, but it's not something I want to deal with right now.

(like selectively pick fonts for certain character ranges)

Nope! It's only a feature request! 😭

Thanks for all the work!

Sure! Thanks for testing/using it.

[13rac1]

FYI The GTK/Pango issue has been acknowledged and the creator of Pango has described a good solution. Not sure the resolution will make it downstream to current distributions, but it'll be corrected in the future. 🎉 😄

If font before and after space are both the same and they support space, then we should use that. However, if the font used for before and after are different, then currently we choose the one used for before. However, we should use the one between before/after that has a higher priority. Fixing that will fix this case.