Add api to set invisible glyph

[behdad]

Currently we use space glyph. Add api to set it differently. We want three different semantics:

• Use space glyph. Current default.
• Delete.
• Specific number, like 0xFFFF.

What's a good API design for that? Oh wait, we already have @HB_BUFFER_FLAG_REMOVE_DEFAULT_IGNORABLES and @HB_BUFFER_FLAG_PRESERVE_DEFAULT_IGNORABLES. So, just need API to support:

• Use space glyph. Current default.
• Specific number, like 0xFFFF.

Maybe we can use 0 to mean "use space glyph", as 0 is otherwise notdef and shouldn't be desirable?

[HinTak]

I have seen some devanagari font recently where .notdef is a tofu (Google lingo for rectangular box).

[khaledhosny]

hb_buffer_set_ignorables_replacement_codepoint () (or something like that), similar to hb_buffer_set_replacement_codepoint ()?

[behdad]

hb_buffer_set_invisible_codepoint?

[twardoch]

@HinTak .notdef is supposed to look like “tofu” and mostly does — tofu is exactly the slang term for .notdef, see https://docs.microsoft.com/en-us/typography/opentype/spec/recom#glyph-0-the-notdef-glyph

@behdad The old PFM font structure had a field that had a similar role:

BYTE dfDefaultChar

This field specifies the default character to be used whenever a character is used that is outside the range of the dfFirstChar through dfLastChar fields in the PFMHEADER structure. The character is given relative to dfFirstChar so that the actual value of the default character is the sum of dfFirstChar and dfDefaultChar. Ideally, the default character should be a visible character in the current font, e.g. a period ('.'). For text fonts this field is normally set to either 0x00 (character space) or 0x75 (bullet).

— https://web.archive.org/web/20040708162047/homepages.muenchen.org/bm134751/pfm_fmt_en.html

This field is also in the Win32 API:

https://docs.microsoft.com/en-us/windows/desktop/menurc/fontdirentry

[khaledhosny]

There is OS/2.usDefaultChar, which I think is the same thing as the PFM stuff above. However, this is basically a way to control glyph id used for unsupported character (default in HarfBuzz is usually 0 or whatever the font function return for unsupported characters), but this is not what we are after here.

[HinTak]

If I remember correctly, mostly of the postscript type 1 font I looked at years ago had .notdef unmarked. That was one of my old complaints with latex - texts/maths going missing when the font I choose hasn't got a representation of that glyph.

Anyway, except for some useful cases where multiple code points maps to the same glyph (there are notable cases where multiple code points of different meanings have the same visual shape), the code-point to glyph mapping/shaping should be reversible...

[HinTak]

Still playing with my python hb-view rewrite prototype, and see that beside the beginning/end, if you shape the Persian word for harfbuzz in Arabic, right in the middle, it puts a space glyph with zero advance in.

[behdad]

This is all expected. Gah.

[jfkthame]

Now that this landed, I suddenly wonder whether it's incorrectly named. Seeing "invisible codepoint" in the API, and especially in the hb-shape/hb-view options, it sounds like it could refer to a Unicode character value. So a user might expect it to be looked up in the cmap and the resulting glyph used in the buffer. But it's not; the value is used directly as a glyph ID.

To avoid such confusion, should we rename the function and option to hb_buffer_set_invisible_glyph etc?

[behdad]

Now that this landed, I suddenly wonder whether it's incorrectly named. Seeing "invisible codepoint" in the API, and especially in the hb-shape/hb-view options, it sounds like it could refer to a Unicode character value. So a user might expect it to be looked up in the cmap and the resulting glyph used in the buffer. But it's not; the value is used directly as a glyph ID.

To avoid such confusion, should we rename the function and option to hb_buffer_set_invisible_glyph etc?

Right. This is why I was hesitating on adding this API. For "replacement_codepoint", I called it that instead of "replacement_character", because common values use might NOT be valid character numbers. Same here, values used (0xFFFF, -1, ...) might NOT be valid glyph indices. That's why I just stuck with "codepoint". WDYT?

[jfkthame]

Well, the value might not be a valid glyph index, but it's being used as if it were. It gets put in the output buffer, in place of whatever glyph index the invisible character originally mapped to; if the user/client provides something that isn't a valid glyph index, it'll be up to the client to handle that appropriately when rendering the glyphs (or whatever it wants to do with the shaped text).

I think "glyph" (or "glyph index") is better than "codepoint" here because (IMO) "codepoint" implies a slot in a character encoding. I realize HB (re-)uses the hb_codepoint_t type for glyph indices as well, but I think that's an idiosyncratic feature of HB rather than a common understanding of the terminology. Arguably, the input buffer should be filled with hb_codepoint_t values representing Unicode characters, and the output buffer should return an array of hb_glyph_index_t values.

(I'm not suggesting it's worth actually creating a distinct type for glyph indexes in HB. But using the term codepoint in naming an API that actually takes glyph indexes feels awkward to me.)

[behdad]

Ok, I'll rename to 'invisible_glyph'.