BOM marker generates zero-advance gid 3 (space glyph) when shaping

[HinTak]

If you compare the result of hb-shape with the src/sample.py before and after this change:
d3e2a06

You would notice that somehow a space is added at the beginning of the buffer, with the shaping result showing an extra item of gid3 on one side (depending on whether you are trying to shape Arabic or not)

[HinTak]

Noticed this issue trying to extent the python example in the direction of hb-shape/hb-view to address #79

[khaledhosny]

That is because text.encode('utf-32') and text.encode('utf-16') insert BOM character at the beginning of the string.

[HinTak]

That may be the case, but should the shaping code treat BOM as a space character???

BOM marker is not a space character, as you should know that.

[khaledhosny]

That is how HarfBuzz handles default ignorable characters, the get converted to the glyph for space character and its width is set to zero.

[khaledhosny]

This way, they work and are invisible even if the font has no glyphs for them or the glyphs are not blank (which is the case for many fonts).

[HinTak]

That sounds a bit wrong. There are probably special language/usage where the space glyph is not inkless...

[anthrotype]

@HinTak if you're interested in python bindings for harfbuzz you may want to also have a look at https://github.com/trufont/uharfbuzz (shameless plug)

[HinTak]

I just thought of one such usage: in the era of color font, a space glyph might be of a plain block of colours. In that case, inserting a space with zero advance could be visible.

@anthrotype : thanks for that plug - I appreciate that. On my quick search I found the gobject inspection one and a project from the same guy who tries to rewrite pycairo in rather odd ways, and a few other projects of not very noticeable maturity. I didn't find/notice the Trufont one. Mainly I am on the gobject introspection one (hence all the issues/pulls in that area) . Already rewritten a good part of hb-shape/hb-view in python. Though I'd be happier with ctype/cffi than gobject...

[anthrotype]

yeah i don't really understand gobject-introspection. The one I linked is wrapped with cython.

[HinTak]

Well, gobject is similar to ctype in that you write stub code telling python (or a non-C language) how to talk to a c-based library. The difference with gobject is that, the bits of "stub code" happens as functional comment blocks next to the c code on the c code's side, rather than on the python side. Those comments could set default parameter values for non-C languages, re-arrange arguments (e.g. The python routine receives multiple results as a list when the C code have output pointers to be filled in) and other tasks.

So perhaps the oddity you notice is due to the fact that the foreign (non-c) interface is declared on the C side, and may not necessarily feel natural on the python side.

[anthrotype]

it's also not well supported on windows -- at least the last time I tried I couldn't make it work. In my cython bindings, I have cmake build a static harfbuzz library from source and embed that in the python extension module so it's self-contained. Anyway, we're digressing :)

[HinTak]

@khaledhosny : I assume you mean zero advance rather than zero width. The problem is you assume that space glyph is unmarked. Let's say you have a color font where the space glyph is a block of plain color. Harfbuzz insert a space for BOM. In Arabic layout, the BOM/space is rearrange to the right. Then you have a visible block of space-colour on the right to your Arabic phrase.

Why is harfbuzz not skipping BOM? That sounds more sensible.

[khaledhosny]

In case of BOM, the client should probably not pass it to HarfBuzz in the first place, so that is an edge case. For the general issue, IIUC, HarfBuzz never skips characters, any character in the input must be accounted for to have reliable mapping between output glyphs and input characters. But anyway, I’m just describing the cuurent HarfBuzz behavior, I neither designed it nor I’m the one you need to convince to change it (if it is ever going to happen).

[behdad]

I thought there's an issue for it. Maybe it was just in the TODO for a while. Anyway, open an issue for API to set "invisible-glyph-id".