FontForge is corrupting TrueType instructions in certain Chinese font that relies on them for correct display (SVTCA)

[v-python]

• Provide a general summary of the issue in the Title above.
• Before you open an issue, please check if a similar issue already exists or has been closed before.

Originally this was thought to be a bug on Firefox for Mac, because the symptom showed up there, and not on Windows or Android. So the bug submitted to Mozilla, and the helpful person that
responded there, helped us realize that although the generated woff2 file works on some platforms,
the fact that it doesn't on others is more likely related to the changes made by FontForge during the conversion, and that some platforms may be more picky about those changes than others.

I certainly don't understand font file internals well enough to understand the issue completely.

The firefox bug report contains the details, some of which I'll repeat below. Here is the link to the firebox bug report

Important

Mark with [x] to select. Leave as [ ] to unselect.

When reporting a bug/issue:

• Screenshot
  Screen shots of good and bad rendering are found in the firefox bug linked above.

• The FontForge version and the operating system you're using

FontForge: 2019-08-01
Windows: 10, Version 1903, build 18362.657

• The behavior you expect to see, and the actual behavior

Expect to see properly rendered font. Actually see garbled rendering.

• Steps to reproduce the behavior

Open .ttf format font in FontForge (the .ttf is archived in the older bug reported to firefox and referenced from the one above; here is the link see atached file kaiu.ttf or can also be obtained for now at this link

Open that file with FontForge, and save as .woff2, and view the file. A test case for viewing is visible for now at this link using the .woff2 created by FontForge. Renders fine on Windows, Android, but not Mac or iOS. The test case is also archived in the first referenced Firefox bug.

• (optional) Possible solution/fix/workaround

Used pyftsubset, part of the fonttools package for Python, and it produced a usable .woff2.

When you open an issue for a change/improvement/feature request:

• A description of the problem you're trying to solve, including why you think this is a problem
• If the feature changes current behavior, reasons why your solution is better
• (optional) Possible solution/fix/workaround

[ctrlcctrlv]

Do you know exactly what FontForge is doing to the output that corrupts it?

[ctrlcctrlv]

Sorry, I have bad eyesight. I didn't see the Firefox issue. I, unfortunately, do not have a Mac @v-python. So I have the following questions for you:

• How do you know the problem is Mac-only?
• It only affects Firefox? Not Chrom(ium)?

My theory based on the bad output in the Firefox issue is that Firefox is getting the hinting wrong, and probably writing out a bad CID-keyed font (we know that CID-keyed font support is not the best in FontForge, see #3955). Perhaps even it's applying the wrong hinting to the wrong glyphs!

But why Mac only? Does Firefox use CoreText on Mac?

Thanks for your time.

[ctrlcctrlv]

One last question: What happens if you tell FontForge not to write out either PostScript or TrueType hints? That'd be File→Generate Fonts…, in the dialog Options, then uncheck both hints under "PostScript®" and "SFNT".

Maybe try either/or, so we can figure out which type of hint is the problem!

[ctrlcctrlv]

Sorry @v-python, I'm probably driving you crazy! I am now quadruple posting, oh dear!

So I looked at the file and noticed it took a very long time to open. FontForge said it was “Fixing up References” for minutes. So this is certainly not a hinting issue, that's the wrong rabbit hole.

When it finally opened, it looked like this:

Having hacked on FontForge quite a lot, I know that hints are not applied in the CharView, CharView glyphs are always unhinted, and they look the same as in the FontView.

So, I know the problem, and I really doubt it's Mac-only.

FontForge is writing out the references wrong. Question is: why? That's the correct rabbit hole. 🐰 🕳️

I wondered: is FontForge also reading the references wrong? So I decided to try to replicate the text in your issue. No small feat—I don't speak Chinese! Luckily Google let me handwrite it. «召唤你的时候» is enough.

Now we're cooking with gas! 汽油

OK, so FontForge is only writing them wrong, but it understands the references the same way CoreText does, and FontForge also produces comparable output to CoreText. Meaning, without a doubt it's writing the file wrong.

To me it seems like it's not so much a matter of FontForge writing out the reference transformations wrong, but it's actually referencing the wrong glyphs. Could be wrong! FontForge also can't understand the original file, nor the WOFF2 created by Mr. Kew from Firefox, they look the same. So it seems like it's reading them in wrong, then writing out its wrong interpretation.

Anyway, your bug is CONFIRMED. I don't know when I'll get to it, but my interest is definitely piqued!

[ctrlcctrlv]

Sorry to quintuple post, but I have yet more information.

I'm removing the "High Priority" label for a few reasons:

1. Inkscape has trouble with this font.
   

I would like to loop @johanmattssonm into the conversation, Birdfont developer. As Birdfont is also affected by this bug, I filed a bug there as johanmattssonm/birdfont#93. (yes it's a build guide but he seemed most interested in the broken TTF output)

I have compiled FreeType 2 in debug&trace mode, and here is the result for rendering glyph 公 at size 36pt:

https://gist.github.com/ctrlcctrlv/dbeefb48c2a7a45db15b6b98491ae6b0

What can it tell us? Well, first of all:

  4 components
    subglyph 0:
      glyph index: 453
      offset: x=92, y=716
    subglyph 1:
      glyph index: 409
      offset: x=76, y=648
    subglyph 2:
      glyph index: 283
      offset: x=232, y=504
    subglyph 3:
      glyph index: 481
      offset: x=240, y=468

Reading the source code that generates this information teaches that if the problem were TTF point matching, or FontForge failing to read the transformation matrix, that would show up here.

Indeed, I have identified the problem: TrueType instructions.

In my naïve pre-18th of February mind, TrueType instructions only occur to educate the rasterizer. Meaning, as far as vectors are concerned, they are not really important. I never gave them much thought.

However, it is indeed exactly instructions which this font needs to display correctly. Let's observe:

How did I achieve this? Literally just «View→Render using Hinting»! But wait, there's more:

The instruction to blame is SVTCA, which can be seen in the FreeType debugger output. https://docs.microsoft.com/en-us/typography/opentype/spec/tt_instructions

SVTCA literally moves the points around. Meaning, FontForge is not wrong to show us these deformed glyphs. It is interpreting them correctly, that's literally how they appear before TrueType instructions.

OK, you say, so what then is going wrong? I have no idea, but we're probably writing out the wrong instructions. I have no interest in pursuing this further, unfortunately. I learned something by pursuing it, but at the end of the day: the font is ancient, and these kind of shenanigans are quite rightly no longer put to use in modern TrueType.

[v-python]

Thanks for all your investigation, it is enlightening.

OK, some information from here. I don't speak Chinese either, but I have a user that does, and we are working on a project that uses this font. The font may be ancient, as you indicate, but there are two things about it that make us want to use it: 1. The Chinese people consider it to be a beautiful font 2. probably because of the weird encoding, it is amazingly more compact, even in TTF form, than other fonts that are similar. And I don't have a Mac either, but have my user does.

As far as how we determined it was only happening on a Mac? (well, and iOS too), it is simply by experimentation. The subset font output from fontforge renders fine on Windows and Android, but not on Mac or iOS. Even I, as a non-Chinese speaker/reader, can tell there are significant differences in the rendering from the screenshots. Using primarily Firefox for development of the project, and Windows as the development platform, the output from FontForge in creating a subset font with only the characters needed for the project looked fine. But when my user tried Mac, it didn't look fine. So we thought it was a platform-deficiency in Firefox at first, hence the Firefox bug, but Jonathan's analysis there, and his knowledge that Firefox uses the Mac system renderer, not its own implementation like it does on Windows and Android, convinced us it was a Mac problem. Although we haven't tested Linux yet.

We had been aware that this font was "different" in some way in its internal coding, because some font tools can handle it and some cannot, but we are not font experts, so really didn't know how it was different.

It is extremely curious that the output from FontForge subset DOES work on Windows & Android given the table differences Jonathan reported. We don't know how the Python fonttools pyftsubset tool internal manipulations and output differs from FontForge's, but using it, we can produce a file that does render properly on Mac & iOS. And Jonathan used yet a different tool from Google (which we couldn't easily compile from source, not having C compilers installed, nor Linux, although we could probably cure both of those given enough time), which is why we tried the fonttools package instead of the Google package. So we have two routes to producing a font that does render properly on the Mac, just not FontForge, which we have successfully used to create some other specialty fonts for the project.

We could certainly share the output from Python fonttools, to compare the tables it produces to those produced by FontForge, although this is a subset font

It would seem likely to us, since Windows can render the FontForge output correctly, and that fonttools can produce one that works on the Mac, that it is possibly just some minor setting in the FontForge output file that prevents proper rendering on the Mac? Your in-depth analysis says that FontForge might be corrupting the instructions, but I wonder whether it is just causing Mac to ignore the instructions, because the instructions seem to exist and be successfully used on Windows and Android. Hence while the font may be ancient, and while the techniques it uses may not be best practices (I have no knowledge from which to contradict your claims and analysis in that regard), it does have the benefits I listed above, and it can be made to work on Mac & iOS using other tools, and works on Windows as output from FontForge.

[ctrlcctrlv]

We'd need some debug output from CoreText (Apple's FreeType) to even begin to understand the problem, in that case. I have no way to debug this further, but someone else might.

The font is compact, and beautiful, sure, but not much more so than AR PL KaitiM GB, which is in a standard font format. Given how little software understands it, and how only one platform can't understand our rewritten instructions, I don't see this as a high priority.

[ctrlcctrlv]

Oh, one more hint.

https://github.com/servo/font-kit/wiki/FAQ#what-about-macos-and-ios

I did notice that the cmap table in this font is changed quite a bit by FontForge. Perhaps Mac OS no longer recognizes the font as CJK?

I really am sorry I don't have more ideas, I don't have a Mac nor do I know much about Apple's text renderer

[v-python]

Just FYI: A quick look by one of my users at the AR PL KaitiM GB font shows that it is missing many characters that my users would need.

[lemzwerg]

In FreeType, fonts like the one described in this issue are called 'tricky'; they are handled specially since TrueType instructions are used to scale and position composites. In particular, FreeType maintains a list of such fonts (see functions tt_check_trickyness_family and tt_check_trickyness_sfnt_ids in file src/truetype/ttobjs.c); bytecode instructions must be applied, otherwise you get the garbage seen in the screenshots above.

Even on recent MacOS versions the TrueType bytecode engine is run for such fonts (and only for such fonts); there is a set of constraints to make that happen. One of them, IIRC, is that the ppem value must be less than 1000. Alas, I forgot the other constraints and can't find them right now.

The theory that recent MacOS versions don't recognize the font in question as being 'tricky' sounds correct to me.