Cantarell CFF2 subsetting corrupts outlines

[madig]

Tested with https://github.com/harfbuzz/harfbuzz/tree/adccc5355b695cfaaf6403a3187c67c7fcae623d.

1. Get https://cantarell.gnome.org/releases/cantarell-fonts-0.303.tar.xz and pull out the prebuilt/*.otf file inside
2. Run hb-subset --glyphs="*" --layout-features="*" --layout-scripts="*" --drop-tables-="*" --instance="wght=100" Cantarell-VF.otf -o /tmp/a.otf

It also fails to load in sites like http://cyreal.org/Font-Testing-Page/, with these error messages in Firefox:

• downloadable font: Variations: Required fvar table is missing (font-family: "a" style:normal weight:400 stretch:100 src index:0)
• downloadable font: Variations: Failed to parse variation region list (font-family: "a" style:normal weight:400 stretch:100 src index:0)
• downloadable font: CFF2: Failed to parse Top DICT Data (font-family: "a" style:normal weight:400 stretch:100 src index:0)
• downloadable font: CFF2: Failed to parse table (font-family: "a" style:normal weight:400 stretch:100 src index:0)
• downloadable font: rejected by sanitizer (font-family: "a" style:normal weight:400 stretch:100 src index:0)
• downloadable font: font load failed (font-family: "a" style:normal weight:400 stretch:100 src index:0)

[behdad]

Oh boy...

[behdad]

• downloadable font: Variations: Required fvar table is missing (font-family: "a" style:normal weight:400 stretch:100 src index:0)

Humm. Is OTS requiring fvar table for CFF2 fonts? If that's the case, that seems wrong. cc @jfkthame @khaledhosny

[behdad]

Oh, it's probably because we didn't null the VarStore in the CFF2 table. Nevermind.

[madig]

Random thought: should subsetting tests employ OTS to sniff-check resulting files?

[behdad]

Random thought: should subsetting tests employ OTS to sniff-check resulting files?

We already do that:
https://github.com/harfbuzz/harfbuzz/blob/main/test/subset/run-tests.py#L22

We just don't have many (any?) tests for CFF instancing.

cc @qxliu76

[behdad]

Passing OTS is rather hard fix :(

[behdad]

I'm pushing some fixes here:
#4126

[behdad]

The good news is that the bug with rendering is NOT in the instancing, but in the desubroutinization. As if it makes it any easier...

[behdad]

Okay, wasn't too hard. A missing flush_hintmaks in CFF2 subset codepath.

[behdad]

The rendering issue is fixed. For ots-sanitize issue, I'm working on that in #4126, though I'm not sure when/if I can finish it.

[behdad]

#4126 fixes the OTS issue.