[hb-subset] Failure to subset NotoSansCJK-Regular.ttc

[madig]

HarfBuzz 2.9.0 from the provided Windows binaries.

1. Get https://github.com/googlefonts/noto-cjk/blob/165c01b46ea533872e002e0785ff17e44f6d97d8/Sans/OTC/NotoSansCJK-Regular.ttc
2. hb-subset.exe --unicodes-file subset.txt --layout-features="*" NotoSansCJK-Regular.ttc --face-index 0 -o out.ttf
3. Observe hb-subset exits with error code 1 and leaves a 0 bytes out.ttf.
4. Remove last Unicode value from subset.txt and try again
5. See it working this time?!

Some times it works with the last value, too, for some reason?!?! If you put the value as the only one into a new txt file and use it, hb-subset also finishes properly...

subset.txt

[garretrieger]

I suspect this is failing due to an overflow writing out the cmap4 subtable. That's why removing certain characters from the subset allows it to work. I'll confirm if that's actually the case.

[garretrieger]

Alright, this is indeed a cmap4 overflow.

[garretrieger]

To explain the cmap4 subtable can be at most 65kb long since the "length" field is a uint16. When subsetting it can actually increase the size of the cmap4 table since the table is range based and the subset can fragment the ranges. If the new size exceeds 65kb the subset fails as there's no way the table can be serialized. I'll check with Behdad and see if there's a reasonable fallback strategy we could use, such as outputting only a cmap12 subtable.

[behdad]

Outputting only cmap12 SGTM. We should make that default sooner than later (outputting just one).

[madig]

Random thought: Should hb-subset trap overflows and explode, even in release builds?

[garretrieger]

It does, that's what caused the subset command to return status code 1 instead of 0 in this case. Though we could certainly improve the debug messaging to better indicate why it failed.

[behdad]

What would be better? Crash or generate broken font?

[madig]

Ah, ok. I was puzzled because there was no message and my shell by default does not show return codes.

Crash is better of course :)

[garretrieger]

I noticed that this subsetting call was quite slow, found some O(n^2) code in the cmap4 handling. Fix is in #3178

[behdad]

And I made it print an error message if operation failed, since we don't expect that to happen.