Version 2.98 crashing LuaTeX for some fonts

[bgvoisin]

Since version 2.98, some macOS fonts which were yielding blank output with LuaTeX and luaotfload now crash it. One such font is Big Caslon Medium:

! error: (file /System/Library/Fonts/Supplemental/BigCaslon.ttf) (type 2): the
re are no glyphs in the subset
! ==> Fatal error occurred, no output PDF file produced!

Reverting back to pre-2.98 behaviour by using feature notdef=false solves the problem (the output is still blank, but it does not crash LuaTeX). The problem doesn't arise with HarfTeX or LuaHBTeX. See attached files.

luaotfload-2.98-issue.zip

[u-fischer]

Reproducible with e.g. the following. Obviously the code doesn't like it if one uses a font, without actually accessing an existing glyph of this font:

\input luaotfload.sty
\font\test={file:logix.ttf}\test
ɟɠ
\bye

[zauguin]

An even more minimal example is

\input luaotfload.sty
\font\test={file:logix.ttf:-notdef}\test
\char"F0000
\bye

(For this font, the .notdef glyph is mapped to U+F0000)

The problem is that LuaTeX, at least for TrueType fonts, does not count the notdef glyph as a used glyph, so if only notdef is used, it thinks that there are no used glyphs at all. I will send a patch to dev-luatex.

[zauguin]

It the meantime we should discuss the elephant in the room: Why is there blank output in the first place?
@bgvoisin Do you know about any font which shows the same problem and which is available without macOS?

[u-fischer]

@zauguin imho the elephant is in the room as @bgvoisin made a rather large testsuite for fonts to compare xelatex/lualatex/harflatex/luahblatex) and in some cases the font actually hadn't the glyphs of the test. So it is a rather academic and not very urgent problem. But if possible it should be avoided that one gets a fatal error in this case.

[zauguin]

@u-fischer I do not think so, because @bgvoisin explicitly mentioned and showed in his test files that harftex and luahbtex can show the missing glyphs, so they must exist somewhere.

[u-fischer]

@zauguin I hadn't the time to look at the test files, I simply took "were yielding blank output" and "the output is still blank" as an indication that no glyph (apart from notdef) is actually used (that's why I then tried with the logix.ttf).

[zauguin]

Reported to the LuaTeX team in https://mailman.ntg.nl/pipermail/dev-luatex/2019-July/006287.html

[bgvoisin]

On 23 Jul 2019, at 12:50, Marcel Krüger ***@***.***> wrote: It the meantime we should discuss the elephant in the room: Why is there blank output in the first place? @bgvoisin Do you know about any font which shows the same problem and which is available without macOS?

Sorry for missing most of the discussion, I was off the internet for a few hours. Unfortunately I don't have access to a non-mac computer right now. The fonts with which the problem was observed are (font name + font file + sample input exhibiting the problem) Heiti TC Light [STHeiti Light.ttc](0) 壹貳參肆伍陸柒捌玖拾 Heiti SC Light [STHeiti Light.ttc](1) 汉体书写信息技术标准相容 Heiti TC Medium [STHeiti Medium.ttc](0) 壹貳參肆伍陸柒捌玖拾 Heiti SC Medium [STHeiti Medium.ttc](1) 汉体书写信息技术标准相容 Big Caslon Medium [BigCaslon.ttf] ABCDEFGHIJKLM GB18030 Bitmap [NISC18030.ttf] 汉体书写信息技术标准相容 Webdings [Webdings.ttf]  Wingdings [Wingdings.ttf]  Wingdings 2 [Wingdings 2.ttf]  Wingdings 3 [Wingdings 3.ttf]  LiHei Pro [LiHeiPro.ttf] 壹貳參肆伍陸柒捌玖拾 LiSong Pro [LiSongPro.ttf] 壹貳參肆伍陸柒捌玖拾 STFangsong [STFANGSO.ttf] 汉体书写信息技术标准相容 STXihei [STXIHEI.ttf] 汉体书写信息技术标准相容 STHeiti [STHEITI.ttf] 汉体书写信息技术标准相容 Yuanti SC Light [Yuanti.ttc](4) 汉体书写信息技术标准相容 Yuanti SC [Yuanti.ttc](0) 汉体书写信息技术标准相容 Yuanti SC Bold [Yuanti.ttc](2) 汉体书写信息技术标准相容 The first group of fonts are core macOS fonts (installed in /System/Library/Fonts), the second group additional fonts (still part of the OS, but installed in /Library/Fonts) and the third group downloadable fonts (not installed by default, downloaded from Apple servers when requested automatically by apps or manually by the user). See <https://support.apple.com/en-us/HT208968>. I don't know which fonts may be obtained independently. Based on <https://docs.microsoft.com/en-us/typography/fonts/windows_10_font_list> I imagine Webdings and Wingdings are also part of Windows, though maybe not in the same versions and formats. The samples are the first line of the font samples in Apple's Fon Book app, pasted directly from Font Book into an UTF-8 text editor. Bruno

[bgvoisin]

The problem also vanishes putting :mode=base instead of :notdef=false. I don't know enough of LuaTeX to know whether that remark makes sense/is helpful.

[zauguin]

@bgvoisin Which problem vanishes then? Do you see the actual glyphs or just an empty line with mode=base?

@u-fischer Do you have access to a Mac where you can test these fonts?

[u-fischer]

@zauguin I don't have a mac, but Paulo (@cereda) can run tests.

[bgvoisin]

@zauguin I meant an empty line with mode=base.

[zauguin]

@bgvoisin or @cereda: Can either of you upload the file bigcaslon.lua? (It probably is stored in ~/.texlive2019/texmf-var/luatex-cache/generic/fonts/otl or whatever the macOS equivalent is)

[bgvoisin]

@zauguin Here it is (it's actually ~/Library/texlive/2019/texmf-var/luatex-cache/generic/fonts/otl).

bigcaslon.lua.zip

[amunn]

I'm happy to test too, if needed.

[zauguin]

@bgvoisin Do you have ttx installed? What does ttx -t cmap -o - /Library/Fonts/BigCaslon.ttf show?

[bgvoisin]

@zauguin I didn't have it but just installed it, it's quite easy actually.

ttx-BigCaslon.txt

[zauguin]

@u-fischer The actual problem is that the OpenType cmap format used by these fonts isn't supported by the fontloader. We could patch fontloader-font-otr.lua to add them, but probably it would be better to ask upstream to avoid carrying patches. Could you do that, I always have trouble with their mailing list?
The necessary cmap formats are "platform 0 (Unicode), encoding 1 (Unicode 1.1(?)), format 12" and maybe as a fallback "platform 1 (Mac), encoding 0 (Roman), format 0".

[u-fischer]

@zauguin yes, I can naturally report it. Do you have an example that I can use in the message? And where in the file should the entries go? If the cmap isn't supported, why did the problem only appear after the missing char change?

[zauguin]

@u-fischer

\starttext
\font\oddfont="Big Caslon Medium" \oddfont A
\stoptext

This file shows the error, but it requires proprietary Mac fonts. I couldn't reproduce it with any more accessible fonts.

In fontloader-font-otr.lua there is a table sequence. I think, at least for the first entry, it should be enough to just add the {0, 1, 12} here. Hans&co will know if additional changes are necessary.
The format 0 would require a new parser formatreaders[0], but most likely they will not support this anyway. (It is Mac specific and only supports 256 codepoints AFAICT)

This is the problem that the characters are not printed which appeared even before the missing char change. The missing char thing only made it more obvious because instead of printing nothing, LuaTeX started to produce error messages.

[u-fischer]

@zauguin Could you provide a patch for the table? I could then make a test version for @bgvoisin to confirm that it works and then pass the diff on upstream.

[u-fischer]

@zauguin I just saw that Hans changed something in the file, and added { 0, 4, 12 }, to the sequence table. I will build a new luaotfload version from this.

[u-fischer]

@bgvoisin I pushed a new version in the branch https://github.com/u-fischer/luaotfload/tree/v2.9902-2019-07-24. Could you please try it? You simply need the texmf tree and add it to your system with tlmgr auxtrees add .... @zauguin

[zauguin]

@u-fischer I created a testing branch macfonts where I uncommented the significant element in all fontloader copies.
@bgvoisin Maybe you could test that version too?

The "pure" patch would be

diff --git a/src/fontloader/misc/fontloader-font-otr.lua b/src/fontloader/misc/fontloader-font-otr.lua
index 76d7390..62c33f7 100644
--- a/src/fontloader/misc/fontloader-font-otr.lua
+++ b/src/fontloader/misc/fontloader-font-otr.lua
@@ -1319,7 +1319,7 @@ local sequence = {
     { 3, 10, 12 },
     { 0,  3,  4 },
     { 0,  1,  4 },
- -- { 0,  1, 12 }, -- maybe for some old mac fonts
+    { 0,  1, 12 }, -- maybe for some old mac fonts
  -- { 0,  4, 12 },
     { 0,  0,  6 },
     { 3,  0,  6 },

[bgvoisin]

@zauguin I'm testing right now the new version from @u-fischer. I made the mistake of using my catalog file of all macOS fonts as a test, typesetting is taking ages (1 hour and 27 minutes now, still running, still recreating many font caches from scratch – lua and luc files). For one Hiragana font the lua file is even 82 MB and took 23 minutes to create ! I should have tested font by font only. I'll let you know of the outcome as soon as typesetting completes. I'll have a look at the macfonts branch afterwards.

[bgvoisin]

@u-fischer Reporting failure unfortunately: the new v2.9902-2019-07-24 yields exactly the same fatal errors as before, for exactly the same fonts. Two caveats:

• I'm not familiar with tlmgr conf auxtrees, I may have misused it. I just ran

sudo tlmgr conf auxtrees add /Users/brunovoisin/Downloads/luaotfload-2.9902-2019-07-24/texmf

and that was it. For example,

ls -l /usr/local/texlive/2019/bin/x86_64-darwin/luaotfload-tool lrwxr-xr-x 1 root wheel 55 Jul 6 12:34 /usr/local/texlive/2019/bin/x86_64-darwin/luaotfload-tool -> ../../texmf-dist/scripts/luaotfload/luaotfload-tool.lua

so luaotfload-tool points to the old script not the new one

~/Downloads/luaotfload-2.9902-2019-07-24/texmf/scripts/luaotfload/luaotfload-tool.lua

in case that matters. If time allows, I'll try doing things the more pedestrian way I'm familiar with, putting the new texmf tree in texmf-local then running mktexlsr and fmtutil-sys manually to make sure everything is properly initialized. (And changing the script symlink manually.)

• I must admit I've lost a bit track of what we're testing here: a fix to the 2.98 update, to take .notdef into account as part of the subset; or a fix to the way some odd Mac fonts are processed, Big Caslon in particular? So I'm not certain to have tested the proper thing.

I won't be back for a few hours.

[u-fischer]

At best test only one of your fonts. E.g. with

\font\oddfont="Big Caslon Medium" \oddfont A

Check the log file: It should tell you which luaotfload version you are using. It should also tell you where the fontloader is that you are using. So it should be easy to sea, if you are using the right one.

We are trying here to fix that your fonts gave a blank output, the notdef business was only a side effect.

[bgvoisin]

@u-fischer It's luaotfload 2019-07-24 2.9902 and fontloader-2019-07-24.lua. With them it's still fatal error. Following is an archive with a tex test file for all the fonts that exhibit the behaviour, together with associated lua and ttx files.

macfonttest.zip

[bgvoisin]

@zauguin With the macfonts branch, exactly the same errors as with the trunk fontloader-2019-07-24 unfortunately.

[zauguin]

@bgvoisin It did fix the problem for Big Caslon, you just have to wipe the font caches before the change applies. You can pull the latest version from the branch first to fix the other fonts too. Then delete everything in ~/Library/texlive/2019/texmf-var/luatex-cache/generic/fonts/otl and try again.

[bgvoisin]

@zauguin Thanks. I should have thought about the caches! It does fix things for all the fonts save one, which is insignificant: GB18030 Bitmap (aka NISC18030.ttf), a bitmap font that has never worked with any TeX engine. Thanks on behalf of the Mac TeX community!

[u-fischer]

@bgvoisin did it work with my version or only with the macfonts test from @zauguin? Or more precisely: is the extra setting { 0, 1, 12 }, needed and if yes for which fonts?

[bgvoisin]

@u-fischer With luaotfload-2.9902-2019-07-24 all the fonts are giving fatal errors. These are

[STHeiti Light.ttc](0) = Heiti TC Light
[STHeiti Light.ttc](1) = Heiti SC Light
[STHeiti Medium.ttc](0) = Heiti TC Medium
[STHeiti Medium.ttc](1) = Heiti SC Medium
[BigCaslon.ttf] = Big Caslon Medium
[Webdings.ttf] = Webdings
[Wingdings.ttf] = Wingdings
[Wingdings 2.ttf] = Wingdings 2
[Wingdings 3.ttf] = Wingdings 3
[LiHeiPro.ttf] = LiHei Pro
[LiSongPro.ttf] = LiSong Pro
[STFANGSO.ttf] = STFangsong
[STHEITI.ttf] = STHeiti
[STXIHEI.ttf] = STXihei
[Yuanti.ttc](0) = Yuanti SC
[Yuanti.ttc](2) = Yuanti SC Bold
[Yuanti.ttc](4) = Yuanti SC Light

With luaotfload-macfonts they all give correct output (ie not blank lines, but exactly the requested glyphs). This was with luaotfload-macfonts downloaded this morning (build 2019-07-25 02:46:50), not the version from yesterday night (build 2019-07-24 17:52:03). Looking very rapidly, it adds not only

{ 0, 1, 12 }

but also

{ 0, 3, 12 }
{ 3, 0, 4 }

[zauguin]

@u-fischer BigCaslon.ttf needs {0, 1, 12}, the W*dings*.ttf fonts need {3, 0, 4} and the affected CJK fonts require {0, 3, 12}.

[u-fischer]

@bgvoisin the values suggested by @zauguin are in the newest version. It would be nice if you could test. The files are in branch https://github.com/u-fischer/luaotfload/tree/v2.9904-2019-08-02.

[bgvoisin]

@u-fischer the newest version work fine.

test-results.zip

[u-fischer]

This will be resolved with the next version.