luaotfload hangs

[hvoss49]

\input luaotfload.sty

  \font\sparkBarMedium     ="file:Sparks-Bar-Medium.otf:mode=node;+calt" 
%  \font\sparkDotLineMedium ="file:Sparks-Dotline-Medium.otf:mode=node;+calt" 

Text {\sparkBarMedium{14,95,68,9,19,41,91,1,81,97,79,45,96,76,17,65,8,92}} Text

%Text {\sparkDotlineMedium{14,95,68,9,19,41,91,1,81,97,79,45,96,76,17,65,8,92}} Text
\bye

Example hangs if I enable the second font. It has 100% CPU load and nearly all of the memory.

luatex zzz2.tex 
This is LuaTeX, Version 1.10.0 (TeX Live 2019) 
 restricted system commands enabled.
(./zzz2.tex
(/usr/local/texlive/2019/texmf-dist/tex/luatex/luaotfload/luaotfload.sty
(/usr/local/texlive/2019/texmf-dist/tex/latex/base/ltluatex.tex)
luaotfload | main : initialization completed in 0.096 seconds
)Getötet

The font files are here:

http://tug.org/~hvoss/Sparks-Bar-Medium.otf
http://tug.org/~hvoss/Sparks-Dotline-Medium.otf

up-to-date TL2019 pretest and Linux

It looks like a problem with the file size. All Sparks Dotline font have a size over 400 kByte:

-rw-r--r-- 1 voss voss 34512 Apr 11 18:54 Sparks-Bar-Extranarrow.otf
-rw-r--r-- 1 voss voss 34476 Apr 11 18:54 Sparks-Bar-Extrawide.otf
-rw-r--r-- 1 voss voss 34412 Apr 11 18:54 Sparks-Bar-Medium.otf
-rw-r--r-- 1 voss voss 34480 Apr 11 18:54 Sparks-Bar-Narrow.otf
-rw-r--r-- 1 voss voss 34384 Apr 11 18:54 Sparks-Bar-Wide.otf
-rw-r--r-- 1 voss voss 34396 Apr 11 18:54 Sparks-Dot-Extralarge.otf
-rw-r--r-- 1 voss voss 34396 Apr 11 18:54 Sparks-Dot-Extrasmall.otf
-rw-r--r-- 1 voss voss 34304 Apr 11 18:54 Sparks-Dot-Large.otf
-rw-r--r-- 1 voss voss 402596 Apr 11 18:54 Sparks-Dotline-Extrathick.otf
-rw-r--r-- 1 voss voss 411520 Apr 11 18:54 Sparks-Dotline-Extrathin.otf
-rw-r--r-- 1 voss voss 403776 Apr 11 18:54 Sparks-Dotline-Medium.otf
-rw-r--r-- 1 voss voss 412044 Apr 11 18:54 Sparks-Dotline-Thick.otf
-rw-r--r-- 1 voss voss 414412 Apr 11 18:54 Sparks-Dotline-Thin.otf
-rw-r--r-- 1 voss voss 34244 Apr 11 18:54 Sparks-Dot-Medium.otf
-rw-r--r-- 1 voss voss 34328 Apr 11 18:54 Sparks-Dot-Small.otf

[u-fischer]

It hangs in context too (at least on my machine and in the time frame I was willing to invest). Could you report the problem on the context list?

[hvoss49]

In ConTeXt, TL 2019 pretest it works for me. It takes a lot of time for the first run but if the large font is in the cache it is a fast run.

\definefont[sparkBar][file:Sparks-Bar-Medium.otf:mode=node;+calt]
\definefont[sparkDotline][file:Sparks-Dotline-Medium.otf:mode=node;+calt]
\starttext
Text \sparkBar{14,95,68,9,19,41,91,1,81,97,79,45,96,76,17,65,8,92} Text

Text \sparkDotline{14,95,68,9,19,41,91,1,81,97,79,45,96,76,17,65,8,92} Text

\stoptext

On Linux and macOS, both TL2019

[zauguin]

I tried both ConTeXt and LuaLaTeX, TL 2019 Pretest. In both cases, the processes needed more than 6GB of memory and therefore died. So I can not see any difference here which makes debugging quite hard. @hvoss49 How much memory is used by ConTeXt on the first run?

[hvoss49]

I see the problem. On my macOS I have 16GB RAM but on my other machine "only" 8GB.
On the Mac the first run needs 9.5GB Memory which I do not have on the Linux machine.
However, caching a 400 kB font needs so much memory is a bit curious ...

[u-fischer]

Yes, but I assume that you will get the problem in context too on the linux machine so the problem should be reported there.

[hvoss49]

The answer from Hans:

On 4/12/2019 3:17 PM, Herbert Voss wrote:

Hello all,

If I run the following document on a 8 GByte RAM Linux machine.
It takes more than 30 minutes until luaotfload has build
the cache for Sparks-Dotline-Medium.otf. If I run another
Program in the background that it never ends (too less memory).
On my Mac the first run with context (upcoming TL2019) needs 9.5 Gbyte
memory and nearly 100% CPU load to scan the Sparks-Dotline font.
This one is, of course, 400 kB big, but why do we need
more than 8 GByte Ram??

• the cpu load is puzzling: assuming that you have multiple cores, luatex will only use one core

• the 30 min on linux is probably due to swapping

• the 8 gig in general is a side effect of the way this font is using pair positioning (half a billion pairs due to classes where most pairs are dummies)

The font files are here:

http://tug.org/~hvoss/Sparks-Bar-Medium.otf
http://tug.org/~hvoss/Sparks-Dotline-Medium.otf

i hacked a faster solution for such fonts (now I need some 25 sec for the initial caching so probably less on your machine and < 0.1 sec for loading the cached font)

-rw-r--r-- 1 voss voss 34412 Apr 11 18:54 Sparks-Bar-Medium.otf
-rw-r--r-- 1 voss voss 403776 Apr 11 18:54 Sparks-Dotline-Medium.otf

These make a pretty good case for using proper drawing package,

[hvoss49]

@ulrike: Can you use the "hack" from ConTeXt for the LaTeX world?

[u-fischer]

If Hans added it to the generic fontloader and nothing else breaks it should work. I will try to make a new version this evening or tomorrow.

[u-fischer]

I imported the files but the fontloader is broken. I get lua errors both with the merged and with the reference fontloader and also with the plain fontloader.

Lua module: luaotfload-loaders 2019-04-12 2.9705 luaotfload submodule / callback
 handling...fload/texmf/tex/luatex/luaotfload/luaotfload-loaders.lua:88: attempt
 to get length of a nil value (upvalue 'sequence')

@zauguin can you take a look? The code is in the v2.9705-2019-04-12 branch.

[hvoss49]

The table sequence{} is not defined, the reason why #sequence fails. I only found a definition in the
file src/auto/fontloader-2019-04-12.lua, line 6553

But I have no idea if this belongs to the table used in luaotfload-loaders

[zauguin]

@u-fischer That's not really the error, the problem is that a ConTeXt-only file is imported, breaking the remaining load process. I pushed a fixed version in a branch , basically it just replaces font-tfm with luatex-fonts-tfm.

Currently this requires

\directlua{ node.direct.getattr = node.direct.get_attribute }

before luaotfload can be loaded, I will create a PR when this is fixed. But I have to go now.

[u-fischer]

@zauguin a yes, I see the change now.

[zauguin]

@u-fischer I am now convinced that this is an upstream bug:

In the generic fontloader, basics-nod.lua, we have

nuts.getglyphdata        = direct.getattr
nuts.setglyphdata        = direct.setattr

This fails, because `direct.[sg]etattr is not defined, it should be either

nuts.getglyphdata        = nuts.getattr

or

nuts.getglyphdata        = direct.get_attribute

(I do not thing setglyphdata is correct either, but it isn't used anyway and set_attribute would be wrong too.)

Could someone report this on the ConTeXt list? Posting there is kind of cumbersome for me.

[u-fischer]

@zauguin I will do it after I changed the files to use the new fonts-tfm and made some tests. Thanks for the investigation!

[hvoss49]

not that I have any knowledge in this area, but in ConTeXt I found:

nuts.getglyphdata          = direct.getglyphdata or get_attribute
nuts.setglyphdata          = direct.setglyphdata or set_attribute

Is this different?

[u-fischer]

I pushed a new version which corrects the bug manually. It is in the branch https://github.com/u-fischer/luaotfload/tree/v2.9705-2019-04-12. You can use the texmf there directly for tests.

Your font worked okay for me with it with a 64bit binary, with 32bit I still get an error (but at least rather fast) when trying to create the luc/lua of the font (but the font can be used by copying the luc/lua from the 64bit system).

[hvoss49]

Thanks, it is not fast on Linux, but it works ... and this was not possible before with the complete document. There I have four diffrent fonts. Am Sa., 13. Apr. 2019 um 17:01 Uhr schrieb Ulrike Fischer < notifications@github.com>:

…

I pushed a new version which corrects the bug manually. It is in the branch https://github.com/u-fischer/luaotfload/tree/v2.9705-2019-04-12. You can use the texmf there directly for tests. Your font worked okay for me with it with a 64bit binary, with 32bit I still get an error (but at least rather fast) when trying to create the luc/lua of the font (but the font can be used by copying the luc/lua from the 64bit system). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#55 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMS0P8LsU92Ubj5YKgmcR3gHT4e2rNxRks5vgfFOgaJpZM4creGb> .

[u-fischer]

version 2.97 will contain this code and has been uploaded to ctan.