Experimenting with OpTeX on LuaHBTeX

[robertbachmann]

Yesterday I tried to use OpTeX with HarfBuzz.

\fontfam[lm]

% from: CTAN (current Google release does not work)
\font\notoemojifont=file:NotoColorEmoji.ttf:mode=harf; at 8pt
% from: https://github.com/mozilla/twemoji-colr/releases
\font\twemjifont=file:Twemoji.Mozilla.ttf:mode=harf; at 8pt

Noto: {\notoemojifont ✅😀👍\char"1F98B}

Twemoji: {\twemjifont ✅😀👍\char"1F98B}

% From FontAwesome website
\font\fa=file:FontAwesome6Free-Solid-900.otf:mode=harf; at 8pt
Fontawesome contextual ligatures:\nl
\code{\{\\Green\\fa music download person location\}}\nl
{\Green\fa music download person location inbox}

\bye

Results:

.ini file I used:

% ophbtex.ini

% luahbtex -ini ophbtex.ini
% ln -s /usr/local/bin/luahbtex /usr/local/bin/ophbtex
\let\dump=\relax

\input optex.ini

\_def\_banner {This is OpTeX (Olsak's Plain TeX) on LuaHBTeX, version <\_optexversion>}%

\_dump

\_endcode % ------------------------------

[Udi-Fogiel]

I use LuaHBTeX all the time, as I need to typeset documents in Hebrew. Without it the nikud is usually misplaced.

[vlasakm]

Essentially luahbtex is just luatex linked with harfbuzz library. The only behavior difference in the LuaTeX engine itself is the fact that this library can be loaded with require in Lua. From what I remember, luaotfload tries to require the harfbuzz library and if it succeeds, it makes available the mode=harf; for \font. That should be the only difference in using luahbtex vs luatex binary for OpTeX.

In https://github.com/vlasakm/mmtex I chose to not link with harfbuzz by default because many years ago the harfbuzz version required by LuaTeX was not available everywhere, but already for a long time I am compiling and distributing binaries linked against harfbuzz on all major Linux distributions, thus essentially I am using ophbtex.

I think making optex always use luahbtex and thus making harfbuzz available to those that want to try it, or in fact need it, like @Udi-Fogiel (#150). OpTeX aspires to support many languages, not just english, and making harfbuzz available by default could help even people who need something more complicated than "latin with accents", until it is implemented in (high quality) Lua code in ConTeXt (which by the way, if available is much better with TeX, as it can be much more flexible and more smart about discretionaries, etc.). Though I have to admit, I have not followed the situation with non-latin scripts and ConTeXt / Harfbuzz for a few years.

Btw, change to luahbtex should be the matter of changing fmtutil.cnf. In TeX Live it looks like this: https://git.texlive.info/texlive/tree/Master/texmf-dist/web2c/fmtutil.cnf?id=7b6535bc2bafbc8588742b9a4f0ac86e206cdd97#n121 - if it's changed to luahbtex instead of luatex, the optex binary will be harfbuzz-enabled. One can of course change it in their own installation (with the user configuration file), but a global change for everyone using TeX Live should be done in this file: https://git.texlive.info/texlive/tree/Master/tlpkg/tlpsrc/optex.tlpsrc?id=7b6535bc2bafbc8588742b9a4f0ac86e206cdd97#n5, which is what is propagated to that TeX Live file.

[olsak]

I suppose that there is no additional difference between LuaTeX and LuaHBTeX. I.e., all changes done by maintainers of LuaTeX are automatically copied to LuaHBTeX. If this is true then I am surprised why we have two different binaries, i.e. two versions of format files in TeX distributions etc. A single binary LuaHBTeX with name LuaTeX should be sufficient and life will be much more simpler. LuaHBTeX enables to load harfbuzz (but it does not it by default) and users can initialize it for individual fonts by mode=harf (but it is not default). So, why we have LuaTeX without HB inside it? Historical reasons? If such reasons is out of date, then we can ask to LuaTeX maintainers to merge LuaTeX to LuaHBTeX. If this will be done then we need to do nothing in OpTeX.

I have nothing against running OpTeX with LuaHBTeX by default (with name optex). If the LuaHBTeX will be still an independent existence in TeX distributions then we have to ask the TeX live maintainers (and MikTeX maybe too) to do relevant changes in their distributions. Maybe there should be one issue: we can try a time tests with \fontfam[lm] and run it with and without HB binary in order to measure if the loading of the harfbuzz library will take some significant time which is useless for most users. If it is true then we can try to disable harfbuzz loading in luaotfload by default and to enable it explicitly by a \harfbuzz macro (for example).

[vlasakm]

I suppose that there is no additional difference between LuaTeX and LuaHBTeX. I.e., all changes done by maintainers of LuaTeX are automatically copied to LuaHBTeX.

I just rechecked whether my old knowledge is still correct. The difference between luatex binary and luahbtex binary is, that compiling luahbtex has an additional define: -DLUATEX_HARFBUZZ_ENABLED, and is statitically linked with libluaharfbuzz.a (lua module written in C making harfbuzz usable in Lua), and libharfbuzz.a (harfbuzz itself).

-DLUATEX_HARFBUZZ_ENABLED results in the following:

1. Compiled with libharfbuzz ; using is printed in luahbtex --credits (source).
2. The names LuaHBTeX and luahbtex are used instead of LuaTeX and luatex respectively. (source). These are used in things like banner and version / credits messages.
3. A reference to symbol luaopen_luaharfbuzz is craeted in the binary, allowing require("luaharfbuzz") in Lua to find the "compiled in" library (source).
4. luaharfbuzz global variable is injected, which makes the luaharfbuzz library also available through that name (source).

That's it! They are otherwise compiled from exactly the same sources. They are essentially not two different projects, but the same project compiled either with the optional harfbuzz feature (-DLUATEX_HARFBUZZ_ENABLED) or not.

If this is true then I am surprised why we have two different binaries, i.e. two versions of format files in TeX distributions etc. A single binary LuaHBTeX with name LuaTeX should be sufficient and life will be much more simpler. LuaHBTeX enables to load harfbuzz (but it does not it by default) and users can initialize it for individual fonts by mode=harf (but it is not default).

That's correct.

So, why we have LuaTeX without HB inside it? Historical reasons? If such reasons is out of date, then we can ask to LuaTeX maintainers to merge LuaTeX to LuaHBTeX. If this will be done then we need to do nothing in OpTeX.

I don't know why exactly it is the way it is. Backwards compatibility may be a concern. IIUC most changes to LuaTeX since 1.0 version have been attempting to not change the default behavior, i.e. keeping compatibility. And as LuaTeX is now in feature freeze, I don't think anyone will be for having just luatex for both the harfbuzz/non-harfbuzz use cases.

I guess one of the advantages of the current way is, that it is possible to compile LuaTeX fully without harfbuzz. But most people don't care about that, since they use binary provided by TeX Live / MikTeX which solve any problems that might be associated.

I have nothing against running OpTeX with LuaHBTeX by default (with name optex). If the LuaHBTeX will be still an independent existence in TeX distributions then we have to ask the TeX live maintainers (and MikTeX maybe too) to do relevant changes in their distributions. Maybe there should be one issue: we can try a time tests with \fontfam[lm] and run it with and without HB binary in order to measure if the loading of the harfbuzz library will take some significant time which is useless for most users. If it is true then we can try to disable harfbuzz loading in luaotfload by default and to enable it explicitly by a \harfbuzz macro (for example).

The overhead possible in LuaTeX itself is the fact, that the luahbtex has statically linked harfbuzz, and is thus a bit bigger, and may take very slightly longer time to be loaded into memory, etc. Then unfortunately luahbtex sets the global variable luaharfbuzz to the loaded luaharfbuzz module, which involves loading the module. The modern way would be to not preload the library, but let it be lazily loaded on first require("luaharfbuzz"). Otherwise harfbuzz functionality in luahbtex has no cost until anyone uses it.

luaotfload is the thing using luaharfbuzz for us. It actually checks whether luaharfbuzz is available by doing require("luaharfbuzz") (source), which means that it actually has a higher cost without luahbtex (because with luatex it has to search on disk in vain, whereas with luahbtex it's already present in memory). If the load is succesfull, then two additional Lua files are loaded (source): luaotfload-harf-define.lua and luaotfload-harf-define.lua and luaotfload-harf-plug.lua which contain the hooks that make mode=harf possible.

I'll try to benchmark the additional overhead of

\fontfam[lm]

Hello world!

\bye

I think something like \harfbuzz which would make harfbuzz available only sometimes is possible, but it would be kind of hacky. I hope the overhead is low enough to allow us to always use luahbtex.

[vlasakm]

I installed a fresh TeX Live minimal installation similarly to my guide here: #41 (reply in thread), with the addition of installing luahbtex and modifying texmf-dist/web2c/fmtutil.cnf to include the line ophbtex luahbtex - optex.ini, and only then ran fmtutil --no-strict --all (note that this is not a proper way to add formats, as this files gets overwritten by TeX Live, user config files belong elsewhere).

I used a few benchmarks:

% require-luaotfload.tex
\directlua { require "luaotfload" }

\bye

% initunifonts.tex
\initunifonts

\bye

% lm-fonts.tex
\fontfam[lm]

\bye

% lm-fonts-hello.tex
\fontfam[lm]

Hello world!

\bye

% lm-fonts-lipsum.tex
\fontfam[lm]

\lipsum[150]

\bye

And compared them with proper warmup and averaging over multiple runs using the excellent hyperfine tool:

$ hyperfine --warmup 20 --runs 30 -L inputfile require-luaotfload.tex,initunifonts.tex,lm-fonts.tex,lm-fonts-hello.tex,lm-fonts-lipsum.tex 'luatex -fmt texmf-var/web2c/luatex/optex.fmt {inputfile}' 'luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt {inputfile}'
Benchmark 1: luatex -fmt texmf-var/web2c/luatex/optex.fmt require-luaotfload.tex
  Time (mean ± σ):      33.1 ms ±   0.1 ms    [User: 26.3 ms, System: 6.9 ms]
  Range (min … max):    32.7 ms …  33.4 ms    30 runs

Benchmark 2: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt require-luaotfload.tex
  Time (mean ± σ):      34.0 ms ±   0.2 ms    [User: 26.9 ms, System: 7.2 ms]
  Range (min … max):    33.6 ms …  34.5 ms    30 runs

Benchmark 3: luatex -fmt texmf-var/web2c/luatex/optex.fmt initunifonts.tex
  Time (mean ± σ):     190.0 ms ±   1.3 ms    [User: 170.9 ms, System: 18.7 ms]
  Range (min … max):   187.8 ms … 192.4 ms    30 runs

Benchmark 4: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt initunifonts.tex
  Time (mean ± σ):     199.0 ms ±   6.1 ms    [User: 174.9 ms, System: 23.6 ms]
  Range (min … max):   191.3 ms … 215.7 ms    30 runs

Benchmark 5: luatex -fmt texmf-var/web2c/luatex/optex.fmt lm-fonts.tex
  Time (mean ± σ):     295.1 ms ±   7.8 ms    [User: 263.1 ms, System: 30.9 ms]
  Range (min … max):   280.2 ms … 308.4 ms    30 runs

Benchmark 6: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt lm-fonts.tex
  Time (mean ± σ):     292.3 ms ±   1.9 ms    [User: 262.0 ms, System: 29.6 ms]
  Range (min … max):   289.3 ms … 296.5 ms    30 runs

Benchmark 7: luatex -fmt texmf-var/web2c/luatex/optex.fmt lm-fonts-hello.tex
  Time (mean ± σ):     308.6 ms ±   3.2 ms    [User: 277.3 ms, System: 30.7 ms]
  Range (min … max):   299.5 ms … 313.7 ms    30 runs

Benchmark 8: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt lm-fonts-hello.tex
  Time (mean ± σ):     312.6 ms ±   3.1 ms    [User: 279.9 ms, System: 32.0 ms]
  Range (min … max):   303.5 ms … 317.2 ms    30 runs

Benchmark 9: luatex -fmt texmf-var/web2c/luatex/optex.fmt lm-fonts-lipsum.tex
  Time (mean ± σ):     585.3 ms ±   2.8 ms    [User: 549.6 ms, System: 34.5 ms]
  Range (min … max):   580.4 ms … 591.4 ms    30 runs

Benchmark 10: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt lm-fonts-lipsum.tex
  Time (mean ± σ):     583.8 ms ±   1.8 ms    [User: 548.6 ms, System: 34.2 ms]
  Range (min … max):   580.8 ms … 587.4 ms    30 runs

Usually luahbtex is consistenly worse by a few milliseconds, but different runs vary a bit (I just show one as example). I would say the difference is not much bigger than noise or the varience. For non-trivial documents the cost is negligible.

[Udi-Fogiel]

Though I have to admit, I have not followed the situation with non-latin scripts and ConTeXt / Harfbuzz for a few years.

Here is an example file comparing the two modes for Hebrew (the font is from here)

\initunifonts

\font\node={file:KeterYG-Medium:script=hebr;}
\font\harf={file:KeterYG-Medium:script=hebr;mode=harf;}
\def\text{%
וּֽבְקֻצְרְכֶם֙ אֶת־קְצִ֣יר אַרְצְכֶ֔ם לֹ֧א תְכַלֶּ֛ה פְּאַ֥ת שָׂדְךָ֖ לִקְצֹ֑ר וְלֶ֥קֶט קְצִֽירְךָ֖ לֹ֥א תְלַקֵּֽט׃
וְכַרְמְךָ֙ לֹ֣א תְעוֹלֵ֔ל וּפֶ֥רֶט כַּרְמְךָ֖ לֹ֣א תְלַקֵּ֑ט לֶֽעָנִ֤י וְלַגֵּר֙ תַּעֲזֹ֣ב אֹתָ֔ם אֲנִ֖י יְהֹוָ֥ה אֱלֹהֵיכֶֽם׃
וְלֹֽא־תִשָּׁבְע֥וּ בִשְׁמִ֖י לַשָּׁ֑קֶר וְחִלַּלְתָּ֛ אֶת־שֵׁ֥ם אֱלֹהֶ֖יךָ אֲנִ֥י יְהֹוָֽה׃
לֹֽא־תַעֲשֹׁ֥ק אֶת־רֵֽעֲךָ֖ וְלֹ֣א תִגְזֹ֑ל לֹֽא־תָלִ֞ין פְּעֻלַּ֥ת שָׂכִ֛יר אִתְּךָ֖ עַד־בֹּֽקֶר׃
לֹא־תְקַלֵּ֣ל חֵרֵ֔שׁ וְלִפְנֵ֣י עִוֵּ֔ר לֹ֥א תִתֵּ֖ן מִכְשֹׁ֑ל וְיָרֵ֥אתָ מֵּאֱלֹהֶ֖יךָ אֲנִ֥י יְהֹוָֽה׃
לֹא־תַעֲשׂ֥וּ עָ֙וֶל֙ בַּמִּשְׁפָּ֔ט לֹא־תִשָּׂ֣א פְנֵי־דָ֔ל וְלֹ֥א תֶהְדַּ֖ר פְּנֵ֣י גָד֑וֹל בְּצֶ֖דֶק תִּשְׁפֹּ֥ט עֲמִיתֶֽךָ׃
לֹא־תֵלֵ֤ךְ רָכִיל֙ בְּעַמֶּ֔יךָ לֹ֥א תַעֲמֹ֖ד עַל־דַּ֣ם רֵעֶ֑ךָ אֲנִ֖י יְהֹוָֽה׃
לֹֽא־תִשְׂנָ֥א אֶת־אָחִ֖יךָ בִּלְבָבֶ֑ךָ הוֹכֵ֤חַ תּוֹכִ֙יחַ֙ אֶת־עֲמִיתֶ֔ךָ וְלֹא־תִשָּׂ֥א עָלָ֖יו חֵֽטְא׃
לֹֽא־תִקֹּ֤ם וְלֹֽא־תִטֹּר֙ אֶת־בְּנֵ֣י עַמֶּ֔ךָ וְאָֽהַבְתָּ֥ לְרֵעֲךָ֖ כָּמ֑וֹךָ אֲנִ֖י יְהֹוָֽה׃
}
\textdirection=1\pardirection=1\node\text

\harf\text

\bye

The node mode does not really cope with situations where there are several nikud or te'amim (like accents) on one base letter.
node:
harf:
node:
harf:
node:
harf:

Note though, for Arabic the node mode is probably better than for Hebrew, as LuaTeX was developed with Arabic in mind, but I don't think it can cope with all possible shapers, section 6.1 of luaotfload's manual.

why we have LuaTeX without HB inside it? Historical reasons? If such reasons is out of date, then we can ask to LuaTeX maintainers to merge LuaTeX to LuaHBTeX. If this will be done then we need to do nothing in OpTeX.

This is probably because of different philosophies. The integration of HarfBuzz into LuaTeX wasn't done by the LuaTeX team, and if I recall correctly, Hans does not want it. This is the reason ConTeXt does not use it (and reading how LuaMetaTeX works this is not really surprising). The LaTeX team did want to use it as their default engine for the LuaLaTeX format, so it is maintained and shipped with TeXLive.