Compatibility with luaotfload 2.6

[eroux]

It seems Gregorio is not compatible with latest luaotfload (2.6), see this thread

[henryso]

@rpspringuel Would you be able to test this against TeX Live 2014?

[henryso]

It's still not working against luaotfload 2.6 and the latest TeX Live 2015. I'll see if I have time this weekend to try to dig into why it's broken, but I'll take any help anyone can offer.

[eroux]

Thanks for your help, I'm still unable to do anything before Monday, but I think a good start would be to isolate the problem outside of Gregorio (working in raw TeXLive, requiring no gregorio font nor files, but including the font access code of Gregorio), and make a PR against the luaotfload test repo. This example should raise an obvious error at compile time if the bug is detected. @phi-gamma would that be ok for you?

[phi-gamma]

It's still not working against luaotfload 2.6 and the latest TeX Live 2015. I'll see if I have time this weekend to try to dig into why it's broken, but I'll take any help anyone can offer.

I”ll look into it as soon as I can, probably on monday. The table
accesses were the first thing that came to my mind as a potential
source of errors. (Indexing font.fonts is destructive and
overwrites parts of fonts defined by Luaotfload.)

[henryso]

The problem is this function in gregoriotex.lua:

local function map_font(name, prefix)
  log("Mapping font %s", name)
  local glyph, unicode
  for glyph, unicode in pairs(get_score_font_resources(name).unicodes) do
    if unicode >= 0 and not string.match(glyph, '%.') then
      log("Setting \\Gre%s%s to \\char%d", prefix, glyph, unicode)
      tex.sprint(catcode_at_letter, string.format(
          [[\xdef\Gre%s%s{\char%d}]], prefix, glyph, unicode))
    end
  end
end

The for glyph, unicode in pairs(get_score_font_resources(name).unicodes) loop only iterates four glyphs from the greciliae.ttf font:

• zwj = 8205
• hyphen = 45
• zwnj = 8204
• space = 32

We have thousands of glyphs in the private use area; none of them are iterated by this loop.

The relevant part of the log is:

luaotfload | cache : Lookup cache loaded from /usr/local/texlive/2015/texmf-var/
luatex-cache/generic/names/luaotfload-lookup-cache.luc.
luaotfload | load : Lookup/name: "greciliae" -> "greciliae.ttf"(load luc: /usr/l
ocal/texlive/2015/texmf-var/luatex-cache/generic/fonts/otf/greciliae.luc)
Module gregoriotex Info: Mapping font greciliae on input line 873
Module gregoriotex Info: Setting \GreCPzwj to \char8205 on input line 873
Module gregoriotex Info: Setting \GreCPhyphen to \char45 on input line 873
Module gregoriotex Info: Setting \GreCPzwnj to \char8204 on input line 873
Module gregoriotex Info: Setting \GreCPspace to \char32 on input line 873

[eroux]

Thanks for the investigation! Do you think you could make a small plain TeX file (as simple as possible, no packages loaded except luaotfload), compilable with the luatex command that would demonstrate the problem (of course it would only compile with greciliae.ttf in the same directory)?

[henryso]

\input luatexbase.sty
\input luaotfload.sty

\newluatexcatcodetable\grecc %
\setluatexcatcodetable\grecc{%
  \catcode`\@=11 %
}%

\font\greciliae={file:greciliae.ttf} at 200pt\relax

\def\luacode{%
  \bgroup
  \catcode`\\=12
  \catcode`\{=12
  \catcode`\}=12
  \catcode`\^^M=12
  \catcode`\#=12
  \catcode`\~=12
  \catcode`\%=12
  \doluacode
}

\bgroup
\catcode`\|=0
\catcode`\^^M=12 %
\catcode`\\=12 %
|long|gdef|doluacode#1^^M#2\endluacode{|directlua{#2}|egroup}%
|egroup

\luacode
local grecc = luatexbase.catcodetables['grecc']
local unicodes = font.getfont(font.id('greciliae')).resources.unicodes
local glyph, unicode
tex.print(grecc, [[Mapping font\par ]])
for glyph, unicode in pairs(unicodes) do
  if unicode >= 0 and not string.match(glyph, [[%.]]) then
    tex.print(grecc, string.format([[%s = %d\par ]], glyph, unicode))
    tex.print(grecc, string.format([[\xdef\GreCP%s{\char%d}]], glyph, unicode))
  end
end
tex.print(grecc, [[That's all, folks!\par]])
\endluacode
And the winner is ... \greciliae\GreCPPunctumInclinatum
\bye

This code works under 2.5. It produces a large PDF and shows a punctum inclinatum at the end. Under 2.6, it breaks with:

luaotfload | main : initialization completed in 0.036 seconds)(load luc: /usr/lo
cal/texlive/2015/texmf-var/luatex-cache/generic/fonts/otf/greciliae.luc)
! Undefined control sequence.
l.43 ...ner is ... \greciliae\GreCPPunctumInclinatum

If you comment out the penultimate line, it produce a single page PDF listing the four glyphs I mentioned earlier.

[eroux]

Thanks a lot! I've reported it in luaotfload issue

[eroux]

The example is fixed when using

local grecc = luatexbase.catcodetables['grecc']
if (luaotfload.version >= "2.6") then
   local descriptions = font.getfont(font.id('greciliae')).shared.rawdata.descriptions
   local name, unicode
   tex.print(grecc, [[Mapping font\par ]])
   for _, description in pairs(descriptions) do
     unicode = description.unicode
     name = description.name
     if unicode and unicode >= 0 and not string.match(name, [[%.]]) then
       tex.print(grecc, string.format([[%s = %d\par ]], name, unicode))
       tex.print(grecc, string.format([[\xdef\GreCP%s{\char%d}]], name, unicode))
     end
   end
else
   local unicodes = font.getfont(font.id('greciliae')).resources.unicodes
   local glyph, unicode
   tex.print(grecc, [[Mapping font\par ]])
   for glyph, unicode in pairs(unicodes) do
     if unicode >= 0 and not string.match(glyph, [[%.]]) then
       tex.print(grecc, string.format([[%s = %d\par ]], glyph, unicode))
       tex.print(grecc, string.format([[\xdef\GreCP%s{\char%d}]], glyph, unicode))
     end
   end
end
tex.print(grecc, [[That's all, folks!\par]])

in your example, I'm not sure about the conditional though...

[henryso]

Would it be better check if there is a field called unicode in shared.rawdata.descriptions and use that rather than checking the luaotfload version?

[eroux]

Why not yes, I'm just waiting for a possible answer on luaotfload tracker, if it doesn't come, it might be the best solution.

[phi-gamma]

The example is fixed when using
…

local unicodes = font.getfont(font.id('greciliae')).resources.unicodes

The problem with that code is that the “unicodes” table is lazily
constructed. In order to access its values, iterating won’t
suffice. Construction can be forced e. g. by mutating use of
__index. I think that rules out a fontloader / luaotfload bug
=)

https://bitbucket.org/phg/lua-la-tex-tests/src/tip/plain/pln-gregorio-1.tex

In any case, I’ve looked at the code and I’d prefer retrieving
the font data at definition time. That’s exactly what we provide
the luaotfload.patch_font callback for -- an API for raw
accesses to the fonts being defined, avoiding fontloader or
luatex internals like font and getfont().

Here’s an example:

https://bitbucket.org/phg/lua-la-tex-tests/src/tip/plain/pln-gregorio-2.tex
https://bitbucket.org/phg/lua-la-tex-tests/src/tip/plain/pln-gregorio-2.lua

This approach splits the task into two subtasks: Collect unicodes
by means of a callback and then inject them from a macro.

[henryso]

How backwards compatible is retrieving "unicodes" via the callback? Does it work in TeX Live 2013? 2014?

[eroux]

I think it should be rather safe. I admit I didn't understand the __index thing immediately... For those in the same situation, you can look here. It's quite a good solution indeed. It looks extremely involved, but it's not... Thanks a lot @phi-gamma !

[phi-gamma]

@henryso The workaround only has an effect if the unicodes table
indeed has a metatable and is safe otherwise. As to the presence of
that table in the font data: all should be considered in flux until Luatex
reaches 1.0. Apart from that general disclaimer, it works with the TL
2014 fontloader shipped with the current Luaotfload release. I didn’t
test on an actual TL 2014, though.