Patch \symbol under XeTeX (or fix upstream)

[wspr]

Brief outline of the bug

In XeTeX \Uchar should be used instead of \char in some cases.
More details here: latex3/unicode-math#499

Minimal example showing the bug

\RequirePackage{latexbug}
\documentclass{article}
\begin{document}
\font\x="[STIX2Math.otf]"\relax\x
{$\tracingall\symbol{"1D400}$}
\end{document}

Log file (required) and possibly PDF file

\symbol #1->\char #1\relax 
#1<-"1D400
{\char}
./small.tex:8: Bad character code (119808).
<to be read again> 
                   \relax 
l.8 {$\tracingall\symbol{"1D400}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[davidcarlisle]

\symbol isn't generally expandable so this should probably be fixed upstream, unless we want to make symbol expandable in all engines. Is it reported at the xetex tracker?

[josephwright]

@davidcarlisle But \symbol is already robust, so using \Uchar internally makes no odds.

[u-fischer]

@davidcarlisle I see no reason why we shouldn't do this

\ifx\Umathcode\@undefined
\DeclareRobustCommand\symbol[1]{\char #1\relax}
\else
\DeclareRobustCommand\symbol[1]{\Uchar #1\relax}
\fi

[davidcarlisle]

@josephwright it isn't really about \symbol (and the extra \relax from the \protect wouldn't hurt here, it is about \char

I don't think this should give an error in xetex (it works in luatex)

\documentclass{article}
\begin{document}
\font\x="[STIX2Math.otf]"\relax\x
{$\char"1D400 $}
\end{document}

[josephwright]

@davidcarlisle OK, so we mark as 'report elsewhere', prod Arthur/Jonathan about XeTeX, close here as it's Not Our Fault?

[FrankMittelbach]

I'm confused. Running the code @davidcarlisle showed above outputs nothing for me (other than the page number) if I use LuaTeX and it makes an error if I use XeLaTeX. And If I replace it with \Uchar the only difference is that then both engines print nothing and no error in math for XeTeX.

Outside of math both \char and \Uchar do the same in both engies and print the symbol.

[FrankMittelbach]

Maybe the question here is what should be the specification of \symbol if used in math, I mean if your math has 8bit fonts then clearly specifying "1D400 doesn't get you far.

[blefloch]

Amusingly, \lccode`*="1D400 \lowercase{\show*} gives the correct output of show (namely a math character) but wrong output in the "recently read" line, which is actually the CJK letter U+D400:

the character 𝐀.

<recently read> \show 퐀

[FrankMittelbach]

@blefloch well that clearly seems to indicate that something is not quite right on the engine level, but I still wonder if we understand what should be considered "right" in the various cases.

[u-fischer]

I'm confused. Running the code @davidcarlisle showed above outputs nothing for me (other than the page number) if I use LuaTeX and it makes an error if I use XeLaTeX.

Inside math cmmi (a legacy font ) is used. This naturally has no glyph in this unicode position. luatex handles this gracefully and simply reports a missing char. Xetex errors.

Outside math, when the stix font is used, both show the correct char.

Using a legacy font in text mode

\font\test=cmr10 \test
\char"1D400

works with both engines too (with missing char messages and naturally no output).

xelatex fails only in math (and the stix font in the example is actually irrelevant):

\documentclass{article}
\begin{document}

$\char"1D400 $
\end{document}

[FrankMittelbach]

I'm confused. Running the code @davidcarlisle showed above outputs nothing for me (other than the page number) if I use LuaTeX and it makes an error if I use XeLaTeX.

Inside math cmmi (a legacy font ) is used. This naturally has no glyph in this unicode position. luatex handles this gracefully and simply reports a missing char. Xetex errors.

understood that part, yes. And assuming that the math is not using a legacy font then luatex would print a char?

if so I don't quite understand why your suggestion to make \symbol use \Uchar with these engines is not and answer to the problem (which seems to be what @davidcarlisle was hinting at).

[u-fischer]

if so I don't quite understand why your suggestion to make \symbol use \Uchar with these engines is not and answer to the problem

It certainly avoids the problem, \Uchar does the right thing, also it is imho sensible anyway to use \Uchar with the unicode engines, so I don't see a problem to change \symbol to use it. But @davidcarlisle is imho right that the xetex behaviour looks like a bug, so it should be reported.

[u-fischer]

And assuming that the math is not using a legacy font then luatex would print a char?

with unicode-math luatex prints a char, but xetex still errors.

\documentclass{article}
\usepackage{unicode-math}
\begin{document}
abc
$\char"1D400 $
\end{document}

[FrankMittelbach]

Never really looked into the \Uchar \char specs in the Unicode engines, but do I see it right that in LuaTeX the both are 100% equiv? And that in XeTeX one could always use \Uchar instead of \char? or are there other differences? Because if not, then we could drop \char in Unicode engines by setting \let\char\Uchar, could we not?

[josephwright]

\Uchar is expandable, and is getting added to pTeX ...

[davidcarlisle]

as it's being added to ptex, we could consider adding it to pdftex as well, but I still would say the main issue here is a xetex bug. If we make \symbol latex \protect ed but with an expandable internal definition I suppose it's more or less compatible with a \char based definition, although there would be a temptation to remove the protection so it expanded in a \typeout in that case

[zauguin]

Never really looked into the \Uchar \char specs in the Unicode engines, but do I see it right that in LuaTeX the both are 100% equiv? And that in XeTeX one could always use \Uchar instead of \char? or are there other differences? Because if not, then we could drop \char in Unicode engines by setting \let\char\Uchar, could we not?

No, they aren't equivalent. Beside ``\Ucharbeing expandable another difference is that\Uchar 32` becomes a space token and therefore inserts (space) glue, while `\char 32` inserts the glyph at code point 32. (This is an artifact of `\Uchar` using the same catcode rules as other expandable primitives expanding to text) E.g.

\documentclass{article}
\begin{document}
\parfillskip 0pt
\hsize 5cm
aaa\Uchar 32 aaa

aaa\char 32 bbb
\end{document}

stretches the first line but not the second.

I think \symbol should definitly give the \char behaviour, so if this is changed to \Uchar there should be a special case for symbol 32.

[blefloch]

On 11/30/19 2:12 AM, Marcel Krüger wrote: I think |\symbol| should definitly give the |\char| behaviour, so if this is changed to |\Uchar| there should be a special case for symbol 32.

Good catch! But special-casing symbol 32 isn't the answer: instead, just use |\Ucharcat #1 12 |.

[davidcarlisle]

@blefloch unless you use the l3 (or ucharcat package) Lua implementation that's not available in luatex. I think we should first try to get \char fixed in xetex.

[blefloch]

Even if \char is fixed in XeTeX it will still be wrong on older installs so rolling forward with latexrelease will not work. We could instead use \Ucharcat exclusively in XeTeX.

[stale]

This issue has been automatically marked as stale because it has not had recent activity.