Scriptstyle primes in XeLaTeX with XITS font

[Caramdir]

When compiling a document with XITS Math with XeLaTeX, primes ' in scriptstyle are not displayed correctly.

For example, the document

\documentclass{article}

\usepackage{unicode-math}
\setmathfont{xits-math.otf}

\begin{document}
$a'\frac{b'}{c'}$
\end{document}

results in

For comparision, LuaLaTeX produces

LM Math and Asana Math seem to produce correct outputs, so I don't know whether the problem lies with unicode-math, the XITS fonts or XeTeX.

[wspr]

Troublesome primes! Thanks for the report; I'll attempt to look into this soon. I don't have any idea off the top of my head why this would be the case.

[khaledhosny]

Both LM Math and Asana Math use large (text size) prime glyphs in the regular slot while XITS and Cumbria use a small one and and ssty feature maps to the large variant, that explains the different behaviour. I suspect there is something wrong with the application of ssty feature.

[wspr]

This appears to be a problem with XITS; it doesn't have a second ssty feature for script-script sizes, and XeTeX and luaotfload differ in their behaviour.

    \documentclass{article}
    \usepackage{ifluatex}
    \ifluatex \usepackage{luaotfload}\fi
    \begin{document}

    \font\1="[xits-math.otf]:script=math"
    \ifluatex
      % n.b. luaotfload's off-by-one selection behaviour
      \font\2="[xits-math.otf]:script=math;+ssty=1"
      \font\3="[xits-math.otf]:script=math;+ssty=2"
    \else
      \font\2="[xits-math.otf]:script=math;+ssty=0"
      \font\3="[xits-math.otf]:script=math;+ssty=1"
    \fi

    \1 x′ \qquad \2 x′ \qquad \3 x′

    \end{document}

In XeTeX, asking for ssty=1 when it doesn't exist ignores it entirely, whereas in luaotfload asking for ssty=2 falls back onto ssty=1.

As there is no way to query this information from within XeTeX (not sure about LuaTeX), I'm going to have to say this is a font bug and close the issue here. For what it's worth, you can work around the issue by requesting the "script style" feature even in "script script style" contexts:

\setmathfont[sscript-features={Style=MathScript}]{xits-math.otf}

The Style=MathScript feature isn't documented yet...

[khaledhosny]

Yes, that is a font bug. It was masked by luatex behaviour, but I found about it while ago, I thought I fixed it already, but may be that was another font. I pushed a fix to the repository.