Math parsing problem of `\boldsymbol{0}` ?

[asmaier]

When converting the following formula to MathML (Mac OS X, v0.8.2 of LaTeXML), it seems the number 0 somehow get's lost:

$ latexmlmath "\displaystyle\boldsymbol{0}" --preload=amsmath
<?xml version="1.0" encoding="UTF-8"?>
<math xmlns="http://www.w3.org/1998/Math/MathML" alttext="\displaystyle\boldsymbol{0}" display="block">
  <mn/>
</math>

One can see in the MathML output, that the number 0 is missing. It doesn't happen with other numbers like \boldsymbol{1}:

$ latexmlmath "\displaystyle\boldsymbol{1}" --preload=amsmath
<?xml version="1.0" encoding="UTF-8"?>
<math xmlns="http://www.w3.org/1998/Math/MathML" alttext="\displaystyle\boldsymbol{1}" display="block">
  <mn>𝟏</mn>
</math>

It also doesn't happen, if I change the formula and write the 0 without \boldsymbol:

$ latexmlmath "\displaystyle 0" --preload=amsmath
<?xml version="1.0" encoding="UTF-8"?>
<math xmlns="http://www.w3.org/1998/Math/MathML" alttext="\displaystyle 0" display="block">
  <mn>0</mn>
</math>

What is going on?

[dginev]

This should be an easy fix for a classic Perl error where you have an if conditional testing if something is defined, but the literal string "0" ends up false in Perl. Sometimes they slip in, it's a great catch

[brucemiller]

Yeah; I have no doubt that this happened, and probably exactly how @dginev says. But it seems that it's already been fixed. If that turns out not to be true, please reopen. Thanks for the report.

[asmaier]

When I try the same example with the LaTeXML version from HEAD, I do see the following:

$ latexmlmath "\displaystyle\boldsymbol{0}" --preload=amsmath
<?xml version="1.0" encoding="UTF-8"?>
<math xmlns="http://www.w3.org/1998/Math/MathML" alttext="\displaystyle\boldsymbol{0}" display="block">
  <mn>�</mn>
</math>

and also

$ latexmlmath "\displaystyle\boldsymbol{1}" --preload=amsmath
<?xml version="1.0" encoding="UTF-8"?>
<math xmlns="http://www.w3.org/1998/Math/MathML" alttext="\displaystyle\boldsymbol{1}" display="block">
  <mn>�</mn>
</math>

So I don't think this issue is fixed correctly.

[brucemiller]

Well, if you output to a file, it seems to be the correct unicode, it's just mangled on stdout. And this seems due to the @dginev's addition of binmode in latexmlmath which seems to me at best to be correct and at worst to be harmless. But apparently it isn't? I'm a bit baffled. Any intuitions on this @dginev?

[dginev]

Yeah, it is obviously double encoded, and upon inspecting the source of latexmlmath, the culprit is also rather proudly announcing itself, namely:

my $newdoc = XML::LibXML::Document->new("1.0", "UTF-8");

Here is a standalone test snippet that demonstrates the double encoding:

use XML::LibXML;

my $newdoc = XML::LibXML::Document->new("1.0", "UTF-8");
my $newel = XML::LibXML::Element->new("проба");
$newdoc->setDocumentElement($newel);
my $serialized = $newdoc->toString(1);

print "1. \n", $serialized,"\n\n";

binmode(STDOUT, ":encoding(UTF-8)");

print "2. \n", $serialized,"\n\n";

which outputs:

1. 
<?xml version="1.0" encoding="UTF-8"?>
<проба/>


2. 
<?xml version="1.0" encoding="UTF-8"?>
<п�оба/>

First being correct, second double-encoded. So I guess latexmlmath was correct before my supposed "fix", as the serialization of libxml took into account the encoding of the document? Confusing...

[dginev]

LaTeXML::Document initializes the document in the exact same way:

my $doc = XML::LibXML::Document->new("1.0", "UTF-8");

So I am starting to go back to confused about why the serialization doesn't encode correctly...

[dginev]

Ah, I think I got it.

The difference is that the Unicode for the math case was generated by LaTeXML - and I believe was already encoded internally as UTF-8, which is why it is getting the double encoding.

My fixes for STDOUT printing were specifically for pass-through content that came from input as Unicode, and got decoded by latexml for internal use.

In other words, we are handling inconsistently encodings of content generated internally by latexml, and content entering from input TeX.

[dginev]

Tracing further, I think it boils down to having %plane1map and similar lookup tables in MathML.pm that are already encoded as Unicode, and end up double-encoded on printing to STDOUT.

Not brave enough to suggest which part to refactor in which direction yet, but I think this may be the root issue here.

[brucemiller]

Well, now I'm more confused than ever! :> Actually, our XML (& its serializations) should always be utf8 (or so I thought); not just with the plane1 conversions, or even for math, but always unless it's in the ASCII subset. So, I'm really confused about what binmode is actually for; I'd thought it just prepared the stream for receiving utf8; if we give it utf8 then no encoding necessary. OTOH, if I take your interpretation, we probably should never use binmode? I don't understand the 2nd comment above about "decoded by latexml for internal use"; if latexml reads in xml, it should also be utf8, right?

[brucemiller]

Ah, I think the issue is that XML::LibXML::Document->toString converts to bytes, not chars, according to the document's encoding, so it's already been encoded (to utf8 in our case). (ugh) This is contrary to other XML toString methods, and the reasoning is apparently so that one can simply use print $string.

That hardly seems desirable, but in any case, we need to be clearer about when we have bytes vs chars.