Two more wrongly encoded cases in LaTeXML.pm

[dginev]

While investigating #948 I spotted a broken bold 0 encoding when printing to stdout from latexmlc.

Turns out there were two cases that needed new patches

• One was the fatal error case for core processing, which was missing the encode call
• The other was the XML format case in post-processing, which was leveraging the duck-typing of all document objects (i.e. all of them having toString) to return differently encoded values -- since exactly one of our toString calls isn't returning bytes. So I added an explicit check there.

Keeps feeling awkward... I still don't care enough to raise a larger stink about refactoring :> If I hit this once again I might.

[dginev]

One way of seeing this may be that every use of the serialize_aux-based toString requires/expects encoded unicode. Which may be an indication that the encode call could be pushed more tightly into the actual method?

[brucemiller]

Actually that's not true. I should have caught that in your comments, but serialize_aux is internal to implementing LaTeXML::Core::Document->toString so (virtually) "nobody" uses it directly, but everybody uses toString and they expect characters. (should correct the comments). As I said elsewhere, only XML::LibXML::Document->toString (and toStringHMTL) return bytes and those really should be locked away to prevent confusion

[dginev]

Ah, but I phrased my claim extremely carefully this time.

All toString calls for the final output of the created Document by latexml, be it Core, Post, or some fragment in the --whatsout=fragment route, are currently expected to return bytes. All classic executables, as well as LaTeXML.pm, do a print assuming the payload is already in bytes.

This is the entire realm I was quantifying over when I said every, I was being quite careful.

The internal toString methods are not something i have studied closely recently, but I do recall the main workhorse internally tend to be the ToString and Stringify global functions instead.

Anyway, I think we are saying the same thing from 2 perspectives. Maybe renaming all Document methods away from toString into a serialize or serializeFinal call, that is always bytes, and mandating all toString calls return chars, would be the right refactor? And we refactor/rename away any uses of the toString calls in the executables and Writer, to be sure.

[brucemiller]

Oh! I forgot to merge... is this one still needed?

[dginev]

It is still needed, yes.

I thought you left it open so that I rework it so that there is a specially named method that always encodes to Unicode, but if you have decided that is a long-term goal, merging here is still a good choice. Fixes a problem.

[brucemiller]

Ah, not really; a nice safe general approach will be good, but I'm not waiting.