Switch to generating HTML ourselves

[bhollis]

Switch to generating HTML ourselves, without the help of any XML/HTML libraries. This gives us much better direct control of HTML output, makes it much easier to provide consistency across Rubies, removes exposure to Nokogiri bugs, and furthers the goal of removing Nokogiri as a required dependency of Maruku.

@distler, I'm especially interested in how this works in your application, since you seem to really put Maruku through its paces and can probably find some problems that aren't yet in tests.

[bhollis]

Rebased.

[distler]

This will require a lot more checking ... :-)

[bhollis]

I added in some fixes to remove the itex2mml and blahtex Nokogiri dependencies. Blahtex looked like it was pretty broken, so I put in some tests and go it working again.

[bhollis]

And now S5 is cleaned up too.

[distler]

Your "blahtex" output looks like MathML to me.

[bhollis]

Thanks, @distler, I'll endeavour to correct my mistakes.

Speaking of which, the TOC generation change affected output, so I rewrote that commit to include modified tests.

[bhollis]

Also, I have no experience with blahtex - I thought the point was for it to output MathML when you use the --mathml flag?

[distler]

The library was designed to do either (output mathml or png's), at the flip of a switch. (It was originally written as a proposed replacement for texvc in MediaWiki. The idea was that Wikipedia could continue generating png's for math, as it currently does and then, at some point in the future, flip a switch and start generating MathML instead. Blahtex never found favour with the MediaWiki people, so development sorta languished.)

In Maruku, Blahtex has mostly been used to output png's (for those who don't want/can't handle MathML).

The PNG output is a little hard to test properly

• It requires a working TeX installation to process the input.
• The output involves a bunch of pictures of the equations we're trying to generate.

But that's what's actually used, in practice, so it's what we want to test.

[bhollis]

OK, sure, it's definitely worth testing the PNG output. My changes simply fix the MathML output (when :html_math_output_png has not been turned on).

In other news, this viewBox thing is turning out to be somewhat difficult. Something is weird where Nokogiri::XML::DocumentFragment will lowercase all attributes when parsing MathML that includes SVG. Nokogiri::XML::Document.parse is fine, and parsing just SVG through DocumentFragment is fine. It smells like this bug: sparklemotion/nokogiri#455 - I wonder if "HTML mode" is getting triggered somehow.

[distler]

Something is weird where Nokogiri::XML::DocumentFragmentwill lowercase all attributes when parsing MathML that includes SVG.

I assume this is a problem with the JRuby version? I have never seen that under MRI.

[bhollis]

Nope, this is 1.6.0 under Ruby 2.0.0. Check it out: sparklemotion/nokogiri#961

[bhollis]

I think I'm too tired to keep hammering on this tonight. I've gotten it so it either preserves case on that attribute, or preserves math entities in the document, but not both. Even when I switch HTMLFragment to use the same exact code for parsing the document that the itex2mml module used to. Maybe this'll make sense later.

[bhollis]

Hmm, now I see the problem - I confused to_utf8 from itex_stringsupport with as_utf8. I'm not exactly sure I understand what to_utf8 is doing though - what library adds convert_to_utf8 to String? That seems to be the key to preventing Nokogiri's freakout.

[bhollis]

That's not even entirely it - adding to_utf8 fixes my reduced repro case, but not the full test.

[distler]

#to_utf8 converts MathML named entitites to the corresponding utf8 characters.
Nokogiri (and any other legitimate XML parser) would freak out if presented with named entities without the DTD which defines them.

Its partner is #to_ncr which converts named entities to numeric character references.

[bhollis]

That makes sense. Do you know where convert_to_utf8 is defined? I'd like to have that code available within Maruku without relying on itextomml.

[distler]

Do you know where convert_to_utf8 is defined?

Sure (since I wrote it). It's in lib/itex_stringsupport.rb in the gem. Or you can go to the source code repository to find it.

[bhollis]

OK, I think I have this thing licked.

After switching to parsing fragments as a full document with a DOCTYPE, we preserve named entites and the case of attributes. itex_stringsupport isn't even needed, but I'm leaving it in because without it, math prints as named entities rather than as Unicode characters.

What do you think?

[distler]

Maruku's output should definitely use either unicode characters or NCRs (rather than named entities).

[distler]

Also, I'm surprised that the presence or absence of a DOCTYPE declaration makes a difference.

[bhollis]

Yeah, it's a pretty weird way to fix it. Let me know what it looks like running on your documents and I'll merge.

[bhollis]

No hurry, but let me know when this is good to merge.

[bhollis]

Manually merged.

[coveralls]

Changes Unknown when pulling 5ce7c41 on string-html into * on master*.