NewsDownloader: process HTML with cre.getBalancedHTML() to ensure self-closing tags like <hr> are closed like <hr/>

[Frenzie]

Works around the issue in #13173 (comment).

---

It's not a proper solution, since traditional HTML like the following would still cause issues, but it catches the low-hanging fruit of self-closing tags. Other than <hr> with its gray color I'm not sure if any of the other elements matter much in practice, but you might sometimes see some slightly curious indentation due to nesting.

<p>Paragraph one
<p>Paragraph two

The simplest solution would be to revert to plain HTML files, but EPUB does offer some advantages. (And I don't know if I want to put in the work required for that. :-)

A better solution might be to leverage crengine or MuPDF's HTML parsing, to output normalized HTML. Or perhaps to put in the mimetype text/html to signal that it's to be parsed differently.

---

This change is 

[poire-z]

Not sure I'd advise to use it - and require() crengine even if reading mostly PDF, although I guess newsdownloader users will end up requireing it - but I added a getBalancedHTML() not yet used in:
koreader/koreader-base#1824
koreader/crengine#569

[Frenzie]

I forgot all about that, thanks. I'll test it out. The replacements in this PR have the advantage of being super fast, but that only goes a small way.

[poire-z]

The available flags are described at https://github.com/koreader/crengine/blob/master/crengine/src/lvtinydom.cpp#L4326-L4344.
It's possible 0x50 adds unwanted newlines, which may cause spurious spaces with some inline elements, ie. foo<b>bar</b> may be serialized back as:

foo
  <b>bar</b>

which may then be rendered as foo bar instead of foobar.
May be 0x30 can avoid this ? Or even 0x0 if no human will ever look at the output :)

[Frenzie]

I'll what 0x20 and 0x30 output tomorrow. Some newlines are nice to have as a human.

[Frenzie]

May be 0x30 can avoid this ? Or even 0x0 if no human will ever look at the output :)

I'm not seeing any difference between the output from 0x0 and 0x30?

[poire-z]

Does 0x00 output a single line?
And 0x30 some more pretty multilines html ?
What about 0x50 ?
(All that is old, use what works for you :))

[Frenzie]

Does 0x00 output a single line?
And 0x30 some more pretty multilines html ?

No, that's what I mean, they both output a single line.

What about 0x50 ?

Yes, that one has the excessive spacing and newlines you referred to. :-)

[poire-z]

Ok, indeed, 0x30 just do as 0x00.
Might be because we don't render the document, so all "rend_method" have their default value of erm_invisible.
https://github.com/koreader/crengine/blob/e53a294e30be165cb2c91f7d7de2dcb9414a6f04/crengine/src/lvtinydom.cpp#L4581-L4597

If there were an erm_unitialized = 0 instead of erm_invisible, I could look the style display value, but not keen on thinking about all that :) Hope you're just fine with 0x00.

[PJansky]

I'm having issues with the the blog
www.wheresyoured.at/rss as it uses <hr> which messes up the formatting, as described in #13173 (comment) and was hoping this PR would fix it.
Unfortunately it seems no longer correctly create EPUBs for this site specifically, other feeds still seem to work. The nightly of 1. Feb works (with the formatting issues), however the nighty from 2. Feb and onwards with 56c0802 fails to correctly create the EPUB.
The created EPUBs open as:
Failed handling EPUB spine item 'OEBPS/content.html' (application/xhtml+xml, content).

[Frenzie]

You should use download full article false unless there's a really good reason not to btw. In this case it works fine with the default settings and it's a lot faster too:

As to the problem, fixed in #13260.

[PJansky]

Thanks a lot for resolving the issue this quickly. The newest nightly seems to work great