Add xml5 parser to html5ever

[Ygg01]

This large commit concerns #43. It adds everything that is currently defined in spec.

Included inside:

• Basic machinery - Tokenizer/States/Tree builder/etc. that implements following spec: Ygg01/xml5_draft
• Tests, which reference: Ygg01/xml5lib-test
• Some basic examples

Missing:

• Namespace resolution
• Doctype handling as per Ygg01/xml5_draft#2
• Entity replacement (this is mostly C'n'P from html5ever character tokenizer), so all html5 entities are valid (e.g. ‐ Á etc.)
• Unification of xml5 and html5 syntax
• C API

[Ygg01]

It seems I'll have to upgrade Rust for Travis to pass. Depends on #124

[Ygg01]

@kmcallister Should I rebase now or wait for Tendril to be integrated?

[bors-servo]

☔ The latest upstream changes (presumably #144) made this pull request unmergeable. Please resolve the merge conflicts.

[SimonSapin]

Sorry to only say this after you’ve done so much work, but does this really need to be part of html5ever? How much code is there in common?

[Ygg01]

@SimonSapin Does it need to be part of html5ever? It depends on whether the servo needs something that parses lax XML files? I think Kuchiki might find it interesting to parse XML. Servo? I dunno, maybe parsing XHTML files?

The html5ever and xml5ever share a surprising amount of code. It could share more though. Lots of code is duplicated, but that is due to the way I handle Tokens (I have some tokens </>, which could be dealt with end tokens with no names and Empty <b/> tokens which could be emitted as start/end tag instead of a special tag). I'm thinking changing spec so so that I could have more or less 1-1 map between XML5 and HTML Tokens and thus less differences between the two parsers.

Major differences are state machine and tree builder, and anything that Token related like TokenSink. Rest of it, is pretty similar.

Is there is no interest in parsing XML in Servo or other libraries, then that's that, but if the problem is lack of common code, I could attempt a second take that will share more code.

[SimonSapin]

Servo will probably want to have some form of XML parsing at some point, but it’s unclear yet whether XML5 is the way to go. As far as I know, XML5 has never been deployed in a web browser, or even used much beyond a few test cases. Don’t get me wrong, I like the ideas in XML5, but it’s not an established solution yet.

My question regarding code in common is: could XML5 parsing be a separate library?( Either one that depends on html5ever if reusing some components is useful, or an independent one.) Or is there so much internals in common that you’d rather not duplicate it?

[Ygg01]

Servo will probably want to have some form of XML parsing at some point, but it’s unclear yet whether XML5 is the way to go.

I see. In that case could you close #43 ?

My question regarding code in common is: could XML5 parsing be a separate library?( Either one that depends on html5ever if reusing some components is useful, or an independent one.)

Difficult question. From my memory, it's plausible that I could separate it into a library. However there will be much code duplication.

I will try during this weekend to separate it and see how it goes, but all signs point to that it's plausible.

[SimonSapin]

I see. Don’t worry about separating it yet, don’t spend too much time on it this week end. The Servo team is meeting next week, I’ll bring this up then so we can decide what to do.

I appreciate the work you’ve done so far and I’d like to figure out something that’s not just dumping a bunch more work on you :)

[metajack]

Agreed that we should discuss it at Whistler, although really, we can't know how good XML5 is without actually doing it. So I am mostly interested what the main drawbacks of landing this are assuming we discover later that it doesn't satisfy our requirements.

[Ygg01]

@SimonSapin Well whatever work I put into it now, it's not going to go in vain either way :)

@metajack

we can't know how good XML5 is without actually doing it.So I am mostly interested what the main drawbacks of landing this are assuming we discover later that it doesn't satisfy our requirements.

Good in what way? Speed? Memory consumption? Compatibility with older XML? Ease of removal?

Main drawback I see is searching all of ways xml parser is referenced and not confusing it for parts of html5ever. However, I could make it easier to remove. I could pull my additions into xml folders so removing them is essentially the same as finding and deleting all xml subfolders within src, removing tests/examples that use them and removing all references that cause compile to fail. It was one of changes I had in mind, amongst other things.

Only change that touches html5ever, that I know of is replacing fn elem_name(&self, target: Handle) with fn elem_name(&self, target: &Handle). And that change removes a bunch of clones, without affecting tests.

[SimonSapin]

I think Jack means good in terms of compatibility with existing web content.

[Ygg01]

Hard to say without some examples of what said web content entails (MathML? SVG? XHTML?).

On token level xml5 and html5 are quite similar. However, when it comes to tree construction they might interpret things differently (e.g. adoption agency algorithm doesn't exist in xml5).

[Ygg01]

@SimonSapin @metajack Was a decision regarding this PR been made?

[SimonSapin]

Sorry, it turned out that we spent more time meeting with other Mozilla teams than actually working on Sevo last week, so this wasn’t discussed.

I still feel that HTML parsing and XML(5) parsing are different things, and would prefer having different things in separate crates. Rather than ask you to do yet more work, I had a go at it: https://github.com/SimonSapin/xml5ever . I started from a rebase of this branch and a new empty crate, copied the lines showed as new in the diff into new files, and added/fixed stuff until it built and passed the tests. It depends on string-cache and tendril, but not html5ever. According to diff stats there are ~1042 lines duplicated with html5ever, including 289 (rcdom) only used in tests.

What do you think?

---

Hard to say without some examples of what said web content entails (MathML? SVG? XHTML?).

Yes, predicting web-compatibility is hard. What Jack meant by actually doing it is that the only way to be sure is to ship a browser to enough users and wait for bug reports. The said web content is all of it. Any web site that users might care enough about to switch browsers if we break it.

[Ygg01]

Well, it's good to know either way :)

@SimonSapin Wait xml5ever doesn't depend on html5ever? Shouldn't it at least reuse the dom sink crate? Nevermind, dom sink depends on html5ever and not the other way around. Which I find weird (shouldn't dom sink be a common interface that html5ever implements?), but is otherwise ok.

Anyway, awesome that you separated it ❤️ , I'll probably start whipping xml5ever into shape. The problem for me is that xml5ever will need regular updates to be in line with html5ever, if it's ever to be integrated. But I assume it's possibly a non-problem.

[SimonSapin]

Nevermind, dom sink depends on html5ever and not the other way around. Which I find weird (shouldn't dom sink be a common interface that html5ever implements?)

As I see it, dom_sink provides two "toy" implementations of the TreeSink trait that are fairly minimal and not meant for practical use. One (rcdom) is used to run tests, and the other (owned_dom) seems to be an experiment to remove reference-counting overhead.

Compare with Kuchiki, another implementation of html5ever’s TreeSink that I hope can support many use cases, like lxml Nokogiri.

At the moment, xml5ever’s TreeSink differs from html5ever’s in adding create_pi and removing set_quirks_mode. If the ecosystem ends up with many parsers and many tree data structures we could have a standardized TreeSink trait to connect them all, but I don’t think we’re there yet.

The problem for me is that xml5ever will need regular updates to be in line with html5ever, if it's ever to be integrated. But I assume it's possibly a non-problem.

Well, my opinion is that it doesn’t need to be integrated and I’m not convinced it should. What do you think? If you think it should, why?

[Ygg01]

What do you think? If you think it should, why?

Good question. I think since the specification is pliable, and XML5 could be modified, it would be possible to make it work with any web content and thus fit the role of XHTML parser. It's what I originally set out to do (add XML5 to html5ever).

On technical side. It occurs to me, that most optimization, at least when it comes to do with tokenization, could apply to xml5ever as well as html5ever (e.g. tendrils, SIMD, etc.). Although to be fair, it's possible you could do extra XML optimization, that probably couldn't apply to HTML. In that sense, I think there could be lot of duplication between the two projects.

Then again, the shared code could be divided into crates and shared between projects.

PS. However, for the time being you're probably right :( I'll go ahead and close this PR.

[SimonSapin]

Tendril is a separate crate that both parsers can use. Could the same be done for other optimizations?

[Ygg01]

Hard to make predicitions, but I don't see why not.

[SimonSapin]

Could you click the Fork button on https://github.com/SimonSapin/xml5ever ? Then I’ll re-fork from you so that you appear as upstream. Thanks!

[Ygg01]

@SimonSapin Done - I guess. Never forked a fork before.

Anyway, thanks sorting xml5ever out 😃

[SimonSapin]

Alright, https://github.com/Ygg01/xml5ever it is :)