html5ever pull tokenizer?

[TyOverby]

I'm working on an application where the push-tokenizer that is built into html5ever is not very ergonomic.

Instead of making a sink and having process_token get called on it, I would rather enqueue a Tendril, and get an Iterator of Tokens that are parsed and returned on-demand.

My question is: would this be an acceptable option for html5ever? I wouldn't mind doing the implementation.

[jdm]

@nox and @SimonSapin, any opinions here?

[TyOverby]

A pull-based model would also allow for abandoning a parse midway through.
I also think you could implement push parsing on top of pull parsing.

for token in tokenize(input)
    sink.process(token)

[SimonSapin]

push parsing on top of pull parsing.

What about the tokenizer’s own input, when it’s not available all at once?

[TyOverby]

My example was highly simplified. I would expect that you could have a PullParser struct that kept parse-state and allowed more source to be added over time. This PullParser would either be an iterator, or have an iter() method.

[Ygg01]

@TyOverby So, would this require a major rewrite of tokenizer and tree builder's rules macros? Do current macros make sense under PullParser approach?

[TyOverby]

@Ygg01: I doubt it, but since I don't see anyone vehemently opposed, I'll prototype this and see how it turns out.

[nox]

@TyOverby Don't forget that tree building alters tokenisation, though.

[Ygg01]

@TyOverby I don't have a vote, but one really great thing about html5ever is that macros follow the spec format closely, so comparing code and spec it's easy to see where there was a divergence.

[TyOverby]

@nox

In what way can the tree builder change tokenisation? set_plaintext_state?

[SimonSapin]

My opinion is that if you manage to add a new pull API without affecting the current push API, go for it. (It’d also be nice to avoid significantly rewriting the tokenizer or tree builder code, as @Ygg01 mentioned.) But I’m skeptical that it’s possible without corountines (that Rust doesn’t have) or threads (that would add synchronization overhead).

[TyOverby]

The tokeniser is already broken up into discrete steps with the run method calling a bunch of steps. My plan (and I haven't looked into this too deeply), is to write an iterators next method that calls into step. step will have to be slightly rewritten (for instance, there are quite a few cases that don't return any tokens), but I don't think the change will be huge.

[nox]

@TyOverby @SimonSapin We may want to discuss that again now that h5e doesn't own its input stream anymore.

[nox]

Closing.