Make tokenizer not own the input stream

[nox]

This change is

[SimonSapin]

Reviewed 2 of 2 files at r1, 6 of 6 files at r2, 1 of 1 files at r3, 9 of 9 files at r4, 1 of 1 files at r5.
Review status: all files reviewed at latest revision, 3 unresolved discussions.

---

Cargo.toml, line 27 at r4 (raw file):

[[test]]
name = "serializer"

Why disable this test?

---

src/tokenizer/buffer_queue.rs, line 121 at r4 (raw file):

    // If so, consume them and return Ok(true).
    // If they do not match, return Ok(false) and don't consume anything.
    // If a partial match is found, return Err(length).

It looks like lenght in the Err case is counted in UTF-8 bytes. Please document it so.

Nit: in the previous doc-comment "not enough characters are available to know" says this happens when the end of the BufferQueue is reached (and more input may be fed later). Here "partial match" sounds like it includes input that has a any common prefix with pat. Maybe rephrase this sentence?

---

src/tokenizer/mod.rs, line 320 at r4 (raw file):

            Err(length) => {
                for _ in 0..length {
                    self.temp_buf.push_char(input.next().unwrap());

This is using next() (which consumes a code point) and push_char, while length is counted in UTF-8 bytes. Is this a bug in non-ASCII cases?

---

Comments from Reviewable

[nox]

Review status: 13 of 15 files reviewed at latest revision, 3 unresolved discussions.

---

Cargo.toml, line 27 at r4 (raw file):

Previously, SimonSapin (Simon Sapin) wrote…

Why disable this test?

It's not part of my changes AFAIK. And it's not really disabled, I don't know what that harness thing is.

---

src/tokenizer/buffer_queue.rs, line 121 at r4 (raw file):

Previously, SimonSapin (Simon Sapin) wrote…

It looks like lenght in the Err case is counted in UTF-8 bytes. Please document it so.

Nit: in the previous doc-comment "not enough characters are available to know" says this happens when the end of the BufferQueue is reached (and more input may be fed later). Here "partial match" sounds like it includes input that has a any common prefix with pat. Maybe rephrase this sentence?

Done.

---

src/tokenizer/mod.rs, line 320 at r4 (raw file):

Previously, SimonSapin (Simon Sapin) wrote…

This is using next() (which consumes a code point) and push_char, while length is counted in UTF-8 bytes. Is this a bug in non-ASCII cases?

Nope, BufferQueue::eat panics in case of non-ASCII chars. I added a comment there.

---

Comments from Reviewable

[SimonSapin]

Reviewed 1 of 2 files at r6, 1 of 1 files at r7.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

---

src/tokenizer/buffer_queue.rs, line 121 at r4 (raw file):

Previously, nox (Anthony Ramine) wrote…

Done.

I meant that input = "foobar" is a partial match for pat = "foofoo", but not "not enough characters available to know".

---

Comments from Reviewable

[SimonSapin]

@bors-servo r+

---

Reviewed 2 of 3 files at r8, 1 of 1 files at r9.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

---

Comments from Reviewable

[bors-servo]

📌 Commit ceb1bd3 has been approved by SimonSapin

[bors-servo]

⌛ Testing commit ceb1bd3 with merge c56d8e5...

[bors-servo]

☀️ Test successful - status-travis