Skip to content

Encoding

James Conway edited this page Nov 19, 2026 · 3 revisions

Pages arrive as bytes, and the bytes don't say what they are. parsec-html-encoding works it out the way a browser does, decodes the page and parses it. The decoding itself is encoding_rs, the implementation of the WHATWG Encoding Standard that Firefox ships, so you get the same text Firefox would show.

On this page: Add it · parse_bytes · Options · Parsed and Confidence · How the encoding is chosen · When a <meta> turns up late · sniff · prescan · locale_default · Decoding and encoding text yourself · ByteParser: bytes as they arrive · Offsets: positions in the bytes · From Python, JavaScript and C · Conformance · Speed · Next to Lexbor

Add it

[dependencies]
parsec-html = { git = "https://github.com/parsec-html/parsec" }
parsec-html-encoding = { git = "https://github.com/parsec-html/parsec" }

The package is parsec-html-encoding; the crate you use is parsec_encoding. It lives in crates/parsec-encoding, beside the parser core, and depends on parsec-html and encoding_rs (MIT or Apache-2.0, with BSD-3-Clause data). You need Rust 1.88 or newer.

parse_bytes

pub fn parse_bytes(bytes: &[u8], options: Options) -> Result<Parsed, ParseError>

Finds the encoding, decodes, and parses with parsec::parse_with. The errors are parse_with's: the decoded page is over MAX_INPUT, or the time budget ran out.

use parsec_encoding::{Confidence, Options, parse_bytes};

let page = b"<meta charset=shift_jis><title>\x93\xfa\x96\x7b</title>";
let parsed = parse_bytes(page, Options::default()).unwrap();
assert_eq!(parsed.encoding.name(), "Shift_JIS");
assert_eq!(parsed.confidence, Confidence::Certain);
assert_eq!(parsed.document.text(parsed.document.root()), "日本");

The confidence is Certain here, though the prescan only guessed: the parser met the same <meta> and agreed. How the encoding is chosen has the whole story.

Options

#[derive(Debug, Clone, Copy)]
pub struct Options {
    pub html: parsec::Options,
    pub transport: Option<&'static Encoding>,
    pub fallback: &'static Encoding,
}
  • html: the parse itself, as for parse_with: scripting, the time budget, the depth limit.
  • transport: the encoding the transport layer gave you, usually the charset in an HTTP Content-Type. Look the label up with Encoding::for_label, which follows the standard's "get an encoding": case doesn't matter, surrounding whitespace is ignored, and an unknown label gives None, which is what you want to pass on, since the standard ignores a charset it doesn't know.
  • fallback: the encoding when nothing names one and the bytes aren't valid UTF-8. The default is windows-1252, what browsers use for English and most of Western Europe. locale_default gives the one for another language.

Options::default() is no transport encoding, a windows-1252 fallback and parsec::Options::default().

use parsec_encoding::{Encoding, Options, parse_bytes};

// From `Content-Type: text/html; charset=windows-1251`.
let options = Options {
    transport: Encoding::for_label(b"windows-1251"),
    ..Options::default()
};
let parsed = parse_bytes(b"<p>\xcf\xf0\xe8\xe2\xe5\xf2", options).unwrap();
assert_eq!(parsed.document.text(parsed.document.root()), "Привет");

Parsed and Confidence

pub struct Parsed {
    pub document: Document,
    pub encoding: &'static Encoding,
    pub confidence: Confidence,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Confidence { Tentative, Certain }

encoding is what the page was decoded with, what document.characterSet says in a browser. confidence is the HTML Standard's:

  • Certain: a byte order mark or the transport layer said so, or a <meta> the parser met agreed with the guess or replaced it.
  • Tentative: a guess, from the prescan or the fallback, that nothing in the page confirmed. A page with no declaration at all ends up here.

How the encoding is chosen

This is the HTML Standard's encoding sniffing algorithm, for a parser that already has every byte. The first step that gives an answer wins.

Step Example Confidence
A byte order mark: EF BB BF, FE FF or FF FE a UTF-8 BOM certain
Options::transport charset=euc-jp from HTTP certain
The prescan of the first 1,024 bytes <meta charset="gbk"> tentative
Autodetection: bytes with something beyond ASCII that are all valid UTF-8 caf\xc3\xa9 tentative, as UTF-8
Options::fallback windows-1252 tentative

The BOM wins even over HTTP, as in every browser, and the BOM is removed from the text. The steps the standard has for browsers only (an encoding the user picked from a menu, the parent frame's encoding, what the page was last time) have no meaning here and aren't taken. The standard lets a browser guess from the bytes before it falls back ("the user agent may attempt to autodetect the character encoding from applying frequency analysis or other algorithms to the data stream"), and browsers do: Firefox's chardetng and Chrome's CED both recognise UTF-8. Parsec takes that one narrow, deterministic guess and no other. If the bytes have anything beyond ASCII and every one of them is valid UTF-8, the page is UTF-8, tentatively, so a <meta> the parser meets later can still change it. Bytes that aren't valid UTF-8 fall back, and so does pure ASCII, which says nothing either way. Legacy text is almost never valid UTF-8 by accident: windows-1252's accented letters, Shift_JIS's and GBK's double bytes all break UTF-8's rules within a few characters. Frequency analysis to tell Shift_JIS from EUC-KR isn't done: you'd get different answers from different libraries, and the html5lib suite's chardet/ sample is the only test of it.

When a <meta> turns up late

A guess can be wrong. The prescan reads 1,024 bytes, and a page can put its <meta charset> after a long comment or a big inline script. When the parser meets a <meta> in the tree builder's "in head" rules while the confidence is tentative, the standard says to change the encoding. A browser restarts the page in the new encoding. parse_bytes does the same with the bytes it already has:

  1. Parse in the guessed encoding.
  2. Find the first HTML <meta> the parser made that declares an encoding, with charset, or with http-equiv="Content-Type" and a content naming one. Elements are made in the order their tags are met, so this is the one the parser met first.
  3. If there's none, keep the parse, still tentative.
  4. If it's the encoding in use, keep the parse, now certain. UTF-16 counts as UTF-8 here, and x-user-defined as windows-1252, and a page decoded as UTF-16 never switches.
  5. Otherwise decode the bytes again in the declared encoding and parse once more, certain.

So a page is parsed twice at most, and only when it declared one encoding past the prescan's reach and was guessed as another. The time budget in Options::html covers both parses together. Browsers can sometimes switch decoders mid-stream instead, when everything so far reads the same in both encodings; a second parse gives the same tree without that bookkeeping.

use parsec_encoding::{Confidence, Options, parse_bytes};

let mut page = b"<!doctype html><!-- ".to_vec();
page.extend([b'x'; 1100]); // past the prescan
page.extend(b" --><meta charset=windows-1251><p>\xcf");
let parsed = parse_bytes(&page, Options::default()).unwrap();
assert_eq!(parsed.encoding.name(), "windows-1251");
assert_eq!(parsed.confidence, Confidence::Certain);
assert!(parsed.document.text(parsed.document.root()).ends_with('П'));

A <meta> that document.write would add needs a script engine, so it's never seen.

sniff

pub fn sniff(
    bytes: &[u8],
    transport: Option<&'static Encoding>,
    fallback: &'static Encoding,
) -> (&'static Encoding, Confidence)

The table above on its own, without parsing: useful when you want the encoding but not the tree, or want to decode yourself.

use parsec_encoding::{Confidence, sniff};

let (encoding, confidence) = sniff(b"<meta charset=euc-kr>", None, encoding_rs::WINDOWS_1252);
assert_eq!((encoding.name(), confidence), ("EUC-KR", Confidence::Tentative));

prescan

pub fn prescan(bytes: &[u8]) -> Option<&'static Encoding>

The standard's prescan a byte stream to determine its encoding, over exactly the bytes you pass (sniff passes the first 1,024). It follows the standard step for step, oddities included:

  • Comments are skipped, and <!--> counts as a whole comment.
  • Only <meta followed by whitespace or / counts: <metacharset=...> doesn't.
  • A content attribute counts only with http-equiv="content-type" on the same tag, in either order.
  • A charset naming no encoding still stops a content attribute from counting.
  • A repeated attribute is ignored, as in the tree builder.
  • Other tags' attributes are read properly, so <a title="<meta charset=x>"> declares nothing.
  • Running out of bytes in the middle of a tag gives up on the <meta> scan, and then a leading <?xml ... encoding="..."> decides, as browsers still allow for compatibility.
  • UTF-16 declared in ASCII means UTF-8, and x-user-defined means windows-1252. The one way the prescan returns UTF-16 is a <?x written in UTF-16 itself.

locale_default

pub fn locale_default(locale: &str) -> &'static Encoding

The fallback a browser set to locale uses, from the HTML Standard's table of suggested defaults: ja gives Shift_JIS, ru windows-1251, zh-CN GBK, zh-TW Big5, and everything not in the table windows-1252. The locale is a BCP 47 tag; case doesn't matter, and for Chinese the script subtag (Hans, Hant) decides before the region.

use parsec_encoding::{Options, locale_default};

let options = Options { fallback: locale_default("ja-JP"), ..Options::default() };
assert_eq!(options.fallback.name(), "Shift_JIS");

If you crawl a site whose undeclared pages you know are in some language, pass that language's default. Firefox goes further and guesses from the top-level domain and the text; the standard doesn't define either, so they aren't here.

Decoding and encoding text yourself

parsec_encoding re-exports encoding_rs and its Encoding, so every encoding in the standard is there, by label, decoding and encoding:

use parsec_encoding::Encoding;

let gbk = Encoding::for_label(b"gb2312").unwrap();
assert_eq!(gbk.name(), "GBK");
let (text, _, had_errors) = gbk.decode(b"\xc4\xe3\xba\xc3");
assert_eq!((&*text, had_errors), ("你好", false));
let (bytes, _, unmappable) = gbk.encode("你好");
assert_eq!((&*bytes, unmappable), (&b"\xc4\xe3\xba\xc3"[..], false));
  • decode is the standard's "decode": a BOM overrides the encoding you named, and bad bytes become U+FFFD. decode_without_bom_handling ignores BOMs; decode_without_bom_handling_and_without_replacement gives None on bad bytes instead.
  • encode is the standard's "encode" as HTML forms use it: a character the encoding can't hold becomes a numeric character reference like &#128512;, and UTF-16 and replacement encode as UTF-8.
  • The labels that name replacement (iso-2022-kr, hz-gb-2312 and the rest) decode a whole input to one U+FFFD. They exist to stop old encodings being used to smuggle markup past filters.

encoding_rs's own documentation covers the rest of its API.

ByteParser: bytes as they arrive

pub struct ByteParser { /* private */ }

impl ByteParser {
    pub fn new(options: Options) -> ByteParser;
    pub fn feed(&mut self, chunk: &[u8]);
    pub fn finish(self) -> Result<Parsed, ParseError>;
}

A page off the network comes in pieces, and you'd rather not wait for the last one before you start. ByteParser takes the pieces as they come, any size, cut anywhere, even in the middle of a character or a byte order mark. What finish hands back is exactly what parse_bytes gives for all the pieces joined: the same tree, the same encoding, the same confidence.

use parsec_encoding::{ByteParser, Confidence, Options};

let mut parser = ByteParser::new(Options::default());
// Shift_JIS for 日本, cut inside the first character.
for piece in [&b"<meta charset=sjis><p>\x93"[..], b"\xfa\x96\x7b"] {
    parser.feed(piece);
}
let parsed = parser.finish().unwrap();
assert_eq!(parsed.encoding.name(), "Shift_JIS");
assert_eq!(parsed.confidence, Confidence::Certain);
assert_eq!(parsed.document.text(parsed.document.root()), "日本");

Here's what happens to the bytes:

  • Until the encoding is known, they wait. The HTML Standard lets a browser hold back up to 1,024 bytes for the prescan, and ByteParser holds back only as long as it must. A byte order mark decides the moment its last byte comes, so EF then BB then BF in three pieces still counts. With no BOM, Options::transport decides as soon as a BOM is ruled out, usually on the first byte. Otherwise the prescan decides once it has read a whole <meta> naming an encoding, or at 1,024 bytes, or at the end of the input, whichever comes first. A <meta charset=iso-8859-1 cut before the 5> doesn't fool it.
  • Then each piece is decoded and parsed as it arrives. encoding_rs's Decoder turns it into text, keeping a character cut between pieces for the next one, and the text goes straight into parsec::Parser, the chunked parser.
  • A late <meta> still changes the encoding. If the encoding was a guess and the parser meets a <meta> naming another, finish decodes and parses once more, exactly as parse_bytes does.

With no <meta> in the first 1,024 bytes, the autodetection looks at the bytes held back, and a UTF-8 character cut off at their end still counts as valid, since the rest of it is on its way. parse_bytes looks at the whole input. They can disagree: UTF-8 that starts after 1,024 bytes of ASCII, or a byte past the first 1,024 that isn't UTF-8. Waiting longer would only move the line, so ByteParser doesn't. While the encoding is a guess it has every byte anyway (see below), so finish runs the same check over all of them and, when the answer differs, parses as parse_bytes would. That's one more parse, as rarely as a late <meta> costs one, and the streamed result stays the same as the whole one for every cut.

That last rule has a price. While the encoding is a guess, ByteParser keeps every byte it's fed, in case it needs them again, so the page sits in memory once more as bytes beside the document, which holds the decoded text anyway. When a BOM or the transport encoding decides, nothing is kept beyond the decoder's few bytes of state. So if you have the HTTP charset, pass it.

A time budget in Options::html counts only the time spent inside feed and finish, both parses together, not the time you spend waiting on the network.

If you want to decode a stream yourself without parsing it, encoding_rs's Decoder is there too: Encoding::new_decoder(), then decode_to_string(chunk, &mut out, last) a piece at a time.

Offsets: positions in the bytes

With spans or errors on, the parser's offsets count UTF-8 bytes into the decoded page. If you have the bytes the page came as, you usually want offsets into those. Offsets turns one into the other:

use parsec_encoding::{Offsets, Options, parse_bytes};

let page = b"<meta charset=shift_jis><p>\x93\xfa\x96\x7b<b>!</b>";
let options = Options {
    html: parsec::Options { spans: true, ..Default::default() },
    ..Options::default()
};
let parsed = parse_bytes(page, options).unwrap();
let offsets = Offsets::new(page, parsed.encoding);
let doc = &parsed.document;
let b = (0..doc.len() as u32).find(|&n| doc.lower_name(n) == Some("b")).unwrap();
let span = doc.span(b).unwrap();          // 33..36 in the decoded text
let (start, end) = (offsets.to_source(span.start), offsets.to_source(span.end));
assert_eq!(&page[start..end], b"<b>");    // 31..34 in the bytes

Offsets::new(bytes, encoding) decodes the bytes again as parse_bytes and ByteParser decoded them, noting where each character's bytes began, and keeps only the places where the two stop moving in step: after a byte order mark, at each character whose length differs (a Shift_JIS kanji is 2 bytes and 3 in UTF-8), and around each run the decoder replaced with U+FFFD. to_source(decoded) and to_decoded(source) look up the nearest of those and count on. ASCII and clean UTF-8 cost nothing (is_identity() says so); text that's all non-ASCII costs 8 bytes a character. For ISO-2022-JP, a character just after an escape sequence starts at the escape.

UTF-8 and the single-byte encodings take one pass. The other multi-byte encodings go through encoding_rs's decoder a byte at a time, since that's the only way it says which bytes made which character: on a 773 KB Shift_JIS page that took 10.8 ms against 4.5 ms for the parse, measured twice on an Apple M2 Max at a load of 3. It's built only when you ask for positions.

crates/parsec-encoding/tests/offsets.rs checks it on every html5lib encoding/ input (as written, behind a UTF-8 byte order mark and widened to UTF-16 both ways) and the WPT legacy multi-byte pages: each of 187,952 element and attribute spans on 334 pages, mapped back, decodes to what the span covers in the decoded page, and maps forward again to where it began.

From Python, JavaScript and C

The bindings take bytes too, with this crate underneath, and give positions as offsets into those bytes:

Whole As it arrives Encoding
Python HTMLParser(data, transport_encoding=..., default_encoding=...) StreamParser(...), feed(data), close() parser.encoding, parser.encoding_confidence; encoding_for_locale(tag) is locale_default
JavaScript parseBytes(bytes, { transportEncoding, defaultEncoding }) new ByteParser(options), feed(bytes), finish() doc.characterSet, doc.encodingConfidence
C parsec_parse_bytes(bytes, len, transport, transport_len, flags, &doc, &encoding) parsec_byte_parser_new, _feed, _finish, _free parsec_document_encoding(doc)

In all three a transport label that names no encoding is passed over, as the HTML Standard says, and each runs the 82 html5lib cases through both entry points, the streamed one in pieces of 1 to 7 bytes, with all 82 passing.

Conformance

cargo test runs two suites from crates/parsec-encoding/tests, and CI publishes their counts with the others on Conformance.

Suite Passed Failed Skipped
html5lib encoding/: tests1.dat, tests2.dat, test-yahoo-jp.dat 82 0 1
WPT encoding labels 228 0 0
WPT single-byte decoders 168 0 0
WPT ISO-2022-JP decoder 38 0 0
WPT GB18030 and GBK decoders 357 0 0
WPT legacy multi-byte pages 62,352 0 0
Encoder round trip over the same pages 53,813 0 0

The html5lib cases run through parse_bytes with no transport encoding and a windows-1252 fallback, and pass when the encoding matches. Seven of them put their <meta> past byte 1,024, so the prescan alone gets 75 of 82 and the late-<meta> reparse gets the other seven. The one skipped case, in scripted/, writes its <meta> with document.write.

The WPT files are pinned in crates/parsec-encoding/tests/wpt with their commit and licence. They're JavaScript and HTML written for a browser, so the runner reads their data and does what the harness would:

  • Labels: every label in encodings.js, as written, uppercased, and wrapped in each of the five whitespace characters, names its encoding.
  • Single-byte decoders: bytes 0 to 254 through every single-byte label give the code points in the standard's index.
  • ISO-2022-JP, GB18030 and GBK: every byte sequence and expected string in their decoder tests, the GB18030 ones through both gb18030 and gbk, plus the four-byte ranges and ISO-2022-JP's two streaming cases.
  • Legacy multi-byte pages: the *_chars.html page for Shift_JIS, EUC-JP, ISO-2022-JP, EUC-KR and Big5 (and Big5's extra page) goes through parse_bytes, which has to find the encoding in the page's <meta>, and every <span> must hold its character.
  • The round trip isn't a WPT count: WPT tests encoders by writing characters into URLs and forms, which needs a browser. Instead each character on those pages must encode back to its bytes, leaving out what the standard's encoders never write (EUC-JP's JIS X 0212 and the Big5 extra page).

What isn't run needs a browser: XMLHttpRequest, form submission, TextDecoder's API and streams.

ByteParser has its own tests in crates/parsec-encoding/tests/streaming.rs, which compare it with parse_bytes on the same bytes: the serialized tree, the encoding and the confidence.

Input Cuts Differ
Every html5lib encoding/ input, the scripted ones too, as written and behind UTF-8, UTF-16LE and UTF-16BE byte order marks: cut in two at every byte, and 100 random cuttings each into pieces of 1 to 300 bytes. The ones without a BOM run again with a transport encoding. 369,396 0
The six WPT legacy multi-byte pages: each one's first 8 KB as a page of its own cut at every byte, then the whole page cut at 40 evenly spaced places, in 4 KB pieces, and in random pieces of up to 1 KB, 4 KB and 64 KB 49,422 0

The WPT pages are sampled because cutting each whole page at every byte would be 3.3 million parses of up to 850 KB, hours in a debug build. The first 8 KB of each holds the <meta> and thousands of two- and three-byte characters, so every cut through the sniffing and through a multi-byte character is in there. The random cuttings use a fixed seed, so a failure can be run again.

Speed

parsec-html-bench encoding times bytes to UTF-8 text, in one process, against Lexbor's encoding module built from the bench's Lexbor submodule. These numbers are from an Apple M2 Max on 18 November 2026, two runs, starting at a load average of 2.9. The two runs agree within 10%. The table shows the first run, in MB of input a second.

Input Size Parsec Lexbor to UTF-8 Lexbor to code points Parsec vs Lexbor to UTF-8
UTF-8, wikipedia-population 1.39 MB 58,889 395 1,905 149x
UTF-8, nytimes 1.57 MB 73,945 394 1,888 188x
UTF-8, dense text 1.00 MB 10,987 458 1,046 24x
Shift_JIS, wikipedia-population 1.40 MB 8,092 370* 1,555* 22x
Shift_JIS, nytimes 1.57 MB 9,811 377 1,632 26x
Shift_JIS, dense text 1.00 MB 719 298* 654* 2.4x
windows-1251, wikipedia-population 1.40 MB 9,783 240 492 41x
windows-1251, nytimes 1.57 MB 10,157 247 491 41x
windows-1251, dense text 1.00 MB 806 211 485 3.8x
GBK, wikipedia-population 1.40 MB 8,947 253 511 35x
GBK, nytimes 1.57 MB 10,043 253 511 40x
GBK, dense text 1.00 MB 501 309 704 1.6x

* Lexbor's text differs from the standard's here, because of its Shift_JIS lead-byte bug (Next to Lexbor).

How to read it:

  • The real pages are UTF-8 originals transcoded with encoding_rs. Characters an encoding can't hold become numeric references, so these pages are mostly ASCII markup, and they mostly measure each decoder's ASCII path. The dense pages are about a megabyte of the encoding's own script, the hard case.
  • Lexbor's decoders write 32-bit code points. A parser needs UTF-8, so "Lexbor to UTF-8" adds Lexbor's own UTF-8 encoder. "Lexbor to code points" is the decoders alone, for reference. On dense GBK the decoder alone is faster than Parsec, 704 MB/s to 501.
  • For UTF-8 input, Parsec only validates the bytes and parses them where they are, and Lexbor still writes every code point. That's a real difference in what you get, but it isn't the same work.
  • Each Lexbor decode gets one output buffer big enough for the whole input. That's its fastest case, and it also avoids the single-byte decoder bug below, which drops bytes when output comes in batches.

Streaming

The same run then parses those transcoded pages into documents, parse_bytes on the whole page against ByteParser fed 4 KB at a time, in MB of input a second. "Encoding given" passes the page's encoding as Options::transport, as an HTTP header would. "Sniffed" passes nothing. These are from the same machine on 19 November 2026, two runs at load averages of 2.2 and 2.5, which agree within 5%; the table shows the first.

Input Size Given, whole Given, 4 KB pieces Sniffed, whole Sniffed, 4 KB pieces
UTF-8, wikipedia-population 1.39 MB 600 549 595 531
UTF-8, nytimes 1.57 MB 1,573 1,441 1,605 1,336
Shift_JIS, wikipedia-population 1.40 MB 533 503 546 490
Shift_JIS, nytimes 1.57 MB 1,387 1,302 1,503 1,296
windows-1251, wikipedia-population 1.40 MB 552 503 558 495
windows-1251, nytimes 1.57 MB 1,381 1,311 1,535 1,284
GBK, wikipedia-population 1.40 MB 552 520 562 486
GBK, nytimes 1.57 MB 1,402 1,303 1,493 1,281

With the encoding given, 4 KB pieces run at 91 to 95% of the whole-page speed over both runs. Most of the gap is copying: parse_bytes parses a UTF-8 page where it lies, and the decoder writes each piece out before the parser reads it. Sniffed, ByteParser also keeps every byte for a possible second parse, which brings it to 83 to 91%. One caveat on the sniffed columns: both pages say <meta charset="utf-8">, which the transcoding didn't change, so the sniffed non-UTF-8 rows are decoded as UTF-8 and full of U+FFFD. That's the same work in both columns, so the comparison between them still holds.

Next to Lexbor

Lexbor has its own implementation of the 40 encodings and the sniffing (lxb_html_encoding_determine). Two differences matter.

Correctness. Run over the same WPT pages, Lexbor, up to its latest release (3.0.1), decodes 5,240 of Shift_JIS's 7,393 characters as the standard does. Its decoder refuses lead bytes 0xE1 to 0xFB, so most of JIS level 2 kanji come out as U+FFFD followed by a stray ASCII character. Its single-byte decoders also drop a byte whenever one lands on a full output buffer. Both are fixed on Lexbor's master (lexbor#424, lexbor#425) and not yet released. EUC-JP, ISO-2022-JP, EUC-KR and Big5 match on every character.

What you get back. Lexbor's decoders write code points, which you then encode to UTF-8 yourself before parsing; parse_bytes does it all, and for a UTF-8 page it only checks the bytes and parses them where they are.

See also

Clone this wiki locally