Skip to content

Encoding

James Conway edited this page Nov 18, 2026 · 3 revisions

Pages arrive as bytes, and the bytes don't say what they are. parsec-html-encoding works it out the way a browser does, decodes the page and parses it. The decoding itself is encoding_rs, the implementation of the WHATWG Encoding Standard that Firefox ships, so you get the same text Firefox would show.

On this page: Add it · parse_bytes · Options · Parsed and Confidence · How the encoding is chosen · When a <meta> turns up late · sniff · prescan · locale_default · Decoding and encoding text yourself · Streams · Conformance · Speed · Next to Lexbor

Add it

[dependencies]
parsec-html = { git = "https://github.com/parsec-html/parsec" }
parsec-html-encoding = { git = "https://github.com/parsec-html/parsec" }

The package is parsec-html-encoding; the crate you use is parsec_encoding. It lives in crates/parsec-encoding, beside the parser core, and depends on parsec-html and encoding_rs (MIT or Apache-2.0, with BSD-3-Clause data). You need Rust 1.88 or newer.

parse_bytes

pub fn parse_bytes(bytes: &[u8], options: Options) -> Result<Parsed, ParseError>

Finds the encoding, decodes, and parses with parsec::parse_with. The errors are parse_with's: the decoded page is over MAX_INPUT, or the time budget ran out.

use parsec_encoding::{Confidence, Options, parse_bytes};

let page = b"<meta charset=shift_jis><title>\x93\xfa\x96\x7b</title>";
let parsed = parse_bytes(page, Options::default()).unwrap();
assert_eq!(parsed.encoding.name(), "Shift_JIS");
assert_eq!(parsed.confidence, Confidence::Certain);
assert_eq!(parsed.document.text(parsed.document.root()), "日本");

The confidence is Certain here, though the prescan only guessed: the parser met the same <meta> and agreed. How the encoding is chosen has the whole story.

Options

#[derive(Debug, Clone, Copy)]
pub struct Options {
    pub html: parsec::Options,
    pub transport: Option<&'static Encoding>,
    pub fallback: &'static Encoding,
}
  • html: the parse itself, as for parse_with: scripting, the time budget, the depth limit.
  • transport: the encoding the transport layer gave you, usually the charset in an HTTP Content-Type. Look the label up with Encoding::for_label, which follows the standard's "get an encoding": case doesn't matter, surrounding whitespace is ignored, and an unknown label gives None, which is what you want to pass on, since the standard ignores a charset it doesn't know.
  • fallback: the encoding when nothing names one. The default is windows-1252, what browsers use for English and most of Western Europe. locale_default gives the one for another language.

Options::default() is no transport encoding, a windows-1252 fallback and parsec::Options::default().

use parsec_encoding::{Encoding, Options, parse_bytes};

// From `Content-Type: text/html; charset=windows-1251`.
let options = Options {
    transport: Encoding::for_label(b"windows-1251"),
    ..Options::default()
};
let parsed = parse_bytes(b"<p>\xcf\xf0\xe8\xe2\xe5\xf2", options).unwrap();
assert_eq!(parsed.document.text(parsed.document.root()), "Привет");

Parsed and Confidence

pub struct Parsed {
    pub document: Document,
    pub encoding: &'static Encoding,
    pub confidence: Confidence,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Confidence { Tentative, Certain }

encoding is what the page was decoded with, what document.characterSet says in a browser. confidence is the HTML Standard's:

  • Certain: a byte order mark or the transport layer said so, or a <meta> the parser met agreed with the guess or replaced it.
  • Tentative: a guess, from the prescan or the fallback, that nothing in the page confirmed. A page with no declaration at all ends up here.

How the encoding is chosen

This is the HTML Standard's encoding sniffing algorithm, for a parser that already has every byte. The first step that gives an answer wins.

Step Example Confidence
A byte order mark: EF BB BF, FE FF or FF FE a UTF-8 BOM certain
Options::transport charset=euc-jp from HTTP certain
The prescan of the first 1,024 bytes <meta charset="gbk"> tentative
Options::fallback windows-1252 tentative

The BOM wins even over HTTP, as in every browser, and the BOM is removed from the text. The steps the standard has for browsers only (an encoding the user picked from a menu, the parent frame's encoding, what the page was last time) have no meaning here and aren't taken. Guessing from the text itself, which the standard allows and discourages, isn't done either: you'd get different answers from different libraries, and the html5lib suite's chardet/ sample is the only test of it.

When a <meta> turns up late

A guess can be wrong. The prescan reads 1,024 bytes, and a page can put its <meta charset> after a long comment or a big inline script. When the parser meets a <meta> in the tree builder's "in head" rules while the confidence is tentative, the standard says to change the encoding. A browser restarts the page in the new encoding. parse_bytes does the same with the bytes it already has:

  1. Parse in the guessed encoding.
  2. Find the first HTML <meta> the parser made that declares an encoding, with charset, or with http-equiv="Content-Type" and a content naming one. Elements are made in the order their tags are met, so this is the one the parser met first.
  3. If there's none, keep the parse, still tentative.
  4. If it's the encoding in use, keep the parse, now certain. UTF-16 counts as UTF-8 here, and x-user-defined as windows-1252, and a page decoded as UTF-16 never switches.
  5. Otherwise decode the bytes again in the declared encoding and parse once more, certain.

So a page is parsed twice at most, and only when it declared one encoding past the prescan's reach and was guessed as another. The time budget in Options::html covers both parses together. Browsers can sometimes switch decoders mid-stream instead, when everything so far reads the same in both encodings; a second parse gives the same tree without that bookkeeping.

use parsec_encoding::{Confidence, Options, parse_bytes};

let mut page = b"<!doctype html><!-- ".to_vec();
page.extend([b'x'; 1100]); // past the prescan
page.extend(b" --><meta charset=windows-1251><p>\xcf");
let parsed = parse_bytes(&page, Options::default()).unwrap();
assert_eq!(parsed.encoding.name(), "windows-1251");
assert_eq!(parsed.confidence, Confidence::Certain);
assert!(parsed.document.text(parsed.document.root()).ends_with('П'));

A <meta> that document.write would add needs a script engine, so it's never seen.

sniff

pub fn sniff(
    bytes: &[u8],
    transport: Option<&'static Encoding>,
    fallback: &'static Encoding,
) -> (&'static Encoding, Confidence)

The table above on its own, without parsing: useful when you want the encoding but not the tree, or want to decode yourself.

use parsec_encoding::{Confidence, sniff};

let (encoding, confidence) = sniff(b"<meta charset=euc-kr>", None, encoding_rs::WINDOWS_1252);
assert_eq!((encoding.name(), confidence), ("EUC-KR", Confidence::Tentative));

prescan

pub fn prescan(bytes: &[u8]) -> Option<&'static Encoding>

The standard's prescan a byte stream to determine its encoding, over exactly the bytes you pass (sniff passes the first 1,024). It follows the standard step for step, oddities included:

  • Comments are skipped, and <!--> counts as a whole comment.
  • Only <meta followed by whitespace or / counts: <metacharset=...> doesn't.
  • A content attribute counts only with http-equiv="content-type" on the same tag, in either order.
  • A charset naming no encoding still stops a content attribute from counting.
  • A repeated attribute is ignored, as in the tree builder.
  • Other tags' attributes are read properly, so <a title="<meta charset=x>"> declares nothing.
  • Running out of bytes in the middle of a tag gives up on the <meta> scan, and then a leading <?xml ... encoding="..."> decides, as browsers still allow for compatibility.
  • UTF-16 declared in ASCII means UTF-8, and x-user-defined means windows-1252. The one way the prescan returns UTF-16 is a <?x written in UTF-16 itself.

locale_default

pub fn locale_default(locale: &str) -> &'static Encoding

The fallback a browser set to locale uses, from the HTML Standard's table of suggested defaults: ja gives Shift_JIS, ru windows-1251, zh-CN GBK, zh-TW Big5, and everything not in the table windows-1252. The locale is a BCP 47 tag; case doesn't matter, and for Chinese the script subtag (Hans, Hant) decides before the region.

use parsec_encoding::{Options, locale_default};

let options = Options { fallback: locale_default("ja-JP"), ..Options::default() };
assert_eq!(options.fallback.name(), "Shift_JIS");

If you crawl a site whose undeclared pages you know are in some language, pass that language's default. Firefox goes further and guesses from the top-level domain and the text; the standard doesn't define either, so they aren't here.

Decoding and encoding text yourself

parsec_encoding re-exports encoding_rs and its Encoding, so every encoding in the standard is there, by label, decoding and encoding:

use parsec_encoding::Encoding;

let gbk = Encoding::for_label(b"gb2312").unwrap();
assert_eq!(gbk.name(), "GBK");
let (text, _, had_errors) = gbk.decode(b"\xc4\xe3\xba\xc3");
assert_eq!((&*text, had_errors), ("你好", false));
let (bytes, _, unmappable) = gbk.encode("你好");
assert_eq!((&*bytes, unmappable), (&b"\xc4\xe3\xba\xc3"[..], false));
  • decode is the standard's "decode": a BOM overrides the encoding you named, and bad bytes become U+FFFD. decode_without_bom_handling ignores BOMs; decode_without_bom_handling_and_without_replacement gives None on bad bytes instead.
  • encode is the standard's "encode" as HTML forms use it: a character the encoding can't hold becomes a numeric character reference like &#128512;, and UTF-16 and replacement encode as UTF-8.
  • The labels that name replacement (iso-2022-kr, hz-gb-2312 and the rest) decode a whole input to one U+FFFD. They exist to stop old encodings being used to smuggle markup past filters.

encoding_rs's own documentation covers the rest of its API.

Streams

encoding_rs decodes a stream in pieces: Encoding::new_decoder() gives a Decoder, and decode_to_string(chunk, &mut out, last) takes one chunk at a time, keeping a character split across chunks. parse_bytes itself needs the whole page, because Parsec's parser takes whole input. Feeding a decoder's output into the parser as it arrives comes with chunked input (#6).

Conformance

cargo test runs two suites from crates/parsec-encoding/tests, and CI publishes their counts with the others on Conformance.

Suite Passed Failed Skipped
html5lib encoding/: tests1.dat, tests2.dat, test-yahoo-jp.dat 82 0 1
WPT encoding labels 228 0 0
WPT single-byte decoders 168 0 0
WPT ISO-2022-JP decoder 38 0 0
WPT GB18030 and GBK decoders 357 0 0
WPT legacy multi-byte pages 62,352 0 0
Encoder round trip over the same pages 53,813 0 0

The html5lib cases run through parse_bytes with no transport encoding and a windows-1252 fallback, and pass when the encoding matches. Seven of them put their <meta> past byte 1,024, so the prescan alone gets 75 of 82 and the late-<meta> reparse gets the other seven. The one skipped case, in scripted/, writes its <meta> with document.write.

The WPT files are pinned in crates/parsec-encoding/tests/wpt with their commit and licence. They're JavaScript and HTML written for a browser, so the runner reads their data and does what the harness would:

  • Labels: every label in encodings.js, as written, uppercased, and wrapped in each of the five whitespace characters, names its encoding.
  • Single-byte decoders: bytes 0 to 254 through every single-byte label give the code points in the standard's index.
  • ISO-2022-JP, GB18030 and GBK: every byte sequence and expected string in their decoder tests, the GB18030 ones through both gb18030 and gbk, plus the four-byte ranges and ISO-2022-JP's two streaming cases.
  • Legacy multi-byte pages: the *_chars.html page for Shift_JIS, EUC-JP, ISO-2022-JP, EUC-KR and Big5 (and Big5's extra page) goes through parse_bytes, which has to find the encoding in the page's <meta>, and every <span> must hold its character.
  • The round trip isn't a WPT count: WPT tests encoders by writing characters into URLs and forms, which needs a browser. Instead each character on those pages must encode back to its bytes, leaving out what the standard's encoders never write (EUC-JP's JIS X 0212 and the Big5 extra page).

What isn't run needs a browser: XMLHttpRequest, form submission, TextDecoder's API and streams.

Speed

parsec-html-bench encoding times bytes to UTF-8 text, in one process, against Lexbor's encoding module built from the bench's Lexbor submodule. These numbers are from an Apple M2 Max on 18 November 2026, two runs, starting at a load average of 2.9. The two runs agree within 10%. The table shows the first run, in MB of input a second.

Input Size Parsec Lexbor to UTF-8 Lexbor to code points Parsec vs Lexbor to UTF-8
UTF-8, wikipedia-population 1.39 MB 58,889 395 1,905 149x
UTF-8, nytimes 1.57 MB 73,945 394 1,888 188x
UTF-8, dense text 1.00 MB 10,987 458 1,046 24x
Shift_JIS, wikipedia-population 1.40 MB 8,092 370* 1,555* 22x
Shift_JIS, nytimes 1.57 MB 9,811 377 1,632 26x
Shift_JIS, dense text 1.00 MB 719 298* 654* 2.4x
windows-1251, wikipedia-population 1.40 MB 9,783 240 492 41x
windows-1251, nytimes 1.57 MB 10,157 247 491 41x
windows-1251, dense text 1.00 MB 806 211 485 3.8x
GBK, wikipedia-population 1.40 MB 8,947 253 511 35x
GBK, nytimes 1.57 MB 10,043 253 511 40x
GBK, dense text 1.00 MB 501 309 704 1.6x

* Lexbor's text differs from the standard's here, because of its Shift_JIS lead-byte bug (Next to Lexbor).

How to read it:

  • The real pages are UTF-8 originals transcoded with encoding_rs. Characters an encoding can't hold become numeric references, so these pages are mostly ASCII markup, and they mostly measure each decoder's ASCII path. The dense pages are about a megabyte of the encoding's own script, the hard case.
  • Lexbor's decoders write 32-bit code points. A parser needs UTF-8, so "Lexbor to UTF-8" adds Lexbor's own UTF-8 encoder. "Lexbor to code points" is the decoders alone, for reference. On dense GBK the decoder alone is faster than Parsec, 704 MB/s to 501.
  • For UTF-8 input, Parsec only validates the bytes and parses them where they are, and Lexbor still writes every code point. That's a real difference in what you get, but it isn't the same work.
  • Each Lexbor decode gets one output buffer big enough for the whole input. That's its fastest case, and it also avoids the single-byte decoder bug below, which drops bytes when output comes in batches.

Next to Lexbor

Lexbor has its own implementation of the 40 encodings and the sniffing (lxb_html_encoding_determine). Two differences matter.

Correctness. Run over the same WPT pages, Lexbor, up to its latest release (3.0.1), decodes 5,240 of Shift_JIS's 7,393 characters as the standard does. Its decoder refuses lead bytes 0xE1 to 0xFB, so most of JIS level 2 kanji come out as U+FFFD followed by a stray ASCII character. Its single-byte decoders also drop a byte whenever one lands on a full output buffer. Both are fixed on Lexbor's master (lexbor#424, lexbor#425) and not yet released. EUC-JP, ISO-2022-JP, EUC-KR and Big5 match on every character.

What you get back. Lexbor's decoders write code points, which you then encode to UTF-8 yourself before parsing; parse_bytes does it all, and for a UTF-8 page it only checks the bytes and parses them where they are.

See also

Clone this wiki locally