A non-ASCII fragment is double-encoded in a document with no charset

[xroche]

In a document that declares no charset, a reference like img src="s.svg#café" reaches the mirror with the fragment double-encoded: the UTF-8 bytes c3 a9 come out as c3 83 c2 a9, so the emitted fragment is #caf%C3%83%C2%A9 and the selector no longer matches.

This happens before the --single-file pass runs, and the mirrored control (a plain crawl, no --single-file) already contains the doubled bytes, so the encoding step is upstream of the inliner. Found while reviewing #804, whose fragment escaping faithfully encodes whatever it is handed and is not the cause.

Worth checking against the charset-detection path more generally, since the same double-encode would affect any non-ASCII text taken from a charset-less document, not just fragments.

[xroche]

I scoped this while fixing it, and the fragment symptom does not reproduce on master, so I am leaving it open rather than closing it on #848.

I crawled a charset-less document through 19 reference shapes: raw, percent-encoded and entity fragments on img, a and xlink:href; url() inline and in an external stylesheet; srcset; meta refresh; script strings; plus path segments, query strings, accented filenames and body text. Plain and under --single-file. Then I grepped the whole mirror for the doubled bytes c3 83 c2 a9 and %C3%83.

The generated top-level index was the only file carrying them, through the title lifted at htsparse.c:688, which converts using the page charset. That is a real bug, and it is what #848 guards.

Fragments came back clean in every shape. The link scanner stops at #, so a fragment never reaches the conversion, and the link path has carried the hts_isStringUTF8 guard since #180. I could not produce the reported #caf%C3%83%C2%A9 at all.

So either the original observation came from a tree carrying #804's branch rather than master, or there is a trigger these 19 shapes miss. If you still have the document that produced it, that would settle it. Otherwise this is probably closeable as not reproducible.