--single-file maintains a second HTML parser instead of reusing htsparse - #1049
Merged
Conversation
…ing the mirror htsparse appends a "#!htsinline" fragment to every reference it saves that --single-file may inline, so the end-of-mirror pass becomes a substitution over those marks and parses no HTML and no CSS of its own. A fragment rather than a scheme, so the mirror still resolves if the pass never runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
The fixture fed unmarked HTML to a scanner that no longer exists. Marking it keeps 28 of the 32 assertions; the four that changed are quoting, which is now preserved rather than normalised, and a page with nothing to inline, which is mark-transparent rather than byte-transparent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
<img src="a.svg#frag"> saved as a.svg#!htsinline#frag, and the pass left the fragment glued to the data: URI. Consume it, and put it back when the reference falls through to a link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
…ts contract Three holes the prototype left. htsparse now disarms any mark already in a fetched document before it parses it, so a page shipping the literal string can no longer make the pass inline a mirrored file into its own prose. The pass strips the marks from non-HTML files once every page has been expanded, so a mirrored stylesheet ships no engine bookkeeping. And the no-delimiter contract the back-walk depends on is now checked where it is made and reported where it is consumed, instead of surfacing as a reference that quietly failed to inline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
The eight seeds carried no marks, so the harness had degraded to a scan that finds nothing: it reached 107 coverage points against 341 for the new seeds, and an off-by-one planted in the back-walk went undetected. The stylesheets the harness lays down were unmarked too, so its recursion was unreachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
The in-band marker was the common root of four injection and correctness defects, so replace it rather than patch the channels. The mark is now #!<16-hex>.<class>.<len>, where the hex is 64 CSPRNG bits drawn once per run and never written to disk: a site cannot spell one, so no sanitiser has to keep hostile bytes away from it and the channel a forgery arrives through stops mattering. <len> is the reference's byte length as emitted, which retires the backward scan and the escaping contract it rested on, and leaves an author's fragment untouched on the far side of the mark. <class> is the referencing context, checked against the resolved type so a stylesheet reference that lands on something else fails loudly instead of inlining it. -%M and --single-file are now refused together; they were silently accepted, with --single-file doing nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
Resolves three collisions the branch had with what landed since it was cut. #804 kept a fragment on an inlined reference by parsing it back out of the reference and re-escaping it. Under the mark design the mark covers the reference alone, so the fragment is document text the pass never touches and reaches the data: URI byte-identical; sf_fragment/sf_append_fragment go with the scanner they were written for. The query has to go the other way -- glued to base64 it corrupts the payload -- so htsparse now writes the mark after it and the marked span carries it. #1011 added a private hts_random_bytes to htsweb.c while this branch added one to htstools.c. htsserver links libhttrack but cannot see a hidden symbol inside it, so the two now share htsrandom.c: master's runtime-resolved RtlGenRandom on Windows, this branch's getrandom-then-/dev/urandom on POSIX. Master's test 94 then caught a bug the branch shipped: sf_rel_is_stylesheet() bounded its outer scan at the tag's '>' but handed strstrcase the raw buffer, so a <link rel="canonical"> on a page carrying any later rel="stylesheet" was classed as one and the pass inlined the page it named. rel is now matched as a token inside the value rech_endtoken() delimits. Only a crawl test sees this -- the engine self-test writes the marks it feeds the pass, so the class is an input there rather than something htsparse decided. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
An adversarial review of the mark rewrite found these; each has a crawl-level test, because the engine self-test writes the marks it feeds the pass and so cannot see a mis-classification at all. An unquoted attribute value ending at '>' clears intag, and htsparse was passing the resulting NULL tag to the classifier, where NULL means "a CSS or JS body". <a href=x.png> and <iframe src=x.png> therefore sailed past every tag-bearing deny rule and were inlined. The tag is now captured before the value scan can clear it, and a reference whose tag we cannot name is not marked; inscript_locked tells the one context that legitimately has no tag. Marks reached the delivered mirror three ways: past the @import nesting cap a stylesheet was base64'd verbatim, marks and all, publishing the run secret inside the payload; the sweep skipped HTML, so any page the pass declined kept its marks; and nothing could clear what an interrupted earlier run left, since its secret died with it. Stripping now matches the mark's shape rather than this run's secret -- expanding one needs the secret, recognising one to delete it does not -- and the sweep covers pages too. A page carrying mark-shaped text of its own loses it, which is the price of healing an abandoned run. A <link> with no rel is no longer inlined: nothing loads it. rel is read only as far as its own value, which an unquoted value ending at '>' used to overrun. The fuzz target did not compile (SINGLEFILE_MARK went away with the fixed- literal design) and could not have reached the mark parser if it had: the secret is drawn per httrackp and the harness built a fresh one per input, so no corpus byte could spell a mark. It now keeps one httrackp for the run and expands \001ref\002 into real marks, which takes corpus replay from 51 to 367 covered edges. Four corpus files were never registered for dist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Review pass: several blocks explained the mechanism or re-derived a fixed bug where one line of why does the job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
enabled auto-merge (squash)
August 7, 2026 16:44
…gone The new CSPRNG file was added to Makefile.am only, so both Visual Studio builds failed to link singlefile_intro. libhttrack compiles it directly; webhttrack needs it too, because hts_random_bytes is internal and the DLL does not export it -- the same reason the autotools build compiles it into htsserver. fuzz/Makefile.am still listed corpus/singlefile/many-attrs.html, deleted when the corpus was re-seeded onto marked inputs, which stopped make dist and with it the distcheck and deb lintian jobs, both of which build a tarball first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--single-fileshipped with its own HTML and CSS scanner inhtssinglefile.c, about 1140 lines doing whathtsparse.calready does. This replaces it. htsparse now appends an unguessable per-run mark to each reference it saves that the pass may inline, and the end-of-mirror pass becomes a substitution over those marks: no HTML and no CSS is parsed there at all. The scanner goes, and with it the private attribute table that had already drifted fromhts_detect[]twice.htssinglefile.cgoes from 1155 to 863 lines;htsparse.cgains 31.The mark is
#!<16-hex secret>.<class>.<len>. It is a fragment, so a mirror an interrupted run left marked still browses. The secret is CSPRNG bits drawn once per run and never written to disk, which is what makes the mark unforgeable: a site cannot spell one, so no sanitiser has to keep hostile bytes away from it.<len>is the length of the reference preceding the mark, so the pass never has to guess where a reference begins.<class>is the context htsparse saw, checked against the resolved type so a mismatch fails loudly instead of inlining whatever the walk found.Behaviour worth knowing about. A fragment comes back byte-identical rather than re-escaped, since the mark covers the reference alone and what follows it is document text nothing touches; the query moved inside the marked span, because glued to base64 it corrupts the payload. Expanding a mark needs the secret, but recognising one well enough to delete it does not, so the end-of-mirror sweep strips by shape. That clears what an interrupted earlier run abandoned, and costs a page any mark-shaped text of its own. A
<link>carrying norelis no longer inlined, which is what the old scanner did.htsserverlinks libhttrack but cannot see a hidden symbol inside it, so the CSPRNG that #1011 added privately tohtsweb.cand this branch added tohtstools.cnow sharehtsrandom.c.Closes #749