Skip to content

v0.29.0

Latest

Choose a tag to compare

@github-actions github-actions released this 08 Nov 15:41

Fixed

  • Breaking by observation: a robots.txt group that names no rule no longer sends its agent to the * group. User-agent: MyBot with only an empty Disallow: is a real group that allows everything (RFC 9309 section 2.2.1), but the crawler read it as "no group" and applied the * rules, so a site that had cleared the crawler by name found it blocked. RobotsTxtInfo.Rules now has an entry, with no paths, for every product token a group names.

  • Breaking by observation: robots.txt patterns and paths are compared percent-encoding normalised (RFC 9309 section 2.2.2). The pattern /foo/%62%61%7A now matches /foo/baz, a non-ASCII pattern (/é) matches the escaped path the crawler requests, and an escaped reserved character (%2F, %2A) stays an escape instead of acting as a slash or a wildcard. The most specific rule is picked by the octet length of the normalised pattern. A site that wrote such patterns now blocks (or allows) the paths it meant to.

  • Breaking by observation: HTML extraction no longer removes a page's own content. The cleaner dropped a header, footer or aside even when it sat inside the article (the article's title block and its "Warning:" callouts vanished), and its id/class patterns matched anywhere in a name: id="cookie-faq" on a <main>, <body class="modal-open">, a wrapper class="modal-layout", id="ad-hoc-queries", class="modality-table" and class="menu" inside an article each removed the content (the first three left the page empty). A link-only list inside an article/main (an API index, a list of references) was removed by the link-density filter, and a main made mostly of links was removed whole. Now header/footer inside article or main and aside inside article are kept (page-level ones are still removed), ambiguous names (cookie, modal, popup, overlay, ad-slot/ad-<number>, menu, widget) are matched as whole words and only outside article/main, never on an element that holds most of the page's text, and no selector removes body, html or a container of main; the density filter leaves link lists inside article/main alone unless their class or id says navigation. Stored text of such pages changes: re-crawl and re-index them.

  • Breaking by observation: HTML extraction stops writing junk into Text and ImageUrls. A lazy-loaded image (data-src, data-original, data-lazy-src …, or a data: placeholder in src) is read from its real address instead of becoming ![alt](); an image with no usable address at all becomes its alt text; ImageUrls is read per image (an empty () or a markdown title no longer swallows the neighbouring image, and data: URIs are not listed). <a> without a usable href (none, empty, #, javascript:) is unwrapped to its text instead of [text](). Elements hidden by hidden, display:none or visibility:hidden are no longer extracted, and select/datalist options no longer fuse into the text (SmallMediumLarge). Titles collapse whitespace runs; zero-width space, soft hyphen, BOM and word joiner are dropped from the text (ZWNJ and ZWJ stay); markdown line endings are \n on every OS (they followed Environment.NewLine, so Windows stored \r\n and the blank-line collapse never matched there). Stored text of affected pages changes: re-crawl and re-index.

  • BasicContentExtractor keeps the structure of what it reads. Its HTML path (also the pipeline's fallback when the Markdown conversion throws) fused every block into one run (<p>Line one<br>Line two</p><p>Next</p> became Line oneLine twoNext, <td>k</td><td>v</td> became kv) and included <script>/<style> text; it now writes one line per block or <br>, cells on one line, and no script or style text, and collapses whitespace in the title. ExtractFromMarkdownAsync reports the front-matter title: (else the first # heading, else the first heading) as Title instead of the placeholder Markdown Content and fills Headings (fenced # lines and indented code are not headings); the text itself is unchanged. ExtractAutoAsync recognises a content type with parameters (text/html; charset=utf-8) and text/x-markdown; before, such a value was read as plain text. A Markdown page with no heading keeps the placeholder title.

  • MetadataExtractor reads what ordinary pages carry and no longer throws on what they contain. Basic.Language now comes from <html lang> (then xml:lang, then <meta name="language">) - the old code looked for the attribute on the document root node, which never has it, so only the rare <meta> form worked. Title, description and keywords are HTML-entity decoded (Tom &amp; Jerry stays Tom & Jerry, it used to be stored escaped) and the title's whitespace collapses. rel="Canonical" is matched case-insensitively and a relative canonical URL is resolved against the page URL. Open Graph tags written with name= instead of property= are read (the property form wins when both exist). JSON-LD is read in all three shapes it is published in - a single object, an array of objects, and an @graph - with @type as a string or an array; an array root or an array-valued @type used to make ExtractMetadataAsync throw InvalidOperationException for the whole page, and an @graph yielded no typed entities. The first entity of each kind wins, MainEntityType prefers an article over a product over software over the rest, and one malformed entity no longer hides the others. Basic.Language, Basic.CanonicalUrl, the decoded strings and SchemaOrg.* change for such pages: re-extract rather than diff.

  • Breaking by observation: the crawler discovers links with an HTML parser instead of a regular expression. BaseCrawler.ExtractLinks (what every crawl strategy follows) now finds href=/about written without quotes (minified HTML), decodes &amp; in query strings (?a=1&amp;b=2 was followed literally and fetched a different URL), resolves relative links against <base href>, and no longer takes a link from an HTML comment, a script string, escaped code in text, data-href or xlink:href. <area href> stays a link; of the <link> elements only rel="next", "prev" and "previous" are links (pagination) - stylesheets, icons, manifests, feeds, canonical and alternate-language links are no longer reported in CrawlResult.DiscoveredLinks and no longer count against MaxPages/MaxDepth. Pages that used to be missed (unquoted links) are now crawled; a crawl may therefore return a different set of pages than before.

  • Breaking by observation: a language nobody declared is no longer reported or applied as Korean. ChunkingOptions.Language now defaults to empty, which lets FluxCurator detect the language of the text; it used to default to "ko" and the chunker took that as the caller's choice, so English pages were cut with the Korean sentence-boundary profile (an abbreviation such as D.C. or no. ended a sentence and a chunk). WebContent.Metadata.Language of a crawled page is the language the page declares (<html lang>, xml:lang, <meta name="language">) or null; it used to be "ko" for every page. A Markdown document without a front-matter language reports null instead of "ko". Chunk boundaries of non-Korean pages that were chunked with the default options, and any stored Language of crawled pages, change: re-chunk, re-index, or set ChunkingOptions.Language to keep the old value.

Removed

  • Polly.Extensions.Http is no longer a dependency: nothing in WebFlux used it, and the package has been deprecated on nuget.org since 2019 (its replacement is Microsoft.Extensions.Http.Resilience). Polly itself stays - the resilience service uses it. Consumers that took Polly.Extensions.Http transitively through WebFlux and call HttpPolicyExtensions must reference it themselves.

Dependencies

  • Re-pinned third-party packages: Markdig 1.5.0 -> 1.5.1, Polly 8.8.1 -> 8.9.0.

  • Re-pinned third-party packages: AngleSharp 1.8.6 -> 1.9.0.

  • Re-pinned third-party packages: AngleSharp 1.9.0 -> 1.9.1.

  • Re-pinned third-party packages: AngleSharp 1.9.1 -> 1.10.0, Markdig 1.5.1 -> 1.6.0.

  • Re-pinned third-party packages: Polly 8.9.0 -> 8.9.1.

  • Re-pinned third-party packages: 12 .NET servicing pins (Microsoft.Extensions) 10.0.13 -> 10.0.14 (the 10.0.14 security servicing release); Microsoft.SourceLink.GitHub 10.0.402 -> 10.0.403; AngleSharp 1.8.5 -> 1.8.6; Markdig 1.4.2 -> 1.5.0; Polly 8.8.0 -> 8.8.1.