Repository navigation
Fixed
-
Breaking by observation: a robots.txt group that names no rule no longer sends its agent to the
*group.User-agent: MyBotwith only an emptyDisallow:is a real group that allows everything (RFC 9309 section 2.2.1), but the crawler read it as "no group" and applied the*rules, so a site that had cleared the crawler by name found it blocked.RobotsTxtInfo.Rulesnow has an entry, with no paths, for every product token a group names. -
Breaking by observation: robots.txt patterns and paths are compared percent-encoding normalised (RFC 9309 section 2.2.2). The pattern
/foo/%62%61%7Anow matches/foo/baz, a non-ASCII pattern (/é) matches the escaped path the crawler requests, and an escaped reserved character (%2F,%2A) stays an escape instead of acting as a slash or a wildcard. The most specific rule is picked by the octet length of the normalised pattern. A site that wrote such patterns now blocks (or allows) the paths it meant to. -
Breaking by observation: HTML extraction no longer removes a page's own content. The cleaner dropped a
header,footerorasideeven when it sat inside thearticle(the article's title block and its "Warning:" callouts vanished), and itsid/classpatterns matched anywhere in a name:id="cookie-faq"on a<main>,<body class="modal-open">, a wrapperclass="modal-layout",id="ad-hoc-queries",class="modality-table"andclass="menu"inside an article each removed the content (the first three left the page empty). A link-only list inside anarticle/main(an API index, a list of references) was removed by the link-density filter, and amainmade mostly of links was removed whole. Nowheader/footerinsidearticleormainandasideinsidearticleare kept (page-level ones are still removed), ambiguous names (cookie,modal,popup,overlay,ad-slot/ad-<number>,menu,widget) are matched as whole words and only outsidearticle/main, never on an element that holds most of the page's text, and no selector removesbody,htmlor a container ofmain; the density filter leaves link lists insidearticle/mainalone unless their class or id says navigation. Stored text of such pages changes: re-crawl and re-index them. -
Breaking by observation: HTML extraction stops writing junk into
TextandImageUrls. A lazy-loaded image (data-src,data-original,data-lazy-src…, or adata:placeholder insrc) is read from its real address instead of becoming![alt](); an image with no usable address at all becomes its alt text;ImageUrlsis read per image (an empty()or a markdown title no longer swallows the neighbouring image, anddata:URIs are not listed).<a>without a usablehref(none, empty,#,javascript:) is unwrapped to its text instead of[text](). Elements hidden byhidden,display:noneorvisibility:hiddenare no longer extracted, andselect/datalistoptions no longer fuse into the text (SmallMediumLarge). Titles collapse whitespace runs; zero-width space, soft hyphen, BOM and word joiner are dropped from the text (ZWNJ and ZWJ stay); markdown line endings are\non every OS (they followedEnvironment.NewLine, so Windows stored\r\nand the blank-line collapse never matched there). Stored text of affected pages changes: re-crawl and re-index. -
BasicContentExtractorkeeps the structure of what it reads. Its HTML path (also the pipeline's fallback when the Markdown conversion throws) fused every block into one run (<p>Line one<br>Line two</p><p>Next</p>becameLine oneLine twoNext,<td>k</td><td>v</td>becamekv) and included<script>/<style>text; it now writes one line per block or<br>, cells on one line, and no script or style text, and collapses whitespace in the title.ExtractFromMarkdownAsyncreports the front-mattertitle:(else the first#heading, else the first heading) asTitleinstead of the placeholderMarkdown Contentand fillsHeadings(fenced#lines and indented code are not headings); the text itself is unchanged.ExtractAutoAsyncrecognises a content type with parameters (text/html; charset=utf-8) andtext/x-markdown; before, such a value was read as plain text. A Markdown page with no heading keeps the placeholder title. -
MetadataExtractorreads what ordinary pages carry and no longer throws on what they contain.Basic.Languagenow comes from<html lang>(thenxml:lang, then<meta name="language">) - the old code looked for the attribute on the document root node, which never has it, so only the rare<meta>form worked. Title, description and keywords are HTML-entity decoded (Tom & JerrystaysTom & Jerry, it used to be stored escaped) and the title's whitespace collapses.rel="Canonical"is matched case-insensitively and a relative canonical URL is resolved against the page URL. Open Graph tags written withname=instead ofproperty=are read (thepropertyform wins when both exist). JSON-LD is read in all three shapes it is published in - a single object, an array of objects, and an@graph- with@typeas a string or an array; an array root or an array-valued@typeused to makeExtractMetadataAsyncthrowInvalidOperationExceptionfor the whole page, and an@graphyielded no typed entities. The first entity of each kind wins,MainEntityTypeprefers an article over a product over software over the rest, and one malformed entity no longer hides the others.Basic.Language,Basic.CanonicalUrl, the decoded strings andSchemaOrg.*change for such pages: re-extract rather than diff. -
Breaking by observation: the crawler discovers links with an HTML parser instead of a regular expression.
BaseCrawler.ExtractLinks(what every crawl strategy follows) now findshref=/aboutwritten without quotes (minified HTML), decodes&in query strings (?a=1&b=2was followed literally and fetched a different URL), resolves relative links against<base href>, and no longer takes a link from an HTML comment, a script string, escaped code in text,data-hreforxlink:href.<area href>stays a link; of the<link>elements onlyrel="next","prev"and"previous"are links (pagination) - stylesheets, icons, manifests, feeds, canonical and alternate-language links are no longer reported inCrawlResult.DiscoveredLinksand no longer count againstMaxPages/MaxDepth. Pages that used to be missed (unquoted links) are now crawled; a crawl may therefore return a different set of pages than before. -
Breaking by observation: a language nobody declared is no longer reported or applied as Korean.
ChunkingOptions.Languagenow defaults to empty, which lets FluxCurator detect the language of the text; it used to default to"ko"and the chunker took that as the caller's choice, so English pages were cut with the Korean sentence-boundary profile (an abbreviation such asD.C.orno.ended a sentence and a chunk).WebContent.Metadata.Languageof a crawled page is the language the page declares (<html lang>,xml:lang,<meta name="language">) ornull; it used to be"ko"for every page. A Markdown document without a front-matterlanguagereportsnullinstead of"ko". Chunk boundaries of non-Korean pages that were chunked with the default options, and any storedLanguageof crawled pages, change: re-chunk, re-index, or setChunkingOptions.Languageto keep the old value.
Removed
Polly.Extensions.Httpis no longer a dependency: nothing in WebFlux used it, and the package has been deprecated on nuget.org since 2019 (its replacement isMicrosoft.Extensions.Http.Resilience).Pollyitself stays - the resilience service uses it. Consumers that tookPolly.Extensions.Httptransitively through WebFlux and callHttpPolicyExtensionsmust reference it themselves.
Dependencies
-
Re-pinned third-party packages:
Markdig1.5.0 -> 1.5.1,Polly8.8.1 -> 8.9.0. -
Re-pinned third-party packages:
AngleSharp1.8.6 -> 1.9.0. -
Re-pinned third-party packages:
AngleSharp1.9.0 -> 1.9.1. -
Re-pinned third-party packages:
AngleSharp1.9.1 -> 1.10.0,Markdig1.5.1 -> 1.6.0. -
Re-pinned third-party packages:
Polly8.9.0 -> 8.9.1. -
Re-pinned third-party packages: 12 .NET servicing pins (Microsoft.Extensions) 10.0.13 -> 10.0.14 (the 10.0.14 security servicing release);
Microsoft.SourceLink.GitHub10.0.402 -> 10.0.403;AngleSharp1.8.5 -> 1.8.6;Markdig1.4.2 -> 1.5.0;Polly8.8.0 -> 8.8.1.