You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Every public type and member is documented. The WebLookup.xml file in the package carried no public-API documentation (only generated regex members), so an IDE showed nothing for SiteExplorer, WebSearchClient, ISearchProvider or RobotsInfo. All 24 public types and their members now have summaries (parameters, return values and the exceptions a call throws), and a public member added without /// fails the build.
RobotsInfo.UserAgents: the user agents the file names, including those whose group holds no rule, so IsAllowed can tell "this agent has a group with no rule" from "this agent is not named".
Fixed
A robots.txt rule ending in $ after a wildcard matches the way RFC 9309 says.RobotsInfo.IsAllowed took the first place the last piece of a pattern occurs, so Disallow: /*.php$ did not match /a.php.php, /*a*b$ did not match /xabab, and a pattern that ends in *$ (/x*$, /*$) matched nothing - the URLs were reported as allowed although the file disallows them. The last piece is now matched at the end of the path, and a trailing * before $ covers the rest. Breaking by observation: a path that was reported allowed can now be disallowed (the file always said so). Migration: none.
Breaking by observation: an agent whose group holds no rule is no longer sent to the * group.User-agent: Bot with only an empty Disallow: is a real group that allows everything (RFC 9309 section 2.2.1); IsAllowed read it as "no group" and applied the * rules, so a path a site had cleared for that agent came back disallowed. The same file now answers allowed for that agent.
Breaking by observation: robots.txt patterns and paths are compared percent-encoding normalised (RFC 9309 section 2.2.2). /foo/%62%61%7A and /foo/baz are the same path, a non-ASCII pattern matches its escaped form, and an escaped reserved character (%2F, %2A) stays an escape. The most specific rule is picked by the octet length of the normalised pattern, not the character count. A site that wrote such patterns now blocks (or allows) the paths it meant to.
A Sitemap: line, or a record the parser does not know, between two User-agent lines no longer splits the group. The first agent used to be left with no rules at all; both now share the rules that follow. Only Allow, Disallow and Crawl-delay lines end the run of User-agent lines.
SiteExplorer identifies itself. The client it creates sent no User-Agent on robots.txt and sitemap requests; it now sends WebLookup/<version>. A client you pass to new SiteExplorer(httpClient) is used as it is, so that constructor is the way to send your own crawler's name (the README now says so).
Sitemap dates and addresses read the same on every machine. A lastmod without a UTC offset (2026-11-04) was read in the machine's time zone (2026-11-04T00:00:00+09:00 on a Korean machine) and is now read as UTC; an explicit offset is kept. <loc> is trimmed, and an entry whose <loc> is empty, relative or not http/https is dropped instead of being returned with an unusable Url. Breaking by observation: a caller that relied on the machine's time zone for LastModified, or on an empty or relative Url appearing among the entries, sees different values. Migration: none for sitemaps that follow the protocol.
Arguments no provider can use are rejected before a provider is called.WebSearchClient.SearchAsync and every provider's SearchAsync throw ArgumentException for a null, empty or white-space query and ArgumentOutOfRangeException for a count below 1 (they used to send the request); RobotsInfo.IsAllowed and AddWebLookup throw ArgumentNullException for null (AddWebLookup(null) was a NullReferenceException). IsAllowed("private") read the path as written and allowed it against Disallow: /private; a path is now read as the URL's path, so a missing leading / is added. Breaking by observation for the calls above that used to go through.
A blocked DuckDuckGo search is an error, not "no results". When DuckDuckGo takes a client for a bot it answers with a challenge page (HTTP 202, or a page with its "anomaly" dialog); DuckDuckGoSearchProvider read that as a search with no hits and returned an empty list, where a 403, 429 or 500 throws. It now throws HttpRequestException (with the status) for the challenge page; a normal results page with no hits still returns an empty list. Result titles, snippets and links now have their HTML entities decoded (Tom & Jerry was returned as written).
WebSearchClient throws when every provider fails. It returned an empty list when its only provider was refused, so a caller that shows "no results" told its user the web has nothing on the query. With several providers, results from those that answered are still returned and the failures are still not reported; if none answers it throws the provider's exception (one provider) or an AggregateException (several). A provider's own client timeout no longer aborts the whole search as if the caller had cancelled; only the caller's CancellationToken does. Breaking by observation: code that treated an empty list from a failing sole provider as "no results" now sees an exception. Migration: catch it where an empty list was handled, or keep a second provider.
A sitemap file that times out is skipped like any other that cannot be fetched. With a client timeout (HttpClient.Timeout) shorter than a child sitemap's response, GetSitemapAsync threw TaskCanceledException and lost the entries of the siblings that had been read, although the README says such a file yields no entries; StreamSitemapAsync delivered the earlier entries and then threw. Now only the caller's own cancellation (CancellationToken) propagates as OperationCanceledException; a timeout skips that file, keeps the rest, and the call returns normally (an empty result can still mean the fetch failed, as documented). Breaking by observation: code that caught the timeout exception around a sitemap call no longer sees one. Migration: compare the entry count with what the site should have, or give the client a timeout long enough for its slowest sitemap.
A sitemap index is followed once per distinct file.GetSitemapAsync / StreamSitemapAsync only had a depth limit, so an index that listed itself twice cost 2,047 requests (2^11 - 1) and returned nothing, two indexes listing each other were fetched five or six times each, and a child listed twice yielded its entries twice. A sitemap file is now requested and read once per call (the address compared without its fragment, with the scheme and host in lower case), so the number of requests is bounded by the number of distinct files the input names, not by its shape.
A call stops at 50,000 distinct sitemap files (the sitemaps.org limit for one index) by throwing InvalidDataException, rather than carrying on requesting whatever a broken or hostile index names. Breaking by observation only for an index that names more than the protocol allows: it used to be fetched without bound, and now the call throws (entries streamed before that point were already delivered). Migration: none for an index within the protocol.
The package links to the repository as its project site (PackageProjectUrl) — its nuget.org page had no project link.
Changed
The packages carry the MIT LICENSE text (LICENSE at the package root) — the license expression alone gave no notice to ship with an app.
The nuget.org «Release Notes» of every package link to the GitHub release of its version, whose notes are this CHANGELOG's section for that version.