ServiceStack.AI.Chat 10.2.0 - Gemini web crawl only saves the start page: link scope is checked against each page's path, not the start URL

[colby02184]

The Gemini extension's web crawler (/ext/gemini/imports/crawl) only follows links that sit underneath the page they were found on. On documentation sites with flat .html pages, the crawl stops after the start page no matter which settings are used (Max pages, Max depth, Include paths, etc.).

I'm on ServiceStack.AI.Chat 10.2.0 and used these settings:
Admin → Gemini → Import → Web crawl, default settings (Max pages 500, Max depth 10, Include /**) Start URL: https://doc.sitecore.com/sai/en/developers/sitecoreai/sitecoreai.html

I was expecting the crawler to follow at least 24 pages under /sai/en/developers/sitecoreai/, e.g .../sitecoreai/experience-edge.html, but it only crawled one page. The site isn't blocking it though, the raw HTML has the links robots.txt allows /, the canonical tag matches the URL, and the site returns 200 to the llms-gemini-crawler/1.0 user agent.

Cause

In GeminiCrawlRoutes.cs, CanonicalUrl(raw, start, options) uses its start argument for two things: resolving relative links and the scope check:

var basePath = start.AbsolutePath.TrimEnd('/');
if (basePath.Length > 0 && uri.AbsolutePath != basePath && !uri.AbsolutePath.StartsWith(basePath + "/", StringComparison.Ordinal)) return null;

But discovered links are passed the current page, not the crawl's start URL:

var clean=CanonicalUrl(link.Url,final,options);

That causes two problems:

1. The scope includes the filename. Starting at /docs/intro.html scopes the crawl to /docs/intro.html/..., so /docs/setup.html is rejected.
2. The scope moves with every page. Even with folder-style URLs, starting at /docs/ and reaching /docs/a means a link from /docs/a to /docs/b is rejected. The crawl can only go deeper, never sideways.

Starting from a folder URL doesn't help on sites like this one, because .../sitecoreai/ redirects to .../sitecoreai.html.

The Python original has the same logic (llms/extensions/gemini/crawl.py canonical_url, called with final_url for discovered links), so it's probably affected too.

Suggested fix

Resolve each link against the page it was found on, but always check scope against the start URL. When the start URL ends in a file name, scope to its folder:

static string CrawlScope(Uri start)
{
    var path = start.AbsolutePath; var slash = path.LastIndexOf('/');
    if (!path.EndsWith('/') && path.IndexOf('.', slash + 1) >= 0) path = path[..slash];
    return path.TrimEnd('/');
}

static Uri? CanonicalUrl(string raw, Uri baseUri, Uri start, JsonObject options)
{
    if (!Uri.TryCreate(baseUri, raw, out var uri) || ...) return null;
    // origin check against start (unchanged)
    var basePath = CrawlScope(start); ...
}

// call sites
CanonicalUrl(link.Url, final, start, options)
CanonicalUrl(parsed.Canonical, final, start, options)

/docs and /docs/ start URLs keep their current scope. Only a start URL that ends in a file name gets the wider, folder-level scope.

I have a patch with a regression test (a start page linking to a sibling page, which links to a nested page, plus an out-of-scope link) and I'm happy to open a PR if that's welcome.

[mythz]

Thanks for the report, I've added a fix for this in ServiceStack/ServiceStack@00db6f0

This change is now available in the pre-release packages.

[colby02184]

Thank you for the quick response!