Gemini web crawl: show the number of pages crawled and whether a limit was reached

[colby02184]

Follow-up to #159 (thanks again for the quick crawler scope fix, it works great in 10.4.1).

Current behavior

After a crawl from Admin → Gemini → Import → Web crawl finishes successfully, the UI re-enables the Crawl website button and opens the saved import, but doesn't show any result. The only way to see what happened is to open View crawled pages and count.

Crawls can take several minutes (500 pages took ~5 minutes on doc.sitecore.com), so it's easy to miss that the crawl finished, or to not notice it was cut short.

Request

1. Show the page count after a crawl. /imports/crawl already returns pages, but startCrawl() only calls loadImports() and editSource(...). A message like "Crawled 500 pages" would be enough.

2. Say when a limit stopped the crawl. My crawl stopped at exactly the default Max pages (500) with more pages left, and nothing indicated that. The loop in CrawlSiteAsync can end for three reasons: the queue is empty, pages.Count >= maxPages, or requests >= maxRequests. Returning the reason (and the number of URLs still queued) would let the UI show something like:

   Crawled 500 pages. Stopped at the Max pages limit (500) with 1,240 links not yet visited. Increase Max pages to crawl more.

   For example, the response could add:

   { "pages": 500, "stopReason": "maxPages", "remaining": 1240 }

   where stopReason is complete, maxPages or maxRequests.

3. Optional: for long crawls, some progress indication (pages saved so far) would help, since the request currently runs for minutes with only a disabled button.

Thank you for your consideration on this!

[mythz]

Thanks for the detailed suggestions. These are now implemented in both the C# and Python versions:

• A completion summary shows how many pages were crawled.
• If Max pages or Max requests stops the crawl, the summary identifies the limit and how many URLs remain queued.
• While crawling, a spinner and live page count update every two seconds.
• Individual page failures are collected and shown in expandable error details, allowing the crawl to continue.
  The same progress and results apply when rerunning a saved web crawl. Each rerun starts fresh, with its progress count beginning at zero.
  The response now includes stopReason (complete, maxPages, or maxRequests), remaining, and per-URL error details. There’s also a Max requests setting in the UI so either limit can be adjusted.

Here's a screenshot showing progress during a crawl:

This is now available in the latest v10.4.1 pre-release packages.