Repository navigation
Do scheduled website syncs re-crawl every page? #3104
Replies: 1 comment 2 replies
|
On current main, scheduled website syncs run the normal loader again, so they re-fetch the selected pages rather than using a changed-page list. The scheduler passes the stored URL or configuration through The scope depends on the source type.
The factory currently constructs those loaders with their defaults, including 10 pages for the crawler and 20 for a sitemap. So “every page” means the configured selection within those limits, rather than every page on the site. The connector fingerprint fast path elsewhere in the worker is separate from these website loaders. I checked repeated loader calls with offline HTTP fixtures. Identical pages were requested again. That verifies loader requests, not live server traffic, embedding-call counts or a complete scheduled job. An added/removed-URL signal could inform an incremental design, but it would also need a way to detect content edits at an unchanged URL. It does not currently bypass the full loader path. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
The Background Jobs docs describe daily, weekly and monthly re-ingestion of synced sources, so we wanted to ask: does a scheduled sync of a website source re-crawl every page today, or only the pages that changed?
All reactions