Web Connector reindexes all documents on each refresh instead of incremental indexing

[ZhipengHe]

Description

I am using the MIT community version of Onyx and hosting in my own server.

Onyx version: v0.27.0-beta.1

When using the Web Connector with sitemap as the Scrape Method, the system reindexes all pages on each refresh attempt instead of only processing new or changed documents. This leads to unnecessary embedding token usage.

• The connector is configured to refresh once per day (1440 minutes)
• Each indexing attempt processes all documents (~5400+) instead of just new/changed ones; See the number of Total Docs
• Other connector types (Google Drive, Wikipedia) properly perform incremental indexing
• This behaviour significantly increases embedding token consumption, since I am using APIs to call Cohere embedding models

Questions

1. Is this behavior specific to the sitemap Scrape Method or does it affect all Web Connector methods (recursive, single, sitemap)?
2. Is there a way to configure the Web Connector to perform true incremental indexing?
3. If not possible currently, can this feature be implemented to reduce token usage?

[rajivml]

can we have an incremental option where we query urls from site map and query urls from index and do a diff to identify the new URL's added and only scrape them

[github-actions]

This issue is stale because it has been open 75 days with no activity. Remove stale label or comment or this will be closed in 15 days.

[github-actions]

This issue was closed because it has been stalled for 90 days with no activity.