-
Notifications
You must be signed in to change notification settings - Fork 0
Description
SparqlItemSelector defaults to a page size of 10 when the selector query has no LIMIT clause. This is independent of the stage's batchSize, which can be much larger (e.g. 3000). The result is that the selector makes many small paginated requests (e.g. 300 requests of 10 items) just to fill one batch of 3000 items for the executor.
For example, with batchSize: 3000 and no LIMIT in the selector query, the selector issues 300 LIMIT 10 OFFSET N requests to accumulate enough items for a single executor batch.
Suggested fix
Default the selector's page size to the stage's batchSize so that each selector page request fills exactly one executor batch. This avoids hundreds of unnecessary round-trips to the SPARQL endpoint.
Alternatively, document that users should set LIMIT in the selector query to match their batchSize.