Skip to content

Selector page size should default to stage batchSize #298

@ddeboer

Description

@ddeboer

SparqlItemSelector defaults to a page size of 10 when the selector query has no LIMIT clause. This is independent of the stage's batchSize, which can be much larger (e.g. 3000). The result is that the selector makes many small paginated requests (e.g. 300 requests of 10 items) just to fill one batch of 3000 items for the executor.

For example, with batchSize: 3000 and no LIMIT in the selector query, the selector issues 300 LIMIT 10 OFFSET N requests to accumulate enough items for a single executor batch.

Suggested fix

Default the selector's page size to the stage's batchSize so that each selector page request fills exactly one executor batch. This avoids hundreds of unnecessary round-trips to the SPARQL endpoint.

Alternatively, document that users should set LIMIT in the selector query to match their batchSize.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions