Skip to content

feat(pipeline-void)!: iterate property partitions per property - #672

Merged
ddeboer merged 2 commits into
mainfrom
feat/per-property-partitions
Jul 28, 2026
Merged

feat(pipeline-void)!: iterate property partitions per property#672
ddeboer merged 2 commits into
mainfrom
feat/per-property-partitions

Conversation

@ddeboer

@ddeboer ddeboer commented Jul 27, 2026

Copy link
Copy Markdown
Member

entity-properties.rq computes COUNT(DISTINCT ?s) and COUNT(DISTINCT ?o) grouped over every property in a single query. On large datasets that materializes the full scan: measured on RKD’s 608M-triple dump (dataset-knowledge-graph), the query exhausts a 12 GB and a 16 GB QLever memory-max-size budget (‘Tried to allocate 24 B, but only 0 B were available’), so the dataset ends up with no property partitions and – because vocabulary detection derives from the void:property output – no void:vocabulary either. Raising memory further is not an option: the need scales linearly with dataset size, and the 86 per-class stages on the same dataset all pass because they are already chunked.

This applies the existing per-class chunking pattern to the property-partition stage:

Validated against the failing dataset

Measured on the RKD Knowledge Graph index (608.3M triples, 440 distinct properties) with QLever at the production 12 GB budget:

Query Result
Old global query ✖ out of memory (also at 16 GB)
Chunked, worst property (rdf:type, 132M triples) ✔ 38.7 s
Chunked, second-heaviest (86M triples) ✔ 24.7 s
Chunked, 10 mid-tier properties in one VALUES batch ✔ 2.7 s

BREAKING CHANGE

  • The vocabularies stage iterates per property by default: a consumer transform attached via transforms[VOID_STAGE_NAMES.vocabularies] that aggregated over the stage’s complete output now sees per-batch windows (like the per-class stages). Pass perProperty: false for the old single-query behavior.
  • withVocabularies is stateful per distribution instance: applying one transform instance to multiple streams for the same distribution emits each void:vocabulary quad only once.
  • The shared voidStages batchSize/maxConcurrency now also apply to the vocabularies stage.

Known follow-up (pre-existing for per-class stages, now also relevant here): SparqlItemSelector requests have no retry, unlike the reader’s pRetry – a transient 5xx on a selector page fails the stage.

The integration test runs the stage chunked (batchSize: 1) and unchunked against a local SPARQL endpoint and asserts identical deduplicated output, correct per-property void:entities/void:distinctObjects, named-graph scoping of the selector, and single-emission vocabulary detection across batches.

ddeboer added 2 commits July 27, 2026 19:21
- Add a property selector mirroring the class selector, so
  entity-properties.rq runs per VALUES-injected property batch instead of
  one GROUP BY over every property, which materializes the dataset's full
  scan and exhausts the query-memory budget on large datasets (measured
  >16 GB on a 608M-triple dataset), leaving them without property
  partitions and void:vocabulary
- Default the vocabularies stage to per-property iteration, with a
  perProperty toggle and batchSize/maxConcurrency knobs matching the
  per-class stages
- Keep perProperty out of the shared voidStages stage options so it
  cannot leak into the global counting stages
- Order both item selectors (ORDER BY) so LIMIT/OFFSET pagination is
  stable; SPARQL guarantees no order without it, so unordered pages
  could skip or repeat items on endpoints with unstable ordering
- Extract the shared distinctItemSelector helper; classSelector and
  propertySelector were near-verbatim copies
- Emit each void:vocabulary quad once per stage run: withVocabularies
  remembers emitted vocabularies per distribution instance (WeakMap),
  so per-property batches produce clean deduplicated output
- Reuse PerClassVoidStageOptions for DetectVocabulariesOptions instead
  of re-declaring batchSize/maxConcurrency
- Move the integration test to a repo-unique port (3011); 3007 collided
  with search-pipeline's suite under concurrent nx runs
- Update the pipeline-void reference docs for perProperty, the widened
  batchSize/maxConcurrency scope, and the per-batch transform window
@ddeboer ddeboer changed the title feat(pipeline-void): iterate property partitions per property feat(pipeline-void)!: iterate property partitions per property Jul 27, 2026
@ddeboer
ddeboer merged commit 643ba20 into main Jul 28, 2026
4 checks passed
@ddeboer
ddeboer deleted the feat/per-property-partitions branch July 28, 2026 18:16
ddeboer added a commit to netwerk-digitaal-erfgoed/dataset-knowledge-graph that referenced this pull request Jul 28, 2026
…441)

Adopts [`@lde/pipeline-void`
0.34.0](ldelements/lde#672): the vocabularies
stage (`entity-properties.rq`) now iterates per property instead of one
`GROUP BY` over every property, bounding query memory by batch instead
of dataset size.

Effect for the DKG: large datasets get property partitions and
`void:vocabulary` where the global query exhausted the QLever budget.
Validated on the RKD Knowledge Graph (608M triples, 440 properties) at
the production 12 GB `QLEVER_MEMORY_MAX_SIZE`: the worst property
(`rdf:type`, 132M triples) computes in 38.7 s; the old global query
failed at 12 GB and 16 GB alike. No infrastructure change needed.

The stage inherits the existing `batchSize: 1` (per
`docs/void-stage-tuning.md` – single-binding queries are the reliable
setting for third-party endpoints) and the default `maxConcurrency`.
Worth watching in the first runs after deploy: per-endpoint
`entity-properties.rq` failures in the logs, since remote endpoints now
receive one query per property (the same request pattern the five
per-class stages already use).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant