feat(pipeline-void)!: iterate property partitions per property - #672
Merged
Conversation
- Add a property selector mirroring the class selector, so entity-properties.rq runs per VALUES-injected property batch instead of one GROUP BY over every property, which materializes the dataset's full scan and exhausts the query-memory budget on large datasets (measured >16 GB on a 608M-triple dataset), leaving them without property partitions and void:vocabulary - Default the vocabularies stage to per-property iteration, with a perProperty toggle and batchSize/maxConcurrency knobs matching the per-class stages - Keep perProperty out of the shared voidStages stage options so it cannot leak into the global counting stages
- Order both item selectors (ORDER BY) so LIMIT/OFFSET pagination is stable; SPARQL guarantees no order without it, so unordered pages could skip or repeat items on endpoints with unstable ordering - Extract the shared distinctItemSelector helper; classSelector and propertySelector were near-verbatim copies - Emit each void:vocabulary quad once per stage run: withVocabularies remembers emitted vocabularies per distribution instance (WeakMap), so per-property batches produce clean deduplicated output - Reuse PerClassVoidStageOptions for DetectVocabulariesOptions instead of re-declaring batchSize/maxConcurrency - Move the integration test to a repo-unique port (3011); 3007 collided with search-pipeline's suite under concurrent nx runs - Update the pipeline-void reference docs for perProperty, the widened batchSize/maxConcurrency scope, and the per-batch transform window
ddeboer
added a commit
to netwerk-digitaal-erfgoed/dataset-knowledge-graph
that referenced
this pull request
Jul 28, 2026
…441) Adopts [`@lde/pipeline-void` 0.34.0](ldelements/lde#672): the vocabularies stage (`entity-properties.rq`) now iterates per property instead of one `GROUP BY` over every property, bounding query memory by batch instead of dataset size. Effect for the DKG: large datasets get property partitions and `void:vocabulary` where the global query exhausted the QLever budget. Validated on the RKD Knowledge Graph (608M triples, 440 properties) at the production 12 GB `QLEVER_MEMORY_MAX_SIZE`: the worst property (`rdf:type`, 132M triples) computes in 38.7 s; the old global query failed at 12 GB and 16 GB alike. No infrastructure change needed. The stage inherits the existing `batchSize: 1` (per `docs/void-stage-tuning.md` – single-binding queries are the reliable setting for third-party endpoints) and the default `maxConcurrency`. Worth watching in the first runs after deploy: per-endpoint `entity-properties.rq` failures in the logs, since remote endpoints now receive one query per property (the same request pattern the five per-class stages already use).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
entity-properties.rqcomputesCOUNT(DISTINCT ?s)andCOUNT(DISTINCT ?o)grouped over every property in a single query. On large datasets that materializes the full scan: measured on RKD’s 608M-triple dump (dataset-knowledge-graph), the query exhausts a 12 GB and a 16 GB QLevermemory-max-sizebudget (‘Tried to allocate 24 B, but only 0 B were available’), so the dataset ends up with no property partitions and – because vocabulary detection derives from thevoid:propertyoutput – novoid:vocabularyeither. Raising memory further is not an option: the need scales linearly with dataset size, and the 86 per-class stages on the same dataset all pass because they are already chunked.This applies the existing per-class chunking pattern to the property-partition stage:
propertySelector()shares adistinctItemSelectorhelper withclassSelector(): a cheapSELECT DISTINCT(now withORDER BY, soLIMIT/OFFSETpagination is stable) feeds batches that the reader injects as aVALUESblock into the innermost subquery, bounding the working set by batch instead of dataset.detectVocabulariesdefaults to per-property iteration (perProperty: true), reusing the per-classbatchSize/maxConcurrencyknobs;perProperty: falserestores the single global query.withVocabulariesnow remembers which vocabularies it emitted per distribution instance, so per-property batches yield eachvoid:vocabularyquad exactly once per stage run.Validated against the failing dataset
Measured on the RKD Knowledge Graph index (608.3M triples, 440 distinct properties) with QLever at the production 12 GB budget:
rdf:type, 132M triples)VALUESbatchBREAKING CHANGE
transforms[VOID_STAGE_NAMES.vocabularies]that aggregated over the stage’s complete output now sees per-batch windows (like the per-class stages). PassperProperty: falsefor the old single-query behavior.withVocabulariesis stateful per distribution instance: applying one transform instance to multiple streams for the same distribution emits eachvoid:vocabularyquad only once.voidStagesbatchSize/maxConcurrencynow also apply to the vocabularies stage.Known follow-up (pre-existing for per-class stages, now also relevant here):
SparqlItemSelectorrequests have no retry, unlike the reader’spRetry– a transient 5xx on a selector page fails the stage.The integration test runs the stage chunked (
batchSize: 1) and unchunked against a local SPARQL endpoint and asserts identical deduplicated output, correct per-propertyvoid:entities/void:distinctObjects, named-graph scoping of the selector, and single-emission vocabulary detection across batches.