feat(pipeline-void): merge namespace-alias variants into one VoID partition#568
Merged
Conversation
e9a2552 to
15d080d
Compare
…tition Datasets that mix http:// and https://schema.org/ produced two void:classPartition nodes for the same class (the queries key partitions on MD5(class), so the two namespace variants hash differently). Downstream consumers that group by class URI broke on the duplicate (dataset-knowledge-graph#334). Merge the variants after aggregation via a namespaceAliases stage option: - Add mergeNamespaceVariants, a per-stage QuadTransform that re-mints each partition IRI from its canonical key components (replicating the queries' MD5(CONCAT(STR(..)))), collapses the duplicate partition nodes, and sums void:entities / void:triples. Attached to the class and property partition stages. - Keep the analysis queries plain except class-properties-objects.rq, which normalizes at query time: its void:distinctObjects is a distinct-count over object values that overlap across variants, so summing would over-count. The void-ext queries emit their cp -> pp parent chain so the transform can re-key them. - The class selector canonicalizes and de-duplicates its bindings and the reader expands each canonical class back to its namespace variants, so a class's variants are co-located in one batch (the merge is self-contained per stage). - Move NamespaceAlias to @lde/pipeline with canonicalizeIri/aliasVariants helpers; re-export from @lde/pipeline-shacl-sampler. Summed measures are exact under subject/class- and predicate-namespace disjointness, documented on the transform and in ADR 7; a resource typed under both variants over-counts its class entities (pinned by a test). With no aliases the transform is a no-op and behaviour is unchanged.
15d080d to
6154b6e
Compare
…ADR 0007 - Rework the Context paragraph so it no longer implies the analysis makes merged void:entities counts correct — it sums under the same subject-disjointness assumption a consumer would. - Keep the genuine correctness win where it holds (void:distinctObjects, which only the analysis can dedupe from raw object values). - State the real motivation for the placement: single-place normalization and not publishing the duplicate partition nodes that crashed the browser. - Link the paragraph to the Assumptions of record section it must agree with.
70aa815 to
834cb25
Compare
ddeboer
added a commit
that referenced
this pull request
Jul 8, 2026
ADR 0007 is now taken by 0007-merge-namespace-alias-partitions on main (#568), so renumber the per-reference-label-sources ADR from 0007 to 0008 to avoid the collision when this branch merges.
ddeboer
added a commit
that referenced
this pull request
Jul 8, 2026
* feat(search)!: resolve reference labels from per-reference label sources
A reference now declares its label source in the schema:
ReferenceField.labelSource names the SearchType whose collection
resolves the reference’s labels. The named type must declare an output,
searchable text field ‘label’ – something to reconstruct a label from
and something to type ahead against – validated schema-wide by
searchSchema, so a dangling or unsuitable source fails at startup. A
reference without a labelSource stays id-only: its IRIs never travel in
a lookup.
The Typesense engine resolves each reference field from its own
source’s collection, all sources bundled into the single multi_search
it already used; the opt-in in-memory cache now caches per source
collection. Labels reconstruct from the source type’s label declaration
(per-locale display fields), with a bare untagged ‘label’ value as the
und fallback. The global sidecar labels collection is gone – a typed
entity collection and a ‘labels collection’ are the same kind of thing
(one entry in options.collections). See docs/decisions/0007.
Also degeneralizes UI-specific ‘sidebar’ wording in docs to ‘a faceted
listing’ – the sidebar is a consumer notion, not LDE’s.
BREAKING CHANGE: TypesenseSearchEngineOptions.labelsCollection is
removed. Declare each label source as a SearchType (with an output,
searchable text field ‘label’), add its collection to
options.collections, set labelSource on each reference field, and
rebuild label collections via buildCollectionSchema so the physical
label fields exist. fetchLabels now takes label-lookup groups instead
of (collection, iris).
* refactor(search): single-source the label-field convention
Review cleanups on the label sources:
- labelFieldOf (exported via @lde/search/adapter) is the one place that
knows what makes a type a label source; the schema validation and the
Typesense engine both consult it instead of re-deriving the ‘label’
convention per package.
- labelValue reuses the existing localizedValue reconstruction and reads
the untagged fallback from the label field’s own name instead of a
literal.
- The engine precomputes each type’s distinct label-source collections
at construction and reuses the merged cached label map until any
constituent collection reloads, instead of re-deduplicating and
re-merging the full maps on every search.
* fix(search): reject a labelSource declared on a non-reference field
assertResolvableLabelSources validated labelSource only on reference fields, so
a labelSource on a keyword or text field – reachable from a generated or
hand-written schema, the untyped path this validator guards – passed searchSchema
silently and then resolved nothing. Throw at startup so the misconfiguration
fails fast instead of surfacing as unlabelled buckets in production.
* fix(search-typesense): query all label locales and keep empty labels id-only
Two per-reference label-resolution fixes found in review:
- LabelSource.queryBy used only the first locale’s folded search field
(physicalFields(labelField).search[0]); join all per-locale search fields so a
label search matches every locale, not just the first.
- a label document with no usable locale column and no bare label produced
label: {}, which referenceValue treats as a present label; labelValue now
returns undefined and the write skips it, so the reference stays id-only.
* fix(search-typesense): isolate label-source failures and honor id-only references
From the #566 review:
- a single label collection erroring inline (e.g. mid-rebuild) made fetchLabels
throw, blanking every reference on the page to id-only; report the failed
source via onLabelError and skip only its entry, so healthy sources still
resolve.
- a reference or facet with no labelSource could still gain a label in cached
mode, where the whole collection is preloaded into the shared map; skip the
lookup for id-only fields and facets so they stay id-only by declaration.
* perf(search-typesense): precompute label-lookup sources, skip non-source facets
From the #566 review: labelLookupGroups re-derived the output reference fields
and re-resolved their label sources on every search/searchFacets call, and
probed `sources` per facet bucket. Precompute the { field, source } pairs once
at construction, and skip a non-source facet in one check instead of per bucket.
Behaviour-preserving; adds a facet test covering an id-only reference facet.
* docs(search): renumber the label-sources ADR to 0008
ADR 0007 is now taken by 0007-merge-namespace-alias-partitions on main (#568),
so renumber the per-reference-label-sources ADR from 0007 to 0008 to avoid the
collision when this branch merges.
* docs(search): mark the label-sources ADR (0008) Accepted
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
namespaceAliasesoption to the VoID stages that mergeshttp://andhttps://schema.org/variants into a single partition per canonical class/property. Fixes the duplicatevoid:classPartitionreported in netwerk-digitaal-erfgoed/dataset-knowledge-graph#334.Problem
Partitions are keyed on an opaque
MD5(class[, property[, …]])IRI, so a dataset that types entities under bothhttp://schema.org/CreativeWorkandhttps://schema.org/CreativeWorkproduces twovoid:classPartitionnodes that, once the class IRI is normalized, describe the same class. Consumers that group by class URI break on the duplicate (it blanked the Dataset Register browser's detail page). Consumers can't cleanly fix it from the published summary either:void:distinctObjectscan never be merged from the counts alone (object sets overlap), andvoid:entitiescan only be summed under a subject-disjointness assumption. The analysis is the right place to normalize — it has the raw object values to dedupedistinctObjects, and it keeps the duplicate partition nodes from ever being published — but for the summed measures it makes the same disjointness assumption a consumer would.Approach
Merge after aggregation, keeping the analysis queries plain:
mergeNamespaceVariants— a per-stageQuadTransformthat buffers a stage's VoID output, re-mints each partition IRI from its canonical key components (replicating the queries'MD5(CONCAT(STR(…))), pinned by a test against the real#334hashes), collapses the duplicate partition nodes, and sumsvoid:entities/void:triples. Attached to the class and property partition stages.void-extqueries emit theircp → ppparent chain (void:class,void:property) so the transform can re-key them from an otherwise opaque hash.class-properties-objects.rqkeeps query-time normalization — itsvoid:distinctObjectsis a distinct-count over object values that overlap across namespace variants (two works can share a name), so summing would over-count. It's deduped in the query and never reaches the transform. The integration test pins this: a shared object value givesdistinctObjects = 3where a naive sum would give 4.NamespaceAliasmoves to@lde/pipelinewith sharedcanonicalizeIri/aliasVariantshelpers; re-exported from@lde/pipeline-shacl-sampler.Assumptions
Summing pre-aggregated counts is exact only under subject/class disjointness (class-partition and triple sums) and predicate-namespace disjointness (property-partition entity sum). Both hold for the schema.org datasets that motivate this — the variants appear as disjoint subsets. A resource typed under both variants over-counts its class entities; this is documented on the transform, in ADR 7, and pinned by a test. With no aliases configured the transform is a no-op and behaviour is unchanged.
Verification
Unit tests for the transform (incl. the MD5-minting coupling and the malformed-input guards) plus an endpoint-backed integration test over the full VoID output of a mixed-namespace fixture: one merged class partition (4 entities), one property partition (4 entities,
distinctObjects3), merged datatype/language/object-class partitions, and the documented over-count under a disjointness violation.Follow-up (dataset-knowledge-graph, separate repo): pass
namespaceAliases: [{ canonical: 'https://schema.org/', alias: 'http://schema.org/' }]tovoidStages()and drop the write-timeschemaOrgNormalizationPluginrewrite.