Skip to content

feat(pipeline-void): merge namespace-alias variants into one VoID partition#568

Merged
ddeboer merged 2 commits into
mainfrom
fix/void-namespace-normalization
Jul 8, 2026
Merged

feat(pipeline-void): merge namespace-alias variants into one VoID partition#568
ddeboer merged 2 commits into
mainfrom
fix/void-namespace-normalization

Conversation

@ddeboer

@ddeboer ddeboer commented Jul 7, 2026

Copy link
Copy Markdown
Member

Adds a namespaceAliases option to the VoID stages that merges http:// and https://schema.org/ variants into a single partition per canonical class/property. Fixes the duplicate void:classPartition reported in netwerk-digitaal-erfgoed/dataset-knowledge-graph#334.

Problem

Partitions are keyed on an opaque MD5(class[, property[, …]]) IRI, so a dataset that types entities under both http://schema.org/CreativeWork and https://schema.org/CreativeWork produces two void:classPartition nodes that, once the class IRI is normalized, describe the same class. Consumers that group by class URI break on the duplicate (it blanked the Dataset Register browser's detail page). Consumers can't cleanly fix it from the published summary either: void:distinctObjects can never be merged from the counts alone (object sets overlap), and void:entities can only be summed under a subject-disjointness assumption. The analysis is the right place to normalize — it has the raw object values to dedupe distinctObjects, and it keeps the duplicate partition nodes from ever being published — but for the summed measures it makes the same disjointness assumption a consumer would.

Approach

Merge after aggregation, keeping the analysis queries plain:

  • mergeNamespaceVariants — a per-stage QuadTransform that buffers a stage's VoID output, re-mints each partition IRI from its canonical key components (replicating the queries' MD5(CONCAT(STR(…))), pinned by a test against the real #334 hashes), collapses the duplicate partition nodes, and sums void:entities / void:triples. Attached to the class and property partition stages.
  • The class selector canonicalizes and de-duplicates its bindings, and the reader expands each canonical class back to its namespace variants, so a class's variants are co-located in one batch — the merge is self-contained per stage, no pipeline-core change needed.
  • The three void-ext queries emit their cp → pp parent chain (void:class, void:property) so the transform can re-key them from an otherwise opaque hash.
  • class-properties-objects.rq keeps query-time normalization — its void:distinctObjects is a distinct-count over object values that overlap across namespace variants (two works can share a name), so summing would over-count. It's deduped in the query and never reaches the transform. The integration test pins this: a shared object value gives distinctObjects = 3 where a naive sum would give 4.
  • NamespaceAlias moves to @lde/pipeline with shared canonicalizeIri / aliasVariants helpers; re-exported from @lde/pipeline-shacl-sampler.

Assumptions

Summing pre-aggregated counts is exact only under subject/class disjointness (class-partition and triple sums) and predicate-namespace disjointness (property-partition entity sum). Both hold for the schema.org datasets that motivate this — the variants appear as disjoint subsets. A resource typed under both variants over-counts its class entities; this is documented on the transform, in ADR 7, and pinned by a test. With no aliases configured the transform is a no-op and behaviour is unchanged.

Verification

Unit tests for the transform (incl. the MD5-minting coupling and the malformed-input guards) plus an endpoint-backed integration test over the full VoID output of a mixed-namespace fixture: one merged class partition (4 entities), one property partition (4 entities, distinctObjects 3), merged datatype/language/object-class partitions, and the documented over-count under a disjointness violation.

Follow-up (dataset-knowledge-graph, separate repo): pass namespaceAliases: [{ canonical: 'https://schema.org/', alias: 'http://schema.org/' }] to voidStages() and drop the write-time schemaOrgNormalizationPlugin rewrite.

@ddeboer ddeboer force-pushed the fix/void-namespace-normalization branch 2 times, most recently from e9a2552 to 15d080d Compare July 8, 2026 11:08
…tition

Datasets that mix http:// and https://schema.org/ produced two
void:classPartition nodes for the same class (the queries key partitions
on MD5(class), so the two namespace variants hash differently). Downstream
consumers that group by class URI broke on the duplicate
(dataset-knowledge-graph#334).

Merge the variants after aggregation via a namespaceAliases stage option:

- Add mergeNamespaceVariants, a per-stage QuadTransform that re-mints each
  partition IRI from its canonical key components (replicating the queries'
  MD5(CONCAT(STR(..)))), collapses the duplicate partition nodes, and sums
  void:entities / void:triples. Attached to the class and property
  partition stages.
- Keep the analysis queries plain except class-properties-objects.rq, which
  normalizes at query time: its void:distinctObjects is a distinct-count
  over object values that overlap across variants, so summing would
  over-count. The void-ext queries emit their cp -> pp parent chain so the
  transform can re-key them.
- The class selector canonicalizes and de-duplicates its bindings and the
  reader expands each canonical class back to its namespace variants, so a
  class's variants are co-located in one batch (the merge is self-contained
  per stage).
- Move NamespaceAlias to @lde/pipeline with canonicalizeIri/aliasVariants
  helpers; re-export from @lde/pipeline-shacl-sampler.

Summed measures are exact under subject/class- and predicate-namespace
disjointness, documented on the transform and in ADR 7; a resource typed
under both variants over-counts its class entities (pinned by a test). With
no aliases the transform is a no-op and behaviour is unchanged.
@ddeboer ddeboer force-pushed the fix/void-namespace-normalization branch from 15d080d to 6154b6e Compare July 8, 2026 12:33
…ADR 0007

- Rework the Context paragraph so it no longer implies the analysis makes
  merged void:entities counts correct — it sums under the same
  subject-disjointness assumption a consumer would.
- Keep the genuine correctness win where it holds (void:distinctObjects,
  which only the analysis can dedupe from raw object values).
- State the real motivation for the placement: single-place normalization
  and not publishing the duplicate partition nodes that crashed the browser.
- Link the paragraph to the Assumptions of record section it must agree with.
@ddeboer ddeboer force-pushed the fix/void-namespace-normalization branch from 70aa815 to 834cb25 Compare July 8, 2026 13:59
@ddeboer ddeboer merged commit 8367750 into main Jul 8, 2026
2 checks passed
@ddeboer ddeboer deleted the fix/void-namespace-normalization branch July 8, 2026 18:56
ddeboer added a commit that referenced this pull request Jul 8, 2026
ADR 0007 is now taken by 0007-merge-namespace-alias-partitions on main (#568),
so renumber the per-reference-label-sources ADR from 0007 to 0008 to avoid the
collision when this branch merges.
ddeboer added a commit that referenced this pull request Jul 8, 2026
* feat(search)!: resolve reference labels from per-reference label sources

A reference now declares its label source in the schema:
ReferenceField.labelSource names the SearchType whose collection
resolves the reference’s labels. The named type must declare an output,
searchable text field ‘label’ – something to reconstruct a label from
and something to type ahead against – validated schema-wide by
searchSchema, so a dangling or unsuitable source fails at startup. A
reference without a labelSource stays id-only: its IRIs never travel in
a lookup.

The Typesense engine resolves each reference field from its own
source’s collection, all sources bundled into the single multi_search
it already used; the opt-in in-memory cache now caches per source
collection. Labels reconstruct from the source type’s label declaration
(per-locale display fields), with a bare untagged ‘label’ value as the
und fallback. The global sidecar labels collection is gone – a typed
entity collection and a ‘labels collection’ are the same kind of thing
(one entry in options.collections). See docs/decisions/0007.

Also degeneralizes UI-specific ‘sidebar’ wording in docs to ‘a faceted
listing’ – the sidebar is a consumer notion, not LDE’s.

BREAKING CHANGE: TypesenseSearchEngineOptions.labelsCollection is
removed. Declare each label source as a SearchType (with an output,
searchable text field ‘label’), add its collection to
options.collections, set labelSource on each reference field, and
rebuild label collections via buildCollectionSchema so the physical
label fields exist. fetchLabels now takes label-lookup groups instead
of (collection, iris).

* refactor(search): single-source the label-field convention

Review cleanups on the label sources:

- labelFieldOf (exported via @lde/search/adapter) is the one place that
  knows what makes a type a label source; the schema validation and the
  Typesense engine both consult it instead of re-deriving the ‘label’
  convention per package.
- labelValue reuses the existing localizedValue reconstruction and reads
  the untagged fallback from the label field’s own name instead of a
  literal.
- The engine precomputes each type’s distinct label-source collections
  at construction and reuses the merged cached label map until any
  constituent collection reloads, instead of re-deduplicating and
  re-merging the full maps on every search.

* fix(search): reject a labelSource declared on a non-reference field

assertResolvableLabelSources validated labelSource only on reference fields, so
a labelSource on a keyword or text field – reachable from a generated or
hand-written schema, the untyped path this validator guards – passed searchSchema
silently and then resolved nothing. Throw at startup so the misconfiguration
fails fast instead of surfacing as unlabelled buckets in production.

* fix(search-typesense): query all label locales and keep empty labels id-only

Two per-reference label-resolution fixes found in review:
- LabelSource.queryBy used only the first locale’s folded search field
  (physicalFields(labelField).search[0]); join all per-locale search fields so a
  label search matches every locale, not just the first.
- a label document with no usable locale column and no bare label produced
  label: {}, which referenceValue treats as a present label; labelValue now
  returns undefined and the write skips it, so the reference stays id-only.

* fix(search-typesense): isolate label-source failures and honor id-only references

From the #566 review:
- a single label collection erroring inline (e.g. mid-rebuild) made fetchLabels
  throw, blanking every reference on the page to id-only; report the failed
  source via onLabelError and skip only its entry, so healthy sources still
  resolve.
- a reference or facet with no labelSource could still gain a label in cached
  mode, where the whole collection is preloaded into the shared map; skip the
  lookup for id-only fields and facets so they stay id-only by declaration.

* perf(search-typesense): precompute label-lookup sources, skip non-source facets

From the #566 review: labelLookupGroups re-derived the output reference fields
and re-resolved their label sources on every search/searchFacets call, and
probed `sources` per facet bucket. Precompute the { field, source } pairs once
at construction, and skip a non-source facet in one check instead of per bucket.
Behaviour-preserving; adds a facet test covering an id-only reference facet.

* docs(search): renumber the label-sources ADR to 0008

ADR 0007 is now taken by 0007-merge-namespace-alias-partitions on main (#568),
so renumber the per-reference-label-sources ADR from 0007 to 0008 to avoid the
collision when this branch merges.

* docs(search): mark the label-sources ADR (0008) Accepted
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant