Extract root types from the Dataset Register endpoint, not only from the distribution

[ddeboer]

Context

search-indexer extracts every root type from the dataset’s distribution – a live SPARQL endpoint, or a dump imported into QLever. REGISTRY_ENDPOINT is used only to select datasets and resolve their distributions; the register’s own triples never reach the extraction.

That works while every indexed class is described by the data the dataset publishes. It breaks for the dataset itself.

A dataset description is governed by a different application profile from the objects it contains – in the NDE stack, DCAT-AP-NL for the dataset and SCHEMA-AP-NDE for the objects – and it lives in the register, not in the distribution. Some publishers happen to ship a schema:Dataset node in their dump; nothing obliges them to. So a deployment that declares a Dataset root type today gets documents for the publishers that self-describe and nothing for the rest, which in turn leaves every reference resolved through it (a labelSource, or a provenance field – see the companion issue) unlabelled.

Registering a dataset is submitting a description, so the register is the one source that covers every dataset a pipeline can select.

Why the registry client’s typed Dataset is not enough

Client.query reads the register through an LDkit lens (createLens(DatasetSchema, …)) and frames the result into @lde/dataset’s Dataset. The raw quads exist inside LDkit and are never surfaced, so the reachable properties are whatever DatasetSchema declares: title, description, license, publisher {iri, name}, creator, language, distributions.

Useful properties that are simply absent: dcat:landingPage (86% coverage in the NDE register), dcat:keyword, dcat:theme, dcterms:spatial, dcterms:temporal, dcat:contactPoint.

Widening the lens is ~2 lines per property, but it makes the upstream model, not the profile, the ceiling on what a deployment can index – and every new property needs an LDE release before a deployment can declare it. The SearchSchema is meant to be the single declaration that decides which properties become fields, facets and API surface; for datasets it currently cannot be.

Proposal

Let the schema-derived extraction run against the registry SPARQL endpoint for a configured set of root types, within the same per-dataset pass:

• distribution → CreativeWork, Person, Organization, Place, Term, Occupation, …
• registry → Dataset, Publisher, …

Same CONSTRUCT generator, same framing, same projection, same writers – only the endpoint differs.

What the spike established

A throwaway spike ran the whole path – schema declaration → generated CONSTRUCT → live register → projection – against https://datasetregister.netwerkdigitaalerfgoed.nl/sparql, rooted at https://id.drapo.nl/dataset/drapo-schemaorg. It confirmed the design and corrected two premises this issue previously stated.

The register’s per-dataset named graph is the unit of work

All 2731 dcat:Dataset registrations live in a named graph called exactly after the dataset IRI (STR(?g) = STR(?d), 2731/2731). So the dataset in hand names its own graph, and no lookup or new scoping machinery is needed – FROM <dataset-iri> is enough, which is exactly the existing Distribution.namedGraph mechanism (withDefaultGraph).

Verified against the endpoint: a FROM <graph>-scoped extraction returns the same result as an explicitly GRAPH ?root-wrapped one.

A registration IS self-contained in its graph (correction)

This issue previously claimed that “a dataset’s graph holds ~49 triples about the dataset, while its publisher’s foaf:name hangs off the publisher IRI outside it”, and concluded that a graph-scoped fetch would yield publisher IRIs with no labels. That is not the case. Register-wide:

• 2731/2731 dataset→publisher pairs have the publisher’s foaf:name inside the dataset’s own graph;
• all 163 distinct publishers have a foaf:name;
• 2730/2731 have the publisher typed foaf:Organization inside that graph too.

So a graph-scoped fetch resolves the publisher hop, and selectByClass works unchanged for a Publisher root type once the source is scoped: ?root a foaf:Organization inside FROM <dataset-iri> selects that dataset’s publisher and nothing else. (Drapo’s graph holds ~100 triples, not 49.)

The path-driven argument still holds – hops come from the declared paths, no CBD rule and no hardcoded predicate list – it just is not load-bearing against graph scoping the way this issue assumed.

Graph scoping is what keeps schema:Dataset out

The register splits the two types across graphs rather than asserting both on one node:

type	graph
dcat:Dataset	the dataset’s own graph (the crawled description)
schema:Dataset	…/registry/registrations (the registration record)

Selecting by dcat:Dataset inside the dataset’s graph therefore never reaches the registrations graph at all. It also excludes the 775 subjects that are schema:Dataset but never dcat:Dataset – registrations whose description was never crawled – which a schema.org-rooted selector would sweep in as content-free documents.

Roots scoped to the dataset in hand

Confirmed: root selection needs no new machinery. The extraction generator leaves the subject variable free, and a one-element VALUES ?root { … } (or the FROM-scoped selectByClass) pins it. Both were run end to end.

Faceting on publisher needs a Publisher root type

Worth stating because it constrains the schema a deployment writes. A facet bucket carries a label only when the reference field declares a labelSource; labelSource must name a type with an output, searchable text field label; and a Reference Type may not declare searchable (assertServiceableNestedFields). Therefore a label source is always a Root Type with its own collection – an inline-nested publisher cannot be faceted (declaring facetable on an inline reference is rejected outright).

Under per-dataset graph scoping this costs nothing: Publisher becomes an ordinary registry-sourced stage selected by foaf:Organization within the dataset’s graph.

One consequence to watch: 19 of 163 publisher IRIs carry two foaf:name values that are nl/en translations (Limburgs Museum@nl / Limburg Museum@en, Open Archieven@nl / Open Archives@en), and the variants can sit in different dataset graphs. Per-dataset scoping therefore writes whichever labels the dataset in hand carries, so a publisher document’s language coverage depends on which dataset wrote it last.

Two data-quality findings, out of scope here

• Registrations that differ only by URI encoding. …/dataset/LGBTI%2B%20Objecten and …/dataset/LGBTI%2b+Objecten are two separate 54-triple graphs with identical content – two documents a user sees as one dataset. Strictly different RDF IRIs, so this is upstream registration identity, not something the indexer causes. Both round-trip byte-identically through injectValues → generator → endpoint → parse.
• The register’s endpoint does not deduplicate CONSTRUCT output. It emits one triple per solution even for a constant template triple within a single graph (SPARQL 1.1 says CONSTRUCT yields an RDF graph, i.e. a set). A Publisher extraction unscoped across graphs returned 617 quads for 3 publishers. SparqlConstructReader already has deduplicate, and per-dataset graph scoping avoids the blow-up anyway; the projection collapses the duplicates correctly either way.

A latent robustness gap

injectValues does not escape IRIs: namedNode('http://example.org/a b') serialises as <http://example.org/a b>, which the register rejects with HTTP 400 – so one malformed root IRI fails the whole batch’s extraction, not just its own row. No dcat:Dataset root contains a character illegal inside <> today (0 of 2731), so this is latent. Non-ASCII is fine (é is legal in an IRIREF, and raw non-ASCII IRIs do occur in the register).

Implementation shape

Given the above, this is routing plus one gap-fill:

1. @lde/pipeline – a per-stage source override. Stage.run(dataset, distribution, …) holds both halves, so a single hook (dataset, distribution) => Distribution covers the selector and every reader at once. A registry-sourced stage returns { accessUrl: REGISTRY_ENDPOINT, namedGraph: dataset.iri }.
2. @lde/pipeline – SparqlItemSelector must honour Distribution.namedGraph. The reader applies it (withDefaultGraph), the selector ignores it. Without this the registry-side selection is unscoped and the first run against a real register selects the entire catalogue – precisely the failure this issue warns about.
3. @lde/search-pipeline – pass the override through SearchStageType, and let searchIndexerPipeline build it for a named set of root types.
4. @lde/search-indexer – the configuration setting alongside REGISTRY_ENDPOINT naming the registry-sourced root types.

No change to the extraction generator, the projection, the writers or SearchType.

Routing belongs in configuration, not in the schema

Which root types come from the registry is deployment topology, and SearchType is defined by its class, not by where its triples come from – a per-type source hint would push deployment concerns into a declaration that is otherwise engine- and deployment-neutral. A configuration setting alongside REGISTRY_ENDPOINT (“these root types extract from the registry”) keeps the schema portable.

Running every root type against every endpoint is not a viable substitute for routing. Class-based self-selection is a useful safety property – a misrouted CreativeWork extraction against the register matches nothing, so a mistake yields empty rather than wrong – but as a mechanism it means N×M queries, most of them pointless, and several of them unscoped queries against a shared public NDE service.

Related

• Make the indexed dataset available at projection time as a declarable field #696 – the dataset IRI at projection time (provenance as a declarable field).
• Single-valued provenance stamp: shared entities lose sources and can be swept while still contributed #697 – the single-valued source stamp bug; independent, but adjacent.