Ship a searchIndexerPipeline convenience that wires the common indexer (schema + datasets -> Pipeline)

[ddeboer]

Motivation

Building a search indexer on the published stack means two costs. The first is discovery — finding the concrete pieces (the local SPARQL server is @lde/sparql-qlever; @lde/sparql-server/@lde/sparql-importer are interface-only stubs; the console reporter is @lde/pipeline-console-reporter; a file-backed provenance store now exists via #634/#636). The second, and larger, is that the wiring itself is boilerplate every consumer repeats near-identically — none of it is domain-specific:

• one stage per root type, each selecting the type’s non-blank roots (see Blank-node roots crash framing; selectByClass selects them #641);
• createQlever + ImportResolver for the import path;
• searchIndexWriter + a per-type InPlaceRebuild;
• provenance store + pipelineVersion, and a reporter.

So rather than only documenting this (the original scope of this issue), ship it as a convenience.

Proposal

A convenience — in @lde/search-pipeline, or a new @lde/search-indexer — that wires the common object-grain indexer and returns the Pipeline:

searchIndexerPipeline({
  schema,               // SearchSchema
  datasets,             // Dataset[] (or a DatasetSelector)
  distributionResolver, // e.g. ImportResolver(new SparqlDistributionResolver(), createQlever(...))
  writerFor,            // (RootType) => Writer, e.g. t => new InPlaceRebuild(client, t)
  provenanceStore,      // optional
  pipelineVersion,      // required with provenanceStore
  reporter,             // optional
}): Pipeline<TypedSearchDocument>

It generates one stage per root type in the schema, each with a blank-node-safe selectByClass (bake in #641), and returns the wired pipeline. Consumers then supply only: the schema, which datasets, the engine writer, and the SPARQL/import adapter.

What stays with the consumer (out of scope)

The domain — the SearchSchema declaration and which datasets — and the deployment shell — the engine client + coordinates, the container. The convenience wires the pipeline; it does not choose the engine or own deployment. (A fully config-driven indexer image is a separate, bigger step that depends on schema-as-data / SHACL.)

Reference implementation to extract from

A downstream project (Linked Open Limburg) already wrote exactly this; its pipeline.ts is the extract-from reference:

export function rootSelectorQuery(classIri: string): string {
  // Blank-node roots have no stable document key and crash framing — exclude them.
  return `SELECT DISTINCT ?root WHERE { ?root a <${classIri}> . FILTER(!isBlank(?root)) }`;
}

function indexerStageTypes(schema: SearchSchema): SearchStageType[] {
  return [...schema.values()].map((searchType: RootType) => ({
    searchType,
    rootVariable: 'root', // must not be `dataset`
    itemSelector: new SparqlItemSelector({ query: rootSelectorQuery(searchType.class) }),
  }));
}

function buildPipeline(config): Pipeline<TypedSearchDocument> {
  const { importer, server } = createQlever({ mode: 'docker', image, dataDir, cacheIndex });
  return new Pipeline<TypedSearchDocument>({
    datasetSelector: new ManualDatasetSelection(datasets),
    distributionResolver: new ImportResolver(new SparqlDistributionResolver(), { importer, server, strategy: 'import' }),
    stages: searchStages({ schema, types: indexerStageTypes(schema) }),
    writers: searchIndexWriter({ schema, writerFor: (t) => new InPlaceRebuild(client, t) }),
    provenanceStore: new FileProvenanceStore({ path }),
    pipelineVersion,
    reporter: new ConsoleReporter(),
  });
}

The only bespoke lines are the domain ones (datasets, schema) and the engine/adapter choices — everything else is what the convenience would own.

Supporting docs (kept from the original scope)

• Document the role → package mapping (dataset selection, distribution resolution via @lde/sparql-qlever, stages, writer via @lde/search-typesense, provenance, reporter).
• Flesh out the stub READMEs of @lde/sparql-server and @lde/sparql-importer to point at @lde/sparql-qlever as the concrete adapter.

Relationships

• Pairs with Blank-node roots crash framing; selectByClass selects them #641 (blank-node-safe root selection) — the convenience should bake it in.
• Would have removed most of the wiring that surfaced createQlever data-dump import fails on current QLever: --parse-parallel rejected + downloader doesn't mkdir #633 / createQlever({mode:'docker'}) silently fails without dataDir (no bind-mount, wrong workdir) #638 on the consumer side.