Move defaultLocale into the SearchSchema so image deployments can set it

[ddeboer]

Problem

defaultLocale is the knob that decides whether untagged text is stemmed. It lives in CollectionDefinitionOptions in @lde/search-typesense, and its default – assume nothing – is right:

Snowball stemming locale for non-localized searchable fields (e.g. en). Unset, those fields are not stemmed – folding still applies – so no language is ever assumed.

The placement is the problem: a deployment running the search-indexer image cannot set it. writerFactoryFrom builds the writer options from IndexerConfig and passes only the schema and an optional collection name:

const options = {
  schema,
  ...(config.collectionPrefix ? { name: `${config.collectionPrefix}${deriveCollectionName(searchType)}` } : {}),
};

There is no defaultLocale in IndexerConfig, no environment variable feeding one, and the mounted SCHEMA_MODULE is a default export of SearchType declarations spread into searchSchema(...) – so it has nowhere to carry one either. The option is reachable only by a consumer that constructs BlueGreenRebuild / InPlaceRebuild itself, which is exactly what ADR 0016 asks deployments not to do.

Why it matters

und is not a rare corner. Across the 525 datasets the NDE Dataset Knowledge Graph validates against SCHEMA-AP-NDE, 232 publish schema:name as a plain xsd:string rather than an rdf:langString – 13 685 violations. @lde/search routes every untagged literal to the reserved und locale, so declaring locales: ['nl', 'und'] is what makes those values searchable at all.

But und is folded and unstemmed unless defaultLocale is set (collection-definition.ts:183), so in an image deployment those values only ever match on exact tokens: a query for schilderij misses schilderijen. Half the corpus gets materially worse recall than the conformant half, and the fix exists but is unreachable.

Proposal

Move defaultLocale into the engine-neutral SearchSchema, and have @lde/search-typesense read it from there rather than from its own options. Keep the current default (unset ⇒ nothing assumed).

Rationale:

• It is a statement about the corpus, not the engine. “Untagged text here is Dutch” is engine-neutral; stem: true, locale is one adapter’s realisation of it. Another adapter renders it as an analyzer, or ignores it. That is the ports-and-adapters split the Stack docs already describe, so the declaration belongs on the neutral side.
• Language configuration is already half in the schema. locales is a per-field declaration; defaultLocale says what und in that array actually means. Splitting the pair across two layers means a deployment configures half its language behaviour somewhere it cannot reach from the other half.
• Index time and query time must agree. Both surfaces are configured from the same field model, and the rule is to fold and stem identically on both sides. Stemming configuration that lives only in write-side engine options is how those two drift.
• It restores the config-driven contract. Anything not expressible in the mounted module does not exist to an image deployment.

Scope: global, and why per-dataset is the real answer

locales is per-field; defaultLocale is currently per-collection. Schema-level (one value for every type and field) is the right scope for this option, but only because the finer scopes are either useless or unreachable:

• Per-field is over-engineering – no plausible case wants name and description stemmed as different languages.
• Per-type looks tempting (a Term collection fed from Getty AAT skews English while CreativeWork titles skew Dutch) but does not survive how collections are used: they are shared across datasets, so one Term collection aggregates terms from every publisher. The variance is not per-type.
• Per-dataset is where the variance actually lives – dataset A publishes English, dataset B Dutch – and it is not expressible through this option at all. Stemming is a field property of the collection ({ name, type, stem: true, locale }), and collections are shared, so a single field cannot stem dataset A as en and dataset B as nl.

So this option is a coarse “most untagged text here is Dutch” heuristic. The default must stay unset: a wrong global guess mostly just fails to stem, but it is still a guess.

This may be subsumed – read before implementing

The per-dataset answer is a different mechanism: tag untagged literals with the dataset’s language before extraction, so they land in the existing nl / en buckets instead of und. Stemming is then correct per dataset through the per-locale fanout that already exists, display carries a language, and the SHACL violation disappears as a side effect. That needs dataset context at transform time (ReaderContext.dataset), which the mounted schema module cannot carry today.

If that lands, most of this issue’s justification goes away: nothing reaches the projection untagged, nothing lands in und, and defaultLocale has nothing to stem. Two things would survive:

• defaultLocale also stems the _search companions of keyword and reference fields (:219), which are never language-tagged by nature. That role is independent of untagged literals – but it is a much thinner case, with no measured impact behind it.
• und stays in locales regardless, as the safety net for any dataset without a configured language. Only its stemming becomes moot.

Recommendation: hold this issue rather than implement it. It is the cheap interim (one field on SearchSchema) if per-dataset tagging stalls; in a predominantly Dutch network a global nl is close to right. But it should not be built while the mechanism that would obsolete it is under discussion, or @lde/search gains a knob whose main use case evaporates.

Adding a field to SearchSchema is additive; moving the option off CollectionDefinitionOptions is breaking, so it needs ! / a BREAKING CHANGE: footer if the old option is removed rather than deprecated.

Context

Raised from netwerk-digitaal-erfgoed/stack#10, which proposes und as a default for @ndes/schema-profile-search-config. That default gives recall; this issue is what makes it good.