search: drop the local flag – every lookup stores the referring document’s copy of the referent

[ddeboer]

Problem

A lookup reference stores ids; local: true makes it also store what the referring document states about the referent, so that a referent the graph names inline – a blank node, a literal – shows at all, and so that a referent whose document is missing from the target’s collection degrades to what the referring document said rather than to a bare IRI.

Conceptually, a local lookup is an inline document that carries a reference with a lookup target. The flag exists because the two halves are wired differently in the collection builder: a local lookup stores a nested object plus a flat ${name}_id companion, and the companion serves filterable only. assertServiceableLocalLookup in @lde/search therefore refuses facetable and joinable on a local lookup, because the builder emits neither a facet field nor an engine reference for the companion. Typesense itself has no such restriction: an inline reference with an identity already facets on its companion, and a companion can carry a reference: <collection>.id so a join works on it.

Proposal

Drop the local flag: every lookup stores the referring document’s copy of the referent beside the id.

• The _id companion serves every role – filter as today, facet as the inline identity companion does, join by declaring it as the engine reference – and the refusal in assertServiceableLocalLookup goes.
• The copy stays stored unconditionally, not only for referents without an id: at index time the question is is this referent identified, at query time it is is that document indexed, and only the first is answerable during extraction. Storing conditionally would shrink the stored document but drop the identified-but-not-indexed fallback, and would not reduce framing reach (see below).
• Resolution stays at query time and replaces the stored copy with the target’s record where the lookup finds one (search-typesense: a resolved local lookup merges the target’s record with what the referring document stated #816). The stored copy remains available should a surface want to show what the referring dataset says about the referent.
• Keyed targets already cover the “sameAs vs node IRI” question: the companion holds the target’s document key, so a deployment decides through key whether that is the authority’s IRI or the node’s.

Cost

Framing reach: the copy is read from the graph while the referring document is framed, and the fields it copies live on the referent’s node, one hop past the edge – and further where the target’s own declaration carries keyed or nested references. Today only fields marked local pay that; without the flag every lookup does, including ones where nobody wants the referring dataset’s view. Extraction moves the referent’s subgraph per referent per document, and each stored document grows by one copy per referent. For narrow targets (a name and a few fields) this is expected to be modest.

Breaking: changes the stored shape of every lookup, so every deployment reindexes.