Serve inline references as nested output

[ddeboer]

Problem Statement

A Deployment that wants a nested object in its API – a root type carrying several
values that all belong to one referenced thing – cannot get one.

CONTEXT.md describes an Inline Reference as serving two jobs, told apart only by
Roles: a reading device (no Roles, pruned before the writer) and an API device
(output, “deliberately surface a nested Reference Type”). ADR
11 rests on that
distinction. Only the reading device works. The API device is declared, validated
and projected, and then silently dropped: the engine stores a reference as an IRI
string, document reconstruction rebuilds it as an id plus an optional label, and the
GraphQL surface serves every reference as a fixed two-field type. A Reference
Type’s own Search Fields never reach storage or any surface.

The workaround is to flatten each of the referent’s values onto the root as its own
Search Field, using an internal Inline Reference plus a Derive. That works for a
single-valued reference and breaks for a multi-valued one: three referents with two
properties each become two parallel arrays with nothing pairing them, and the
consumer has to guess that index n of one lines up with index n of the other.
Deployments hit this the moment a record has more than one of anything interesting.

Solution

Make the API device real end to end, so a Reference Type declared inline with
output is stored, reconstructed and served as a nested object carrying its own
Search Fields.

A Deployment then declares the nesting once in its Search Schema and receives it in
every surface, with multi-valued references keeping their values grouped per
referent instead of smeared across parallel arrays.

User Stories

1. As a Deployment, I want a Reference Type declared inline with output to reach
   the API as a nested object, so that the API device documented in ADR 11 actually
   works and I do not have to flatten referents onto the root.
2. As a Deployment, I want a multi-valued inline reference to keep each referent’s
   values grouped together, so that consumers never have to pair parallel arrays by
   index.
3. As an API consumer, I want to select a nested object’s fields directly, so that
   the response shape matches the data’s shape and I can render one referent at a
   time.
4. As a Deployment, I want nested reconstruction to happen below the surface, so that
   a second surface added later inherits it without reimplementation.
5. As a Deployment, I want an Internal Field inside a Reference Type to stay pruned,
   so that the reading device keeps working exactly as it does today.
6. As a Deployment, I want a nested declaration that the engine cannot serve to fail
   at startup rather than per query, so that misconfiguration surfaces at boot like
   every other schema invariant.

Implementation Decisions

Modules built or modified

• Collection definition (engine adapter). A reference Search Field with an
  output Inline Reference currently maps to string / string[]. It must instead
  fan the Reference Type’s own Search Fields into nested Physical Fields. The naming
  convention stays owned by physicalFields() so projection, collection definition
  and the query compiler cannot disagree – the same rule that already governs the
  flat case.
• Document reconstruction (engine adapter). Reference reconstruction currently
  yields an id plus an optional label. It must yield a nested Search Document built
  from the Reference Type’s declaration when the reference is inline and output.
• Logical value model. SearchValue has no object member today; the only
  object-shaped value is a reference of id plus label. It gains a nested Search
  Document member, and the Reference shape stays as-is for idOnly and
  labelOnly.
• GraphQL surface. Reference GraphQL types are currently hardcoded to id and a
  label field. For an inline output reference the type is built from the Reference
  Type’s output Search Fields, reusing the same per-kind field configuration the
  root types already use.

Placement

Nested reconstruction lands at the engine-adapter and Search Document level, not
in the GraphQL surface. Building nested types only in GraphQL is the smaller change
and the wrong one: a second surface would have to reimplement it. The surface’s job
stays serialisation of a nested value that already exists.

Scope of Roles inside a Reference Type

Only output (and the existing role-less Internal Field behaviour) is in scope.
Filtering, faceting and sorting on a nested field require query-compiler and engine
support that is a separate decision; a declaration asking for them fails schema
validation with a clear message rather than being silently ignored.

Referent identity

Open question that must be answered before implementation, because it may bound what
is achievable: whether a referent with no stable identity survives framing into a
nested document. A Deployment has reported that named-node referents survive inline
framing where blank-node referents do not. Nesting needs the referent’s fields
rather than its id, so this may already work – but it must be verified against the
framing and projection code, with a fixture covering blank-node referents, before the
nested shape is promised to anyone. If blank-node referents cannot be framed, that
constraint belongs in CONTEXT.md next to the Inline Reference definition.

Validation

searchSchema() keeps being the only constructor and gains the invariants the nested
shape needs, so a dangling or unserviceable nesting fails at startup. This follows the
precedent set by label sources, which are validated schema-wide for the same reason.

Breaking change

Engine adapters and any consumer reading reference values are affected. Mark the
commit ! per the workspace convention.

Testing Decisions

A good test here states an externally observable fact about the model: given a Search
Schema and a set of source quads, this is the collection definition, this is the
reconstructed Search Document, this is the GraphQL response. It never asserts on
intermediate structures or on how many physical fields were emitted for their own
sake.

• Contract suite (@lde/search/testing). The executable port contract gains
  coverage for storing and reconstructing a nested inline reference, so every engine
  implementation is held to the same behaviour. Prior art: the suite already covers
  the port methods added by earlier ADRs.
• Collection definition. Nested Physical Field naming for single- and multi-valued
  inline references, and that an Internal Field inside a Reference Type contributes
  nothing. Prior art: the existing per-kind collection-definition tests.
• Document reconstruction. A multi-valued inline reference reconstructs as a list
  of nested documents with each referent’s values grouped, including the case where
  one referent is missing an optional field.
• Framing with unidentified referents. A fixture where referents are blank nodes,
  asserting whatever the answer to the identity question turns out to be.
• GraphQL surface. The generated type carries the Reference Type’s output fields,
  and a schema requesting an unsupported Role on a nested field fails to build with a
  message naming the field. Prior art: the existing surface-building tests.

Out of Scope

• Filtering, faceting and sorting on nested fields.
• Any second API surface. This work makes one possible; it does not add one.
• Reference strategies other than inline. idOnly and labelOnly keep their
  current shape and their current reconstruction path.
• Arbitrary nesting depth beyond what the schema’s existing inline framing depth
  already allows.
• Any domain vocabulary. This is a capability of the search model; no type, field or
  format name from any Deployment belongs in it.

Further Notes

This is a prerequisite for per-record enrichment work that wants to attach several
computed values to each of a record’s referents. It is worth doing on its own merits:
it closes a gap between what CONTEXT.md and ADR 11 promise and what the code does,
and every Deployment with a multi-valued reference is currently paying for that gap in
parallel arrays.

The referent-identity question is the one real risk to sizing. Answer it first.

[ddeboer]

Split out of #654. The first known consumer is
netwerk-digitaal-erfgoed/stack#1, which stays entirely outside LDE – no image or IIIF
vocabulary belongs in any @lde/* package.

[ddeboer]

Settled the referent-identity question from Implementation Decisions by running it:
referents without an @id are dropped, and named referents nest correctly.

Same graph, twice, through projectRoots – so through framing, not a hand-built framed
node. Named referents:

[
  { "id": "https://ex/m/1", "contentUrl": ["…/1/full/max/0/default.jpg"], "width": 4096 },
  { "id": "https://ex/m/2", "contentUrl": ["…/2/full/max/0/default.jpg"], "width": 800 }
]

The identical graph with _:m1 / _:m2 yields no key on the document at all. The gate
is one line in applyInlineReference:

const referents = valuesOf(node, alias)
  .filter(isObject)
  .filter((referent) => typeof referent['@id'] === 'string')

So this is not a framing limitation and not the root-selection rule (that one is
separate and correct – a blank node has no stable document key, so it can never be a
root). It is a filter on the nesting path, and nesting does not need identity: a
surfaced nested object is read, not addressed.

Bringing this into scope:

• Nest a referent that has no @id, omitting the id key from the nested Search
  Document rather than dropping the referent.
• Keep the filter for idOnly and labelOnly, where the IRI is the value and a
  referent without one genuinely has nothing to contribute.
• Test both directions: a mix of named and unnamed referents under one field nests all
  of them, with id present only on the named ones.

This matters beyond neatness. Heritage MediaObjects are frequently blank nodes – it is
why Linked Open Limburg flattens contentUrl and thumbnailUrl onto the work as
parallel keyword arrays – so without it the multi-valued case this issue exists to fix
stays broken for the shape that most needs it.

[ddeboer]

The referent-identity question is answered in #677: blank-node referents do not survive framing today – JSON-LD 1.1 prunes the singly-used @id, and applyInlineReference filters on it – so an inline reference carries named nodes only. #677 has the reproduction and the fix options; worth settling before this is sized.

[ddeboer]

The referent-identity blocker is resolved and released: #687, in @lde/search
0.14.0.

Inline referents no longer need an @id – nesting carries the referent’s fields,
not a document key – so a blank-node MediaObject nests exactly like a named
one, minus the id. Roots stay keyed, and a blank node label never becomes one.
derive now receives a ProjectedNode, which is the breaking part.

What that unblocks and what it does not: the projection now produces the
nested shape, but nothing downstream stores or serves it. The rest of this issue
stands unchanged –

• nested Physical Fields in the collection definition, which still maps a
  reference to string / string[];
• nested reconstruction in the engine adapter, which still rebuilds a reference
  as { id } or { id, label };
• nested GraphQL types, which are still hardcoded to id and a label field.

The placement decision is the one to hold on to: reconstruction belongs at the
engine-adapter and Search Document level, not in the GraphQL surface, so a second
surface inherits it rather than reimplementing it.