Problem Statement
A Deployment can intercept a pipeline’s quad stream, but it cannot enrich a projected
Search Document.
ADR 2 unified
every pipeline extension on the quad transform, and the stage documents the position
plainly: a quad enrichment is a transform attached to a reader, which runs before the
one type-changing seam. That is the right shape for anything expressible on the
graph, and the wrong shape for enrichment that is inherently per-record:
- A reader-attached transform only ever sees its own reader’s output. It cannot
read what sibling readers produced, so to act on a record it must fire its own query
against the endpoint for data the stage has already read – a second round trip per
batch for values that are in hand.
- Some enrichment can only be decided after projection, because what to do depends
on what the source did or did not supply. A transform that runs before projection
cannot see that the projection came up empty.
- An enrichment that draws on something outside the graph – a store, a service, a
file – has no natural expression as a quad transform at all. It has to mint IR
Aliases by hand to be visible to the projection, and CONTEXT.md forbids exactly
that: IR Aliases are mechanical, per field, never hand-written, precisely so the
extraction cannot drift from the schema.
The result is that a Deployment with a per-record enrichment has no supported place to
put it.
Separately, and for the same class of work: the workspace already contains a
host-limited concurrent map – global and per-host caps with no head-of-line blocking –
and it is private to one package, so the next consumer either re-implements it or
does without and hammers a single host.
Solution
Add a second extension point to @lde/pipeline: a transform over a stage’s projected
items, running after projection and before the writer. It receives what the projection
produced and returns the same type, so it can fill, correct or annotate a record using
anything available to it.
Promote the host-limited concurrent map into a shared module so any package doing
bounded outbound work can use it.
User Stories
- As a Deployment, I want to transform a projected item before the writer sees it, so
that I can enrich a record with values that are not in the source graph.
- As a Deployment, I want the enrichment to see the whole projected record, so that I
can decide what to do based on what the projection actually produced.
- As a Deployment, I want to enrich without minting IR Aliases by hand, so that the
Search Schema stays the single source of truth for extraction.
- As a Deployment, I want a failing enrichment to degrade the item rather than abort
the run, so that one bad record does not cost me an index.
- As a Deployment, I want per-item enrichment failures reported, so that I can decide
my own policy on how much degradation is acceptable.
- As a pipeline operator, I want enrichment to stay inside the batch’s memory bound,
so that a stage’s footprint is still a function of the unit of work and not the
input.
- As a Deployment that composes prebuilt images, I want to point the indexer at an
enrichment service through configuration, so that I can enrich without building my
own image.
- As a Deployment that builds its own image, I want to implement the enrichment port in
process, so that I do not pay a network hop for enrichment I already own.
- As a pipeline operator, I want an enricher to be able to add fields but never alter
or remove projected ones, so that a third-party implementation cannot corrupt a
record.
- As a package author, I want a shared host-limited concurrent map, so that outbound
work is bounded globally and per host without me re-implementing the scheduler.
Implementation Decisions
Modules built or modified
@lde/pipeline stage. A new optional transform over the stage’s output items,
applied after the existing type-changing seam and before the item reaches the
writer. Signature is item-to-item, so the stage’s output type is unchanged and the
writer contract is untouched. Applied within the batch, so the number of items in
flight stays bounded by the existing batch size and queue capacity, per ADR
12 and ADR
13.@lde/search-pipeline. The new seam is exposed as an option on the search stages
and threaded through the convenience pipeline wiring, so a Deployment can attach one
per Search Type without hand-assembling stages.- Shared concurrency module. The host-limited concurrent map moves out of the
package that currently owns it privately, keeping its behaviour: a global cap and a
per-host cap, no head-of-line blocking (a saturated host’s item yields to a later
item on another host), results in input order, and the requirement that the task
never rejects. Its current consumer switches to the shared module.
An enrichment port, with an HTTP adapter
The seam alone forces every Deployment to build its own indexer image to use it, which
one known Deployment cannot do – it composes prebuilt images and a schema module, with a
four-line Dockerfile. So alongside the seam, @lde/search-pipeline defines a port for
enriching projected documents, and ships an HTTP adapter for it. The indexer
instantiates the adapter from environment configuration and leaves it absent when unset,
exactly as it already does for its other optional components.
This is the workspace’s established shape – a narrow interface with swappable backends,
as the provenance store, the engine, readers, writers and the task runner all are – and
it means the deployment style is the Deployment’s choice rather than the pipeline’s: a
consumer that builds its own image implements the port in process with no network hop; a
consumer that composes prebuilt images points the adapter at a container.
Two properties of the contract, both load-bearing once the implementation may be
out-of-process:
- Batched. One call per batch, so the port inherits the batch’s memory bound rather
than introducing one of its own.
- Documents in, documents out. The same shape as every other extension in the
pipeline – ADR 2’s quad transform, and this issue’s own item seam. An enricher is a
component the operator configured, at the same trust level as the reader, the writer
and the engine, so the contract does not police what it may change; constraining it to
additive patches would also foreclose legitimate uses such as normalising or
correcting a projected value.
- Integrity-checked, not constrained. The returned batch must carry the same document
identities, and as many of them as went in. That catches a broken or misconfigured
enricher without narrowing what a working one is allowed to do.
Failure is per batch and non-fatal, consistent with the seam’s outcome reporting: a
timeout, a transport error, a malformed response or a failed integrity check leaves the
batch as projection produced it and is reported.
LDE learns only that a Deployment may point it at an enricher. It never learns what any
enricher does.
Boundary between the two extension points
This needs an ADR amending ADR 2, because ADR 2’s thesis was that every extension is
the same operation. The boundary to state: a quad transform is for anything
expressible over the graph, and remains the default; the item seam is only for
enrichment that requires the projected record – because it depends on what projection
produced, or because its source is not a graph. Without that boundary written down,
the item seam becomes the place everything lands.
Failure policy
The seam reports per-item outcomes rather than deciding what they mean. An enrichment
that fails leaves the item as projection produced it and records the failure; whether
that is acceptable, and at what rate, is the Deployment’s call. This mirrors how batch
facet queries already report per-query outcomes instead of failing a whole batch, and
how distribution health degrades an item while reporting the aggregate.
Ordering
The seam runs after the type-changing step and after any Internal Field pruning that
the writer relies on, so an enrichment sees the same record the writer would have
received. If an enrichment needs a value that pruning removes, that is a schema
question for the Deployment, not a reason to reorder.
Testing Decisions
A good test here asserts what a stage produces for a given input, not how the runner
sequenced its internals. The seam is observable entirely through the items the writer
receives and the outcomes reported, so tests assert on those.
- Stage seam. An attached item transform is applied to every projected item; the
writer receives transformed items; a transform that throws leaves that item
untransformed, reports the failure, and does not stop the run or affect its
siblings. Prior art: the existing stage tests covering readers, attached quad
transforms and the projection seam.
- Bounding. With a small batch size and queue capacity, the number of items
concurrently in the seam stays within the configured bound. Prior art: the existing
batching and queue-capacity tests.
- Search stages wiring. An enrichment attached per Search Type reaches only that
type’s items. Prior art: the existing per-root-type stage tests.
- Enrichment port contract. An executable contract suite, as
@lde/search/testing
provides for engines, so every implementation is held to the same behaviour: enriched
documents reach the writer; a returned batch missing a document, carrying an unknown
one, or of the wrong size fails the integrity check and leaves the batch as projected;
an implementation that returns its input unchanged is a no-op.
- HTTP adapter. One request per batch; a timeout, a non-2xx response and a malformed
body each leave the batch untouched and report; the request carries the batch’s
documents and nothing else. Tested against a stub server, never a live one.
- Absence. With no enricher configured, the stage behaves exactly as it does today –
the adapter is optional and nothing about the default path changes.
- Host-limited concurrent map. Global and per-host caps hold under a mix of hosts;
a saturated host does not block items for other hosts; results come back in input
order; non-HTTP-style keys get their own budget rather than sharing one. Prior art:
the tests that exist today for it in its current home, moved with it.
Out of Scope
- What any Deployment enriches, and where it enriches from. This issue adds a seam, a
port, one adapter and a scheduler; it adds no source, no store, and no vocabulary.
- Any transport beyond HTTP. Further adapters are additive and belong to whoever needs
them; the port exists so that the HTTP one is a choice rather than the mechanism.
- Requiring an enricher at all. With none configured the pipeline is unchanged.
- Persistence of any kind. The seam holds nothing between runs.
- Retry and backoff policy. A Deployment that wants them composes them inside its own
transform; the pipeline does not impose a policy.
- Query-time or read-side enrichment. This is a write-side seam.
- Replacing or deprecating quad transforms. They stay the default extension point.
Further Notes
The seam is the smaller half of this issue and the ADR is the load-bearing part:
ADR 2’s unification is a real constraint that has served the pipeline well, and
amending it deserves an explicit boundary rather than an exception.
Extracting the host limiter is independent and can land first. It is worth doing even
if the seam is rejected.
Problem Statement
A Deployment can intercept a pipeline’s quad stream, but it cannot enrich a projected
Search Document.
ADR 2 unified
every pipeline extension on the quad transform, and the stage documents the position
plainly: a quad enrichment is a transform attached to a reader, which runs before the
one type-changing seam. That is the right shape for anything expressible on the
graph, and the wrong shape for enrichment that is inherently per-record:
read what sibling readers produced, so to act on a record it must fire its own query
against the endpoint for data the stage has already read – a second round trip per
batch for values that are in hand.
on what the source did or did not supply. A transform that runs before projection
cannot see that the projection came up empty.
file – has no natural expression as a quad transform at all. It has to mint IR
Aliases by hand to be visible to the projection, and
CONTEXT.mdforbids exactlythat: IR Aliases are mechanical, per field, never hand-written, precisely so the
extraction cannot drift from the schema.
The result is that a Deployment with a per-record enrichment has no supported place to
put it.
Separately, and for the same class of work: the workspace already contains a
host-limited concurrent map – global and per-host caps with no head-of-line blocking –
and it is private to one package, so the next consumer either re-implements it or
does without and hammers a single host.
Solution
Add a second extension point to
@lde/pipeline: a transform over a stage’s projecteditems, running after projection and before the writer. It receives what the projection
produced and returns the same type, so it can fill, correct or annotate a record using
anything available to it.
Promote the host-limited concurrent map into a shared module so any package doing
bounded outbound work can use it.
User Stories
that I can enrich a record with values that are not in the source graph.
can decide what to do based on what the projection actually produced.
Search Schema stays the single source of truth for extraction.
the run, so that one bad record does not cost me an index.
my own policy on how much degradation is acceptable.
so that a stage’s footprint is still a function of the unit of work and not the
input.
enrichment service through configuration, so that I can enrich without building my
own image.
process, so that I do not pay a network hop for enrichment I already own.
or remove projected ones, so that a third-party implementation cannot corrupt a
record.
work is bounded globally and per host without me re-implementing the scheduler.
Implementation Decisions
Modules built or modified
@lde/pipelinestage. A new optional transform over the stage’s output items,applied after the existing type-changing seam and before the item reaches the
writer. Signature is item-to-item, so the stage’s output type is unchanged and the
writer contract is untouched. Applied within the batch, so the number of items in
flight stays bounded by the existing batch size and queue capacity, per ADR
12 and ADR
13.
@lde/search-pipeline. The new seam is exposed as an option on the search stagesand threaded through the convenience pipeline wiring, so a Deployment can attach one
per Search Type without hand-assembling stages.
package that currently owns it privately, keeping its behaviour: a global cap and a
per-host cap, no head-of-line blocking (a saturated host’s item yields to a later
item on another host), results in input order, and the requirement that the task
never rejects. Its current consumer switches to the shared module.
An enrichment port, with an HTTP adapter
The seam alone forces every Deployment to build its own indexer image to use it, which
one known Deployment cannot do – it composes prebuilt images and a schema module, with a
four-line Dockerfile. So alongside the seam,
@lde/search-pipelinedefines a port forenriching projected documents, and ships an HTTP adapter for it. The indexer
instantiates the adapter from environment configuration and leaves it absent when unset,
exactly as it already does for its other optional components.
This is the workspace’s established shape – a narrow interface with swappable backends,
as the provenance store, the engine, readers, writers and the task runner all are – and
it means the deployment style is the Deployment’s choice rather than the pipeline’s: a
consumer that builds its own image implements the port in process with no network hop; a
consumer that composes prebuilt images points the adapter at a container.
Two properties of the contract, both load-bearing once the implementation may be
out-of-process:
than introducing one of its own.
pipeline – ADR 2’s quad transform, and this issue’s own item seam. An enricher is a
component the operator configured, at the same trust level as the reader, the writer
and the engine, so the contract does not police what it may change; constraining it to
additive patches would also foreclose legitimate uses such as normalising or
correcting a projected value.
identities, and as many of them as went in. That catches a broken or misconfigured
enricher without narrowing what a working one is allowed to do.
Failure is per batch and non-fatal, consistent with the seam’s outcome reporting: a
timeout, a transport error, a malformed response or a failed integrity check leaves the
batch as projection produced it and is reported.
LDE learns only that a Deployment may point it at an enricher. It never learns what any
enricher does.
Boundary between the two extension points
This needs an ADR amending ADR 2, because ADR 2’s thesis was that every extension is
the same operation. The boundary to state: a quad transform is for anything
expressible over the graph, and remains the default; the item seam is only for
enrichment that requires the projected record – because it depends on what projection
produced, or because its source is not a graph. Without that boundary written down,
the item seam becomes the place everything lands.
Failure policy
The seam reports per-item outcomes rather than deciding what they mean. An enrichment
that fails leaves the item as projection produced it and records the failure; whether
that is acceptable, and at what rate, is the Deployment’s call. This mirrors how batch
facet queries already report per-query outcomes instead of failing a whole batch, and
how distribution health degrades an item while reporting the aggregate.
Ordering
The seam runs after the type-changing step and after any Internal Field pruning that
the writer relies on, so an enrichment sees the same record the writer would have
received. If an enrichment needs a value that pruning removes, that is a schema
question for the Deployment, not a reason to reorder.
Testing Decisions
A good test here asserts what a stage produces for a given input, not how the runner
sequenced its internals. The seam is observable entirely through the items the writer
receives and the outcomes reported, so tests assert on those.
writer receives transformed items; a transform that throws leaves that item
untransformed, reports the failure, and does not stop the run or affect its
siblings. Prior art: the existing stage tests covering readers, attached quad
transforms and the projection seam.
concurrently in the seam stays within the configured bound. Prior art: the existing
batching and queue-capacity tests.
type’s items. Prior art: the existing per-root-type stage tests.
@lde/search/testingprovides for engines, so every implementation is held to the same behaviour: enriched
documents reach the writer; a returned batch missing a document, carrying an unknown
one, or of the wrong size fails the integrity check and leaves the batch as projected;
an implementation that returns its input unchanged is a no-op.
body each leave the batch untouched and report; the request carries the batch’s
documents and nothing else. Tested against a stub server, never a live one.
the adapter is optional and nothing about the default path changes.
a saturated host does not block items for other hosts; results come back in input
order; non-HTTP-style keys get their own budget rather than sharing one. Prior art:
the tests that exist today for it in its current home, moved with it.
Out of Scope
port, one adapter and a scheduler; it adds no source, no store, and no vocabulary.
them; the port exists so that the HTTP one is a choice rather than the mechanism.
transform; the pipeline does not impose a policy.
Further Notes
The seam is the smaller half of this issue and the ADR is the load-bearing part:
ADR 2’s unification is a real constraint that has served the pipeline well, and
amending it deserves an explicit boundary rather than an exception.
Extracting the host limiter is independent and can land first. It is worth doing even
if the seam is rejected.