Skip to content

@lde/search-pipeline: searchStages, retire projectGraph and the buffering writer #623

Description

@ddeboer

Compose per-type projecting stages in @lde/search-pipeline, retire projectGraph and the buffering searchIndexWriter, and prove the memory bound. The scalability landingADR 13. Depends on Stage<Out> and projectRoots.

Scope

  • searchStages(options): Stage<TypedSearchDocument>[] — one stage per root type. Sketch:
    { schema, types: [{ searchType, rootVariable, itemSelector, readers, batchSize? }], writerFor }
    Each stage's project closure: projectRoots(quads, bindings.map(b => b[rootVariable].value), schema, searchType), tagged { searchType, document }. rootVariable couples the selector's projected variable to what the projection reads — settle this interface here, against a working Stage<Out>.
    • selectByClass(searchType) helper: ?x a <class> — one word for the object grain, where class is the source class. Not a default (three of DR's four catalog types have no source class).
  • Move TypedSearchDocument from @lde/search to here: it exists only because a terminal routes, which is glue, not projection.
  • searchIndexWriterWriter<TypedSearchDocument>: keeps ADR 9's per-collection fan-out (route by searchType.class to per-type runs) and the run lifecycle; stops projecting and stops buffering.
  • Delete projectGraph, rootsByType, frameByType from @lde/search.

Acceptance

  • Memory flat by counting (ADR 12): run a synthetic input at 10 and 10,000 roots, batchSize: 10, against a recording fake Writer; assert project invoked ceil(roots/batchSize) times, and peak live document count identical at both sizes. Fails against today's searchIndexWriter; passes here.
  • No behavioural regression: the DR-shaped multi-collection integration test (Typesense testcontainer) — same documents land in the same collections.
  • projectGraph gone from the workspace.

Design: ADR 13. Parent: #606.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions