feat(search-indexer): ship the search indexer as a CLI and Docker image#655
Merged
Conversation
- loadSchemaModule() loads and validates a mounted schema-declaration module (an .mjs default-exporting SearchType declarations as plain data) and returns the validated schema plus the raw exports, so each consumer validates its own optional exports - Node-only, so it lives in a separate entry point like ./adapter and ./testing, keeping the main entry point runtime-agnostic - The one loader both the indexer image and the served-API image boot from, so the write and the read side cannot disagree about the schema
…odule - The generic load-and-validate logic moves to the shared loader; what stays here is validating the read side's optional exports (schemaOptions, engineOptions) - Add a test for the non-object optional export failure mode
- New @lde/search-indexer package: a Commander CLI wrapping searchIndexerPipeline, configured entirely from environment variables (boot-or-fail, all problems reported in one error) - Mounts the same schema-declaration module as the search-api-server image, loaded with the shared @lde/search/module loader - Dataset selection against a DCAT registry: explicit IRIs (DATASETS), search criteria (DATASET_CRITERIA), or the whole registry - Typesense rebuild writers per root type: in-place (default) or blue-green, with an optional derived-collection-name prefix - Optional file provenance (PROVENANCE_FILE + PIPELINE_VERSION); rejected with blue-green rebuilds, which would drop skipped datasets - Optional QLever import path behind QLEVER_IMAGE: data dumps are imported into pipeline-controlled sibling containers over the Docker socket (see ADR 16 for why a static QLever service cannot work) - --check validates configuration and schema module, then exits; the Docker smoke test boots the image with it to catch pruned-image module-resolution failures in CI - The docker.yml release workflow now derives the package from the tag, serving both search-api-server and search-indexer images
ddeboer
force-pushed
the
feat/search-indexer
branch
from
July 24, 2026 12:35
6cdfec3 to
ac70566
Compare
ddeboer
added a commit
that referenced
this pull request
Jul 24, 2026
…nd Docker image (#646) * feat(search): reject an unknown field kind at declaration time - validateSearchType silently accepted a kind outside the FieldKind union: every kind-dependent rule passed vacuously, deferring the failure to whatever consumer read the declaration first - guards the plain-JS declaration path (a mounted schema module, a SHACL generator) that the runtime validation exists for - adds the 'unknown-kind' issue reason * feat(search-api-server): serve the search API as a bootable process and Docker image - new composition package binding the engine-agnostic search-api-graphql handler to a Typesense engine: environment config, a mounted plain-data schema-declaration module (validated at boot), and a node:http server answering /graphql, /health and a root redirect - the search-api-server bin boots from environment variables alone and reports every configuration problem in one error - the image is built from the workspace's own outputs, never from npm: docker:build (inferred by @nx/docker) stages the compiled package, the same-commit builds of its @lde/* dependencies (copy-workspace-modules) and a pruned lockfile (prune-lockfile plus an npm --package-lock-only repair step for the Nx pruning fallback); the Dockerfile is npm ci + COPY - docker:smoke boots each affected image in CI with a fixture schema and probes /health and the SDL over HTTP, so runtime-only failures (the ERR_MODULE_NOT_FOUND class) cannot pass a green build - each release tag rebuilds, smoke-tests and pushes ghcr.io/ldelements/search-api-server:<version>, independent of the npm publish - the read-only image omits the write-side peers (@lde/pipeline, @lde/dataset): every runtime import is 'import type' - recorded as ADR 15 * feat(search-indexer): ship the search indexer as a CLI and Docker image (#655) * feat(search): add the schema-module loader as @lde/search/module - loadSchemaModule() loads and validates a mounted schema-declaration module (an .mjs default-exporting SearchType declarations as plain data) and returns the validated schema plus the raw exports, so each consumer validates its own optional exports - Node-only, so it lives in a separate entry point like ./adapter and ./testing, keeping the main entry point runtime-agnostic - The one loader both the indexer image and the served-API image boot from, so the write and the read side cannot disagree about the schema * refactor(search-api-server): load the schema module via @lde/search/module - The generic load-and-validate logic moves to the shared loader; what stays here is validating the read side's optional exports (schemaOptions, engineOptions) - Add a test for the non-object optional export failure mode * feat(search-indexer): ship the search indexer as a CLI and Docker image - New @lde/search-indexer package: a Commander CLI wrapping searchIndexerPipeline, configured entirely from environment variables (boot-or-fail, all problems reported in one error) - Mounts the same schema-declaration module as the search-api-server image, loaded with the shared @lde/search/module loader - Dataset selection against a DCAT registry: explicit IRIs (DATASETS), search criteria (DATASET_CRITERIA), or the whole registry - Typesense rebuild writers per root type: in-place (default) or blue-green, with an optional derived-collection-name prefix - Optional file provenance (PROVENANCE_FILE + PIPELINE_VERSION); rejected with blue-green rebuilds, which would drop skipped datasets - Optional QLever import path behind QLEVER_IMAGE: data dumps are imported into pipeline-controlled sibling containers over the Docker socket (see ADR 16 for why a static QLever service cannot work) - --check validates configuration and schema module, then exits; the Docker smoke test boots the image with it to catch pruned-image module-resolution failures in CI - The docker.yml release workflow now derives the package from the tag, serving both search-api-server and search-indexer images
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #652.
Stacked on #646 (
feat/search-graphql-docker): it reuses that PR’s whole delivery pattern (ADR 15 image staging,docker:smokeCI gate,docker.yml) and its schema-module contract, so it targets that branch – three commits on top of it. Retarget tomainonce #646 lands.What
@lde/search-indexer: the write-path counterpart of@lde/search-api-server– the object-grain search indexer as a Commander CLI, npm package and prebuilt Docker image (ghcr.io/ldelements/search-indexer), configured entirely from environment variables. Mount the same schema-declaration module into this image and the API-server image, point both at the sameTYPESENSE_*coordinates, and the write and read sides cannot disagree about the schema.feat(search)): the schema-module load-and-validate logic moves fromsearch-api-serverinto a new@lde/search/moduleentry point; both composition packages boot from it. The server keeps validating its read-side optional exports (schemaOptions,engineOptions), with a new test for the non-object export failure mode.DATASETS) or registry criteria (DATASET_CRITERIA, JSON) →RegistrySelector;TYPESENSE_*(admin key);REBUILD_MODEin-place(default) /blue-greenwith optionalCOLLECTION_PREFIX;PROVENANCE_FILE+PIPELINE_VERSION→FileProvenanceStore;ConsoleReporterby default. Two combinations are rejected at boot: provenance with only one of its halves, and provenance with blue-green rebuilds (a skipped dataset would be missing from the fresh collection the swap makes live).QLEVER_IMAGEenablesImportResolver+DockerTaskRunner– the indexer spawns and fully controls sibling QLever containers over the mounted Docker socket (dockerode, no docker CLI in the image). Default stays endpoint-only, which runs anywhere. Native mode in-image is the documented Kubernetes follow-up; the statically-declared-sidecar option is rejected in the ADR (no control plane for the per-dataset index/restart loop).--checkvalidates configuration and schema module then exits – used bytools/docker-smoke-indexer.shto boot the built image in CI, so pruned-image module-resolution failures (ERR_MODULE_NOT_FOUND) fail the PR. The image needsnpm ci --legacy-peer-depsfor the same reason as the server image: thedocker:lockfilerepair step installs with that flag, so peers it leaves out (typesense’s@babel/runtime) would otherwise failnpm ci.docker.ymlgeneralized: the workflow now derives package name and version from the release tag (keeping the env-based, injection-safe ref handling) and serves both@lde/search-api-server@*and@lde/search-indexer@*.Out of scope (per #652)
Bespoke root selectors and per-stage tuning (library path), SHACL as schema source (#495), engines other than Typesense, native-mode QLever (follow-up).
After merge
First publish needs the manual Trusted-Publishing bootstrap, like every new
@lde/*package.