fix(retriever): rebuild a missing vector store engine on demand

[ochanism]

Closes #2356

Overview

The engine registry is per-process, so a store's engine is missing from any process that did not build it. That happens three ways: startup skips a store whose engine fails to construct and never retries it, a store registered on one instance is absent on every other, and registration can fail on the instance handling the create. Each one answers vector store is currently unavailable (2201) for as long as the process lives, and the first is not fixed by restarting.

GetOrLoadByStoreID falls back to building the engine from the store row when the lookup misses, and the two bound-store lookups now go through it. Everything it needs was already at hand: the ownership check one line earlier reads the same row, and the engine factory is already in the container.

Why

The lookup gave up while holding everything required to recover. The row had just been read to verify ownership, the factory that builds engines from rows was already wired, and the failure was reported as if the store did not exist. The codebase names the consequence in three places — "will be available after restart", "sibling replicas continue serving the engine from their own caches until process restart", and a test comment describing a row whose "engine initialization failed at app start" — so the behaviour was understood, just never closed.

The method goes on RetrieveEngineRegistry rather than StoreRegistry, because that is the interface the factory functions receive. The two declare GetByStoreID separately and neither embeds the other, so putting it on StoreRegistry does not compile at the call site.

Changes

internal/types/interfaces/retriever.go (+15): GetOrLoadByStoreID on RetrieveEngineRegistry, documented as tenant-scoping its own database lookup while still expecting callers to verify ownership first.

internal/application/service/retriever/registry.go (+254/-2):

• GetOrLoadByStoreID — miss → load the row → build → publish. Concurrent callers collapse onto one build via singleflight.
• The shared build runs on context.WithoutCancel. Callers share its result, so letting the first one's cancellation abort it would fail everyone else waiting on it.
• A panic inside the build is contained. singleflight re-raises it on a goroutine of its own (go panic(e); select{}), out of reach of the HTTP recovery middleware, and the build calls third-party client constructors — so without this an availability fix could end the process.
• Publishing is guarded by a per-store generation sampled before the build. RegisterWithStoreID and UnregisterByStoreID both bump it, so a build that finishes late cannot undo a registration or a removal, or leave the engine it replaced orphaned with its connections open.
• A failed build starts a cooldown. Collapsing only helps concurrent callers; without a cooldown a backend that stays down costs a full build timeout on every sequential request, in exactly the outage this change is meant to soften.
• Failures are logged where they happen, with the panic stack. The cause names an endpoint and cannot travel back to the caller, so previously it was discarded with the error — leaving an operator nothing when self-healing failed to heal.
• EngineBuildTimeout is exported so a caller budgeting a sequence of resolutions can size its ceiling above one build instead of guessing.

internal/application/service/retriever/factory.go (+56/-10):

• Both bound-store lookups route through GetOrLoadByStoreID.
• ErrVectorStoreUnavailable — "the store exists but its engine cannot be produced right now" — separated from ErrVectorStoreNotFound. The async delete handlers answer not-found with asynq.SkipRetry, so reporting a database blip or a backend outage that way permanently discarded knowledge-base and index delete tasks and orphaned the vector data they exist to remove. The repository already distinguishes the two, returning (nil, nil) for a missing row and an error for a failure.
• A caller's own cancellation is never folded into a store sentinel. The ownership check runs first against the same context, so on a shutdown that is usually where cancellation is seen.

internal/application/service/knowledgebase_search_storegroup.go (+27/-1): a budget over the resolution loop. A search resolves one engine per distinct store, sequentially, and this server sets no HTTP read or write timeout, so nothing bounded the total. Only resolution runs under it — folding the embedding calls into the same ceiling would cut them short for an unrelated reason — and it derives from the caller's context so a client that allows less time still wins. An exhausted budget is reported as an unavailable store rather than an internal error, since the build is detached and keeps warming, so a retry is likely to succeed.

internal/application/service/knowledgebase.go (+15/-2), tag.go (+2): the KB delete handler returns on the retryable sentinel instead of logging and continuing, which would have reported success while leaving the embeddings in place. validateVectorStoreBinding maps the new sentinel to 2201 and lets a context error through rather than answering a client disconnect with a 500.

internal/container/container.go (+5/-1): the registry is given the repository and the factory. Both are constructor arguments rather than an optional extra — a registry without them satisfies the interface and serves every lookup, so leaving them out would go unnoticed until a store went missing in production, which is the failure this change removes.

Behaviour change worth calling out

An ownership lookup failure was reported as ErrVectorStoreNotFound, and TestCreateRetrieveEngineForKB_OwnershipLookupError asserted that deliberately. A database failure says nothing about whether the store exists, and treating it as permanent is what discards the task, so that assertion is inverted rather than preserved.

Core

1. GetOrLoadByStoreID in registry.go — the collapse, the detached build context, the panic guard, and the generation check are each load-bearing for a different failure; the tests below pin them individually.
2. The sentinel split in factory.go — this is what decides whether async work is retried or dropped.

Test Plan

• go build ./..., go vet ./internal/..., gofmt clean on changed files, and no added line over 120 columns
• go test -race ./internal/... — pass. Four suites fail identically on main and are untouched here: TestTenantAPIKeyServiceAuthenticateThrottlesLastUsedUpdates (data race), internal/datasource/connector/notion, TestEventBus_EmitAndWait (data race), and internal/infrastructure/docparser TestResolveRemoteImages_* under -count>1.
• New tests, each verified to fail when the mechanism it names is removed: waiters survive a cancelled leader (fails with singleflight.Do, and separately when collapsing is removed); concurrent misses build once; a panicking factory fails one request and not the process; an unregistration or a concurrent registration during a build is not overwritten; a failed build enters a cooldown and leaves it; a database failure is retryable while an absent store is not; a cancellation is not reported as a store verdict on either lookup path; the resolution budget applies and never outlives the caller; the container builds a registry that can actually rebuild.
• Two instances against one database, no restart anywhere: an OpenSearch store created through A, then a knowledge base bound to it requested through B. On main B answers 2201 on three consecutive attempts; with this change B succeeds. Taking OpenSearch down mid-test shows the rebuild genuinely dials it and logs the transport error it used to swallow, a request inside the cooldown window spends no further attempt, and bringing OpenSearch back lets the same request succeed without restarting anything.

No new dependencies — golang.org/x/sync is already required directly.