ADR 0005 rag store

ADR-0005: RAG store — bundled Well-Architected reference (no managed vector DB)

Status: Accepted (2026-07-22)

Context

The Reconcile stage (ADR-0003) needs the AWS Well-Architected Framework criteria to map scanner findings into pillar/question language and to reason over the residual questionnaire items. The source design provisioned a managed vector database (Bedrock Knowledge Base / OpenSearch) as the RAG store.

Two facts about this specific workload argue against a managed vector store:

• The corpus is small, curated, and slowly-changing — six pillars, ~50-60 questions, best-practices and lens text (a few MB), revised by AWS only a few times a year. It is a bounded reference set, not a large or churning corpus.
• The agent queries it once per night. Managed vector stores (OpenSearch Serverless, and Bedrock Knowledge Bases which runs on OpenSearch Serverless) bill a 24/7 minimum-capacity baseline. Paying for an always-warm cluster to serve a ~30-minute nightly batch is the wrong cost shape.

Decision drivers

• Cost fit for a once-nightly query (no always-on infra for a batch).
• Provider portability — retrieval must not add a Bedrock coupling that works against ADR-0002.
• Reproducibility — the criteria used on a given run should be pinned and auditable (consistent with ADR-0003's determinism).
• Low operational surface; right-size, don't over-engineer.

Decision

Ship the WA Framework corpus as a bundled, versioned reference, not a managed vector database.

• The pillars / questions / best-practices are packaged as structured files (JSON/Markdown) versioned in the CodeBuild image (or an S3 object the image pins by version), so a run's criteria are fixed and auditable.
• Retrieval is in-process. Primary path: deterministic pillar/question keying — the reconciler looks up criteria by the pillar/question a finding maps to (no embeddings, fully reproducible). Optional: a small local embedded index for fuzzy matching, with embeddings computed at image-build time using a bundled embedding model — so runtime retrieval touches no external service and stays provider-agnostic.
• No managed vector DB (no OpenSearch Serverless, no Bedrock KB, no Kendra).

Consequences

• Zero extra infrastructure and zero always-on cost — nothing runs between nightly builds.
• Provider-agnostic retrieval — retrieval is independent of Bedrock / Claude Platform on AWS, reinforcing ADR-0002; a provider swap doesn't touch the RAG path.
• Reproducible & auditable — the corpus version is pinned with the image, so you can say exactly which WA criteria a given nightly run evaluated against.
• Corpus refresh is a maintenance task — when AWS revises the framework or a lens, the bundled reference is re-versioned and the image rebuilt. Low frequency (a few times a year), and version control makes the change explicit.
• Embedding model bundled at build (only if the local index path is used) — adds image size, but keeps runtime dependency-free.
• Diagram — the "OpenSearch vector DB (RAG)" node becomes "WA reference — bundled & versioned (local index)"; Reconcile still "queries WA criteria."

Alternatives considered

Option	Verdict	Why
Bundled versioned reference + in-process retrieval	Chosen	No always-on cost for a nightly batch; provider-agnostic; reproducible; right-sized for a small static corpus.
OpenSearch Serverless (own ingest + embed)	Rejected (now)	Provider-agnostic, but a 24/7 OCU baseline is wasteful for a once-nightly query. The scale-out target — see revisit trigger.
Bedrock Knowledge Bases (managed RAG)	Rejected	Lowest to stand up, but inherits the OpenSearch Serverless baseline and adds a Bedrock coupling that works against ADR-0002 portability.
Amazon Kendra	Rejected	High baseline cost and enterprise-search features that are overkill for a small, static reference corpus.
Load the entire corpus into the prompt each run	Rejected	Feasible on 1M-context models, but wastes tokens every run, offers no targeting, and churns the prompt cache. Keyed retrieval sends only the relevant criteria.

Revisit trigger

Move to a shared managed vector store (OpenSearch Serverless, or Bedrock Knowledge Bases if the Bedrock coupling is acceptable) when:

• the corpus grows beyond what is practical to bundle or load, or becomes heterogeneous enough to need true semantic search; or
• the agent fans out to many concurrent workloads/accounts that should share one always-warm RAG — at which point the 24/7 baseline is amortized across many consumers and becomes justified.

References

• ADR-0003 (detection & orchestration — the Reconcile stage that consumes these criteria)
• ADR-0002 (LLM provider portability — retrieval stays provider-agnostic)
• ADR-0001 (compute — the build image the reference is bundled into)