Module 10 RAG OpenSearch and LiteLLM

📚 Module 10 · RAG, OpenSearch and LiteLLM

Retrieval done properly — chunking, hybrid search, reranking, and an evaluation gate.

Module 10 of 16 · track Ground · Estimated time: 8–10 hours · Prerequisites: Module 02. Module 06 or 08 for the agentic RAG labs.

The largest module, and the one with the most production value. Naive RAG is easy and usually bad. This module walks the whole pipeline — corpus, chunking, lexical and dense retrieval, fusion, reranking, context packing, evaluation — with a reusable ragkit library you can lift into your own work.

Open the module on GitHub ↗: the slides, notebooks, exercises and solutions are there. This page is the reading order, with the tutorial and the workbench alongside.

Read first, on the site

The lesson that frames this module is Is 82% Good Enough? How to Prove an AI Agent Meets Its Bar (6 min): the evaluation gate retrieval has to pass. Then try the acceptance bar calculator in the workbench: the same decision, with your numbers.

What you will be able to do

• Build RAG by hand before using any managed service
• Choose a chunking strategy from document structure rather than habit
• Combine lexical and dense retrieval with reciprocal rank fusion
• Add reranking and measure whether it actually helped
• Gate a RAG release on evaluation metrics, not impressions
• Route across model providers with LiteLLM and fail over cleanly

Concepts in this module

Portable GenAI concepts	AWS-specific surface
Chunking	Amazon OpenSearch Serverless
Embeddings	Bedrock Knowledge Bases
Lexical vs dense retrieval	Bedrock embeddings
Hybrid search and RRF	Bedrock Runtime
Reranking
Context packing
RAG evaluation
Model routing

Portable concepts transfer to any stack. The AWS column is where this module touches the cloud — see docs/concepts/portability-matrix.md.

Run it in this order

#		Step	What it is
1	📖	slides/02_RAG_slides.md	RAG fundamentals
2	📖	src/rag_by_hand.py	RAG by hand — no library, no magic
3	💻	labs/rag-labs/01_foundations.ipynb	Lab 1 — foundations
4	💻	labs/rag-labs/03_index_design.ipynb	Lab 3 — index and chunking design
5	💻	labs/rag-labs/04_retrieval_reranking.ipynb	Lab 4 — retrieval and reranking
6	✏️	exercises/EX3_RAG_interim1.md	RAG interim exercise 1
7	💻	labs/rag-labs/05_context_generation.ipynb	Lab 5 — context packing and generation
8	💻	labs/rag-labs/06_evaluation_gate.ipynb	Lab 6 — the evaluation gate
9	✏️	exercises/EX5_RAG_final.md	RAG final exercise
10	💻	labs/rag-labs/07_tokens_cost.ipynb	Lab 7 — tokens and cost
11	💻	labs/rag-labs/08_agentic_fde_lab.ipynb	Lab 8 — agentic RAG
12	📖	slides/AWS_OpenSearch_Deck_1.pptx	OpenSearch, part 1
13	💻	labs/opensearch/opensearch_lab.py	OpenSearch lab
14	📖	slides/01_LiteLLM_slides.md	LiteLLM — provider routing
15	💻	notebooks/01_LiteLLM_notebook.ipynb	LiteLLM notebook
16	✏️	exercises/EX6_consolidated_takehome.md	Consolidated take-home

📖 read   💻 run   ✏️ practise   📊 workbook   🔖 reference

Worked solutions live in solutions/. Attempt every exercise closed-book first — the solutions are written to be read after you have a wrong answer to compare against.

Project artefact

labs/rag-labs/ragkit is a complete, reusable retrieval library — chunking, fusion, reranking, evaluation, cost. Lift it into your own projects.

Common mistakes

• Chunking by character count because it was the default. Structure beats size.
• Reranking added without measurement — it costs latency and may not help your corpus.
• Evaluating RAG on vibes. Build the golden set first.

When it breaks

Every error anyone has hit in this course is in the Error Index, by the exact string. Model access and region problems are in Model and Region Notes. If it is a question rather than an error, check the answered ones first, then ask in Q&A.

Field guide for this module

Frameworks

• Grounding Triangle — The three sides most systems confuse
• Context Budget Ledger — Pack to a token budget, not a top-k
• Evidence Ladder — Why the golden set must be frozen first

Quick reference

• RAG pipeline — Stage by stage, with the failure decoder
• Model selection — Routing and provider portability

Recipes and procedures

• Runbook · stale knowledge base — Index freshness as a gate check

---

← Module 09 · LLM Memory Mechanics · All modules · Module 11 · Amazon Bedrock AgentCore →

_{✏️ This page is generated from modules/10-rag-opensearch-litellm/README.md. To change it, edit the module README: an edit made here is replaced at the next sync.}