Interactive Retriever - Customer Support Automation

README outline

• README outline
• Main goals
• Task Outline
• Solution Description
  • Approaches
  • Technologies
• Results
  • BM25 results
  • Parent-Child results
• Installation
  • Experiments notebook setup
  • AWS Knowledge base setup
  • AWS Lambda setup
• Lambda function Postman API example

Main goals

• Experimenting with different RAG approaches to find the best answer generation performance on the Question - Answer dataset: LangChain Docs Q&A.

• Interactive retriever: setting up a Chatbot with history and RAG over the LangChain documentation that can be called through a Lambda function.

Task specification

1. Preparations:

   • Choose embedding and text gen models from Bedrock.

   • Create AWS cost estimation.

   • Explore Advanced RAG techniques: choose 2 other retrievers while taking into considerations the features of the dataset.

   • Check Ragas and Langsmith for evaluation techniques and metrics, choose 5 for validation.

2. RAG experimentation:

   • Implement ingestion and inference pipeline for the selected 3 RAG techniques locally.

   • Compare the results and choose the best for this use case.

3. Cloud deployment:

   • Create ingestion pipeline to AWS Knowledge Base and ingest the data.

   • Setup S3 as persistent memory storage.

   • Implement Lambda Function.

Solution Description

Approaches

• Three different RAG sub-techniques

• Chat history implementation with the AWS Converse API over a Lambda function

• Query rewriting for naive retriever in AWS Knowledge Base

---

Technologies

• Cloud: AWS Bedrock

  • LLM: Amazon Nova Micro for question answering, Anthropic Claude 3.5 Sonnet for Query rewriting
  • Embedding Model: Amazon Titan Text Embeddings V2

• RAG approaches

  • Naive retriever with Pinecone
  • Custom Parent-Child retriever with Pinecone
  • BM25 from Langchain

• Evaluation

  • Ragas metrics
    • ContextPrecision
    • ContextRecall
    • Faithfulness
    • BLUE
    • ROUGE

• AWS Lambda

  • Retrieval with AWS Knowledge Base using naive Pinecone
  • Data stored on S3
  • Chat history managed with Converse API

Results

Among the RAG approaches investigated, the Parent-Child retriever outperformed the other two on the benchmarked metrics, balanced by its higher complexity.

BM25 offers very fast initialization and query time, but with 20% lower retrieval accuracy compared to Parent-Child.

Optuna grid parameter search results

BM25 results

	k1	b	epsilon	top_k	ContextPrecision	ContextRecall	Faithfulness	Blue	Rouge
0	1.300000	0.500000	0.250000	3	0.645604	0.725484	0.620222	0.099129	0.189244
1	1.300000	0.500000	0.250000	5	0.651252	0.757625	0.576593	0.119858	0.190237
2	1.300000	0.700000	0.250000	3	0.631868	0.719804	0.558512	0.095685	0.187126
3	1.300000	0.700000	0.250000	5	0.631426	0.795937	0.603476	0.113690	0.189978
4	1.300000	0.700000	0.250000	10	0.601835	0.824621	0.607423	0.158535	0.212955
5	1.300000	0.900000	0.250000	3	0.623626	0.666279	0.482375	0.095501	0.204845
6	1.300000	0.900000	0.250000	5	0.613599	0.720058	0.571203	0.111747	0.202184
7	1.500000	0.500000	0.250000	3	0.642857	0.722255	0.679987	0.112833	0.187256
8	1.500000	0.500000	0.250000	5	0.649939	0.765873	0.631159	0.120628	0.190556
9	1.500000	0.700000	0.250000	3	0.630952	0.720498	0.588889	0.100062	0.180367
10	1.500000	0.700000	0.250000	5	0.629075	0.837907	0.585238	0.115513	0.183973
11	1.500000	0.750000	0.150000	3	0.675824	0.664782	0.633411	0.140505	0.232004
12	1.500000	0.750000	0.250000	5	0.643056	0.742529	0.764073	0.146480	0.227122
13	1.500000	0.900000	0.250000	3	0.621795	0.632941	0.502768	0.107415	0.204883
14	1.500000	0.900000	0.250000	5	0.621322	0.724022	0.602209	0.110017	0.204832
15	1.700000	0.500000	0.250000	3	0.637363	0.742843	0.699967	0.111143	0.185863
16	1.700000	0.500000	0.250000	5	0.641056	0.805221	0.623826	0.110399	0.188370
17	1.700000	0.700000	0.250000	3	0.636447	0.724330	0.582083	0.099912	0.191906
18	1.700000	0.700000	0.250000	5	0.634829	0.793750	0.532648	0.112121	0.194999
19	1.700000	0.900000	0.250000	3	0.618132	0.594574	0.511643	0.104510	0.203546
20	1.700000	0.900000	0.250000	5	0.623062	0.727877	0.576381	0.110086	0.204188
21	1.950000	0.900000	0.250000	5	0.624588	0.709921	0.645437	0.160753	0.223479

Parent-Child results

	Child chunk size	Child overlap	top_k	ContextPrecision	ContextRecall	Faithfulness	Blue	Rouge
0	150	50	5	0.828480	0.784510	0.670657	0.119527	0.230889
1	250	0	5	0.820833	0.767049	0.626882	0.123225	0.240077
2	350	50	5	0.806410	0.804810	0.706889	0.135123	0.244033
3	500	100	5	0.810546	0.830882	0.785000	0.147975	0.220829
4	750	100	5	0.833578	0.843746	0.759490	0.122855	0.210450

Installation

Experiments notebook setup

Create a virtual environment (venv recommended) with the packages installed from the included requirements.txt file.

pip install requirements.txt

---

AWS KnowledgeBase setup

Use AWS UI or CLI.

1. Upload your documents into an S3 bucket.

2. Initialize your Pinecone Index.

3. Make a secret in AWS SecretManager with your Pinecone API key.

4. Setup necessary permissions with Roles and Policies.

5. Follow the steps in the AWS UI or run the commands.

---

AWS Lambda setup

1. Create a Lambda function through AWS UI or CLI with the necessary permissions to Bedrock resources.

2. Upload the files from the lambda_files folder, compressed in a zip.

3. Make a Function URL.

4. Call the URL.

Lambda function Postman API example

First message:

Second message:

Third message:

---