RAG

📚 RAG (Retrieval-Augmented Generation)

Pipeline completo para RAG: carregar documentos, splitter, embedding, vector store e chain de Q&A.

---

Visão Geral

Document → Loader → Splitter → Embedding → VectorStore
                                                    ↓
Query → Embedding → Retriever → Top-K Documents → RAGChain → Answer

---

Document

Modelo base para documentos:

from mangaba.rag import Document

doc = Document(
    content="This is the document text content",
    metadata={"source": "file.pdf", "page": 1},
)

Propriedades

Propriedade	Tipo	Descrição
content	str	Conteúdo do documento
metadata	dict	Metadados (fonte, página, etc.)

---

Loaders

TextLoader

from mangaba.rag import TextLoader

loader = TextLoader("path/to/document.txt")
documents = loader.load()
# [Document(content="...", metadata={"source": "document.txt"})]

CSVLoader

from mangaba.rag import CSVLoader

loader = CSVLoader("data.csv", text_columns=["description", "notes"])
documents = loader.load()
# Cada row vira um Document

WebPageLoader

from mangaba.rag.loaders import WebPageLoader

loader = WebPageLoader()
documents = loader.load_urls(["https://example.com/article"])

---

Splitters

RecursiveTextSplitter

from mangaba.rag import RecursiveTextSplitter

splitter = RecursiveTextSplitter(
    chunk_size=500,     # Tokens por chunk
    chunk_overlap=50,   # Overlap entre chunks
)

# Split texto
chunks = splitter.split_text("Long document text here...")

# Split documents
documents = loader.load()
split_docs = splitter.split_documents(documents)

Parâmetros

Parâmetro	Tipo	Default	Descrição
chunk_size	int	500	Tamanho do chunk em tokens
chunk_overlap	int	50	Overlap entre chunks

---

Embeddings

OpenAIEmbedding

from mangaba.embeddings import OpenAIEmbedding

embed = OpenAIEmbedding(api_key="OPENAI_API_KEY")

# Texto único
vector = embed.embed_text("Hello world")
# [0.012, -0.034, ...]  # 1536 dimensions

# Batch
vectors = embed.embed_batch(["Hello", "World"])
# [[...], [...]]

GoogleEmbedding

from mangaba.embeddings import GoogleEmbedding

embed = GoogleEmbedding(api_key="GOOGLE_API_KEY")
vector = embed.embed_text("Hello world")
# 768 dimensions (text-embedding-004)

HuggingFaceEmbedding

from mangaba.embeddings import HuggingFaceEmbedding

# Local (requires: pip install mangaba[embeddings] or sentence-transformers)
embed = HuggingFaceEmbedding(
    model="sentence-transformers/all-MiniLM-L6-v2",
    use_local=True
)
vector = embed.embed_text("Hello world")
# 384 dimensions, fast and lightweight

# Or using HuggingFace Inference API
embed = HuggingFaceEmbedding(
    model="BAAI/bge-m3",
    api_key="HF_TOKEN",
    use_local=False
)
vector = embed.embed_text("Hello world")
# 1024 dimensions, multilingual

# Batch processing with automatic caching
vectors = embed.embed_batch(["Hello", "World", "Hello"])  # "Hello" cached

Available models:

• sentence-transformers/all-MiniLM-L6-v2 - 384d, fast, lightweight
• BAAI/bge-m3 - 1024d, multilingual, multi-granularity
• intfloat/multilingual-e5-large-instruct - 1024d, 100+ languages

---

Retriever

Bridge entre embeddings e vector store:

from mangaba.rag import Retriever
from mangaba.rag import Document
from mangaba.embeddings import OpenAIEmbedding
from mangaba.vectorstores import InMemoryVectorStore

retriever = Retriever(
    embedding=OpenAIEmbedding(api_key="KEY"),
    store=InMemoryVectorStore(),
)

# Adicionar documentos
docs = [
    Document(content="Python is a programming language", metadata={"topic": "programming"}),
    Document(content="Machine learning uses data to learn patterns", metadata={"topic": "ml"}),
]
retriever.add_documents(docs)

# Buscar
results = retriever.search("What is Python?", top_k=2)
for doc in results:
    print(f"{doc.content} (score: {doc.metadata['score']:.3f})")

# Limpar
retriever.clear()

---

RAGChain

Pipeline completo de Q&A com contexto:

from mangaba.rag import RAGChain
from mangaba.rag import Retriever
from mangaba.core.llm import create_llm_client

# Setup
llm = create_llm_client(provider="google", api_key="KEY", model="gemini-2.5-flash")
retriever = Retriever(
    embedding=OpenAIEmbedding(api_key="KEY"),
    store=InMemoryVectorStore(),
)

# Adicionar documentos ao retriever
retriever.add_documents(documents)

# Criar chain
chain = RAGChain(
    llm=llm,
    retriever=retriever,
    top_k=3,  # Documentos para incluir no contexto
)

# Fazer pergunta
answer = chain.ask("What is machine learning?")
print(answer)
# "Machine learning is a subset of AI that..."

---

Pipeline Completo

from mangaba.rag import (
    Document,
    TextLoader,
    RecursiveTextSplitter,
    Retriever,
    RAGChain,
)
from mangaba.embeddings import OpenAIEmbedding
from mangaba.vectorstores import InMemoryVectorStore
from mangaba.core.llm import create_llm_client

# 1. Carregar documentos
loader = TextLoader("data/knowledge_base.txt")
documents = loader.load()

# 2. Split
splitter = RecursiveTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)

# 3. Setup embedding + vector store
embedding = OpenAIEmbedding(api_key="KEY")
store = InMemoryVectorStore()

# 4. Retriever
retriever = Retriever(embedding=embedding, store=store)
retriever.add_documents(chunks)

# 5. RAG Chain
llm = create_llm_client(provider="google", api_key="KEY", model="gemini-2.5-flash")
chain = RAGChain(llm=llm, retriever=retriever, top_k=3)

# 6. Perguntar
answer = chain.ask("How do I configure the system?")
print(answer)