Skip to content

RAG Pipeline

Jump to bottom
Valerio edited this page Apr 26, 2026 · 2 revisions

RAG Pipeline

Harpyx's retrieval-augmented generation pipeline turns uploaded files into searchable, citable, LLM-answerable chunks. It is split across the WebApp (intake) and the Worker (extraction, chunking, embedding, indexing) so that user-facing response time stays decoupled from the heavy work.

Stages

┌───────────┐   ┌───────────┐   ┌────────────┐   ┌──────────┐   ┌──────────┐
│  Upload   │──▶│  Extract  │──▶│  Chunk     │──▶│  Embed   │──▶│  Index   │
│ (WebApp)  │   │ (Worker)  │   │ (Worker)   │   │ (Worker) │   │ (Worker) │
└───────────┘   └───────────┘   └────────────┘   └──────────┘   └──────────┘
                                                                     │
                                                                     ▼
                                                                 OpenSearch

                                  ┌────────────┐
                   User query ───▶│  Retrieve  │──▶ LLM answer with citations
                                  │  (WebApp)  │
                                  └────────────┘

1. Upload (WebApp)

Handled by IDocumentService. The user uploads a file to a project; the WebApp:

  1. Applies IUploadSecurityPolicyService — validates file extension and MIME type against UploadSecurity:AllowedExtensions / UploadSecurity:AllowedContentTypes.
  2. If MalwareScan:Enabled=true, streams the file to ClamAV (IFileMalwareScanner). On detection, the document is stored in state Quarantined; jobs are later skipped with an audit trail. If ClamAV is unreachable and MalwareScan:FailClosed=true, the upload is rejected.
  3. Persists metadata (owner tenant/workspace/project, filename, MIME type, size, state = Pending) into SQL Server.
  4. Uploads the blob to MinIO via IStorageService under a tenant/project-scoped key.
  5. Enqueues a ParseDocumentJob on RabbitMQ through IJobQueue.

An upload audit event is recorded. The user sees an immediately-responsive UI; all downstream processing is asynchronous.

2. Extract (Worker)

The Worker's JobConsumerService consumes the job and delegates to IRagIngestionService, which orchestrates the extraction.

The extractors live in Harpyx.Infrastructure.Services:

Extractor Handles
DocumentTextExtractionService Router — picks a strategy per MIME/extension
DocumentContainerExpansionService ZIP, RAR, 7z, tar.gz (via SharpCompress) + .eml, .msg
NormalizedEmailMessageParser Email bodies + attachments
CliOcrService Tesseract CLI for image-based OCR (Rag:OcrLanguages)
LlmOcrService LLM-based OCR fallback (when configured at project level)
PDF (UglyToad.PdfPig) Text-first; falls back to OCR if below PdfTextMinCharsBeforeOcr
Office (DocumentFormat.OpenXml) DOCX / XLSX / PPTX
MSGReader Outlook MSG messages
RtfPipe RTF
YamlDotNet / System.Text.Json / XML Structured formats

Containers are expanded recursively and each inner file becomes its own document (or chunk source, depending on configuration). Images extracted from PDFs/containers are routed through the OCR services.

3. Chunk

TextChunkingService splits extracted text into overlapping chunks using the parameters from appsettings.json:

"Rag": {
  "ChunkSizeChars": 1400,
  "ChunkOverlapChars": 250
}

Chunks keep source metadata — document id, page number (when available), position — so retrieval can produce citations that point back to the originating file.

KeywordExtractionService derives lightweight keyword signals per chunk that complement vector search at retrieval time.

4. Embed

LlmEmbeddingService routes chunks through the embedding provider resolved by ILlmClientResolver. The provider is picked in this order:

  1. Project-level override (Project.RagEmbeddingProviderId when RagLlmOverride is not WorkspaceDefault).
  2. Workspace default.
  3. Platform default configured via IPlatformSettingsService.

Supported personal providers are OpenAI, Anthropic Claude, and Google Gemini (see Harpyx.Infrastructure.Services.*LlmClient). Admin-managed hosted models can also expose OpenAI-compatible or local model endpoints for Chat, RAG Embedding, and OCR. See Hosted Models for configuration details.

The active embedding model name lives in Rag:OpenAiEmbeddingModel or Rag:GoogleEmbeddingModel for personal providers; hosted models use the model id published by the admin. Embedding requests are batched by Rag:EmbeddingBatchSize.

Embeddings are serialized by EmbeddingVectorSerializer before being sent to OpenSearch.

5. Index

OpenSearchChunkIndexService writes each chunk into OpenSearch under the alias OpenSearch:IndexAlias → index OpenSearch:IndexName. Each chunk document carries:

  • Document / project / tenant ids (for filtering)
  • Chunk text + keyword tokens
  • Dense vector (dimensions from OpenSearch:EmbeddingDimensions, default 1536)
  • Source metadata (page, offset)

Use of the index alias allows online reindexing — you can build a new index version, flip the alias, and delete the old one without downtime. The RagIndexVersion column on Project supports per-project reindexing.

On completion, the document transitions to Ready and the job to Completed. A job_end audit event is written. On failure, the document transitions to Failed, the job to Failed, a job_error event is written, and RabbitMQ routes the message to the dead-letter queue after retries.

6. Retrieve

At query time, the WebApp calls IRagRetrievalService.BuildContextAsync(projectId, documentIds, query, …):

  1. LlmEmbeddingService embeds the user query through the same provider used for the corpus (keeping vector spaces compatible).
  2. OpenSearchChunkIndexService performs a hybrid dense + keyword search filtered to the project and optional document id subset.
  3. The top-K chunks are materialized into a RagContextResult containing the chunks and their citation metadata.

The chat service (IProjectChatMessageService) passes that context to the chat ILlmClient alongside the user's prompt and any ProjectPrompt templates to produce the final answer. Chat provider selection follows the same override hierarchy as embeddings (ChatLlmOverride, ChatProviderId).

Per-document URL fetch

UrlFetcherService (bound to UrlFetch:TimeoutSeconds, UrlFetch:MaxContentBytes, UrlFetch:AllowHttp) allows projects to ingest content by URL rather than only by file upload. Fetches still go through the same extraction → chunk → embed → index path.

Tuning knobs

Setting Effect
Rag:ChunkSizeChars Larger → fewer, denser chunks; risk of context dilution
Rag:ChunkOverlapChars Protects against boundary-straddling facts
Rag:PdfTextMinCharsBeforeOcr Below this, PDF falls through to OCR
Rag:OcrLanguages Tesseract language packs (e.g. ita+eng)
Rag:EmbeddingBatchSize Balances throughput vs provider rate limits
OpenSearch:EmbeddingDimensions Must match the embedding model's output dimension
OpenSearch:RequestTimeoutSeconds Individual OS request timeout

Clone this wiki locally