This document describes the architecture and operation of ClioDeck's RAG (Retrieval-Augmented Generation) system, an academic writing assistant for historians and humanities researchers.

## Table of Contents

- [Overview](#overview)
- [Global Architecture](#global-architecture)
- [Indexing Pipeline](#indexing-pipeline)
- [Search System](#search-system)
- [Assisted Generation (RAG)](#assisted-generation-rag)
- [External Integrations](#external-integrations)
- [Performance and Optimizations](#performance-and-optimizations)

---

## Overview

ClioDeck is an Electron application that combines:
- **Frontend**: React/TypeScript interface with a CodeMirror 6 editor (live markdown rendering)
- **Backend**: Node.js/TypeScript services for indexing and search
- **Python Services**: Topic modeling and textometric analysis
- **LLM**: a typed provider registry — Ollama, embedded (`node-llama-cpp`), Anthropic, OpenAI-compatible, Mistral, Gemini — for embeddings and text generation
- **Storage**: SQLite for data and vector indexes

### Design Goals

1. **Local-first**: All data remains on the user's machine
2. **Performance**: Optimized for modest machines (8-16 GB RAM, CPU only)
3. **Academic**: Complete traceability and verifiable citations
4. **Interoperability**: Integration with Zotero, BibTeX, Markdown, PDF

---

## Global Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    ClioDeck Electron App                      │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Frontend   │  │   Backend    │  │   Services   │      │
│  │  React/Vite  │◄─┤  TypeScript  │◄─┤    Python    │      │
│  │              │  │              │  │              │      │
│  │ • Editor     │  │ • PDF Index  │  │ • Topic      │      │
│  │ • Chat UI    │  │ • Vector DB  │  │   Modeling   │      │
│  │ • Config     │  │ • Search     │  │ • BERTopic   │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                            │                │                │
│                            ▼                ▼                │
│                    ┌──────────────┐  ┌──────────────┐      │
│                    │   Ollama     │  │   SQLite     │      │
│                    │   (Local)    │  │ + HNSW Index │      │
│                    └──────────────┘  └──────────────┘      │
│                                                               │
└─────────────────────────────────────────────────────────────┘
```

### Key Technologies

| Component | Technology | Role |
|-----------|------------|------|
| **UI Framework** | React 18 + TypeScript | User interface |
| **Editor** | CodeMirror 6 + Lezer | Live markdown rendering, Pandoc dialect |
| **Build Tool** | Vite | Bundling and hot reload |
| **Desktop** | Electron 40 | Multi-platform desktop app |
| **Database** | better-sqlite3 | Synchronous local storage |
| **Vector Index** | hnswlib-node | Fast vector search |
| **Sparse Index** | natural (BM25) | Keyword search |
| **LLM** | Typed provider registry | Ollama, embedded, Anthropic, OpenAI-compatible, Mistral, Gemini |
| **PDF Processing** | pdfjs-dist | Text extraction |
| **Bibliography** | BibTeX parsing | Reference management |
| **Topic Modeling** | Python + BERTopic | Thematic analysis |

---

## Indexing Pipeline

The indexing pipeline transforms PDFs into queryable vector chunks.

### Step 1: PDF Extraction

**File**: `backend/core/pdf/PDFExtractor.ts` (not `PDFIndexer.ts`, which only orchestrates and delegates here)

```typescript
// 1. Load PDF with pdfjs
const pdf = await pdfjs.getDocument(pdfPath).promise;

// 2. Extract text page by page
const pages = [];
for (let i = 1; i <= pdf.numPages; i++) {
  const page = await pdf.getPage(i);
  const textContent = await page.getTextContent();
  const text = textContent.items.map(item => item.str).join(' ');
  pages.push({ pageNumber: pageNum, text });  // field is `text`, not `content` — PDFExtractor.ts:174-176
}

// 3. Extract metadata
const metadata = await pdf.getMetadata();
const title = metadata.info.Title || path.basename(pdfPath);
```

**Output**: Array of pages with raw text + document metadata

> **This code never runs directly inside the Electron main process.** Real
> PDF indexing (`src/main/services/pdf/PdfIndexer.ts`, `pdf-service.ts`)
> calls `extractPdfIsolated()` (`src/main/services/pdf-extract-isolated.ts`),
> which spawns the *system* Node binary as a child process and runs
> `PDFExtractor` inside a separate worker script — a workaround for a
> pdfjs-dist crash on Electron's bundled Node. Three real, previously
> undocumented consequences for indexing a large PDF:
> - **120-second hard timeout per PDF.** A pathological or very large scan
>   that takes longer fails with `"PDF extraction timed out after 120s"` —
>   there is no way to raise this from the UI.
> - **Global sequential queue, one extraction at a time, app-wide.**
>   `extractPdfIsolated()` chains every call onto a single `pending`
>   promise specifically to bound RAM — importing a large batch of PDFs
>   (e.g. a big Zotero library) processes them strictly one by one, never
>   in parallel, regardless of CPU cores available.
> - **Requires a system Node 20+ binary on PATH** (or at a few hardcoded
>   Homebrew/nvm paths) separate from Electron's own bundled Node — if
>   none is found, extraction fails with an explicit "requires a system
>   Node.js binary" error rather than a silent hang.
>
> **A fourth consequence, real bug, verified against code:** that 120-second
> window (or longer, queued behind other pending PDFs in the same
> sequential lane) is plenty of time for a user to switch ClioDeck
> projects. Unlike the Zotero/Tropy/Obsidian bugs found elsewhere in this
> audit (issues #33/#34/#35 — those *misattribute* data to the wrong
> project), PDF indexing does NOT have that problem: `PdfIndexer`
> (`src/main/services/pdf/PdfIndexer.ts`) receives its `vectorStore`
> through constructor injection, captured once, so an in-flight
> `indexPDF()` call keeps writing to the **correct** (original) project's
> store even after a switch — no cross-project data corruption. But
> `pdf-service.ts`'s `init(projectPath)` — called on every project
> switch — unconditionally does `this.vectorStore.close()` on the
> *previous* project's store before setting up the new one, with no check
> for an in-flight indexing operation still using that exact same store
> instance. `PdfIndexer.indexPDF()` writes to `this.vectorStore` again
> after the (possibly 120-second) extraction finishes
> (`setDocumentCollections(...)`, plus the embedding/chunk-storage steps
> inside `this.backend.indexPDF(...)`) — against a `better-sqlite3` handle
> that's by then closed. The practical effect: switching projects while a
> PDF was indexing made the job throw once it tried to write its results.
> **Fixed in RC4** — in-flight indexing is drained before the store closes.
> The manuscript index service, written in the same cycle, reproduced the
> same defect and was fixed with it.

---

### Step 2: Adaptive Chunking

**Files**:
- `backend/core/chunking/AdaptiveChunker.ts` (recommended)
- `backend/core/chunking/DocumentChunker.ts` (fallback)
- `backend/core/chunking/SemanticChunker.ts` (advanced semantic-aware chunking)
- `backend/core/chunking/ChunkDeduplicator.ts` (removes near-duplicate chunks)
- `backend/core/chunking/ChunkQualityScorer.ts` (scores chunk quality for filtering)
- `backend/core/chunking/EmbeddingCache.ts` (caches embeddings for performance)

#### Structure Detection

The adaptive chunker automatically detects document structure:

```typescript
// Header detection — patterns as they actually are in AdaptiveChunker.ts,
// stricter than earlier versions of this page implied
const patterns = {
  markdown: /^#{1,6}\s+(.+)/,                          // # Title, ## Subtitle
  numbered: /^(\d+(?:\.\d+)*)\.\s+([A-Z][^.]{2,50})$/, // "1. Introduction" — needs a trailing dot and a capitalized 2-50 char title
  uppercase: /^([A-Z][A-Z\s]{5,50})$/,                 // ALL CAPS, 6-51 chars, also checked against a known-header allowlist (ABSTRACT, INTRODUCTION, …)
  roman: /^([IVX]+)\.\s+([A-Z][^.]{2,50})$/            // "I. Introduction" — same capitalized-title constraint as `numbered`, not a bare "I. anything" match
};

// Automatic classification
const sectionTypes = {
  abstract: /abstract|résumé/i,
  introduction: /introduction/i,
  methodology: /method|méthodologie/i,
  results: /results|résultats/i,
  discussion: /discussion/i,
  conclusion: /conclusion/i,
  references: /references|bibliographie/i
};
```

#### Chunk Creation

**Principles**:

1. **Respect sentence boundaries**: Chunks always end with `.`, `!`, `?` or `;`
2. **Document context**: Each chunk includes title and section as prefix
3. **Skip references**: Bibliography section is not indexed
4. **Smart overlap**: Overlaps occur on complete sentences

**Generated chunk example**:

```
[Doc: Active Learning Strategies in Higher Education | Section: Methodology]

The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment. Students were divided
into control and experimental groups. The control group followed traditional
lecture-based instruction, while the experimental group participated in
collaborative problem-solving activities.
```

**Configuration**:

```typescript
interface ChunkingConfig {
  maxChunkSize: number;     // Target max size in words (300-800)
  overlapSize: number;      // Overlap in words (50-100)
  minChunkSize: number;     // Minimum words, avoids tiny leftover chunks
}

const CHUNKING_CONFIGS = {
  cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },
  standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },
  large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};
```

There is no `maxTokens` field anywhere in this config — an earlier version
of this page invented one; token limits for embedding models are handled
separately, not as part of chunking config.

**Output**: Array of chunks with metadata

```typescript
interface DocumentChunk {
  id: string;                 // UUID
  documentId: string;         // Document reference
  content: string;            // Chunk text (with context)
  pageNumber: number;         // Source page
  chunkIndex: number;         // Position in document
  startPosition: number;      // Character offset, start
  endPosition: number;        // Character offset, end
  // No `embedding` field here — embeddings live in the DB row
  // (pdf_chunks.embedding) or the vector store, not on this object.
  metadata?: {
    sectionTitle?: string;    // "Methodology", "Results"
    sectionType?: string;     // "methodology", "results"
    sectionLevel?: number;    // 1, 2, 3...
  };
}
```

---

### Step 3: Embedding Generation

**File**: `backend/core/llm/providers/ollama.ts` (one of six embedding providers behind the typed registry — see [Embedded LLM Support](#embedded-llm-support) below for the others)

```typescript
// 1. Connect to Ollama — the MODERN /api/embed endpoint, batched, not the
// legacy /api/embeddings single-prompt endpoint an earlier version of this
// page showed. ollama.ts's own comment explains why: "the legacy
// /api/embeddings endpoint ignores `truncate` and 500s — which is exactly
// the PDF-indexing crash this replaces."
const response = await fetch('http://localhost:11434/api/embed', {
  method: 'POST',
  body: JSON.stringify({
    model: 'nomic-embed-text',  // 768 dimensions
    input: [chunkContent, /* ...more chunks, batched */],
    truncate: true
  })
});

// 2. Retrieve vectors (one per input, in order)
const { embeddings } = await response.json();
// embeddings: number[][] (768 dimensions each)
```

**Model used**: `nomic-embed-text`
- Dimensions: 768
- Max context: 8192 tokens (model-dependent — `nomic-embed-text` itself advertises 2048 and ignores larger `num_ctx` values)
- Size: ~274 MB
- Performance: ~100 embeddings/second on average CPU

**Oversized input, Ollama path**: inputs that overflow the model's context are
**truncated server-side** by Ollama itself (`truncate: true` in the `/api/embed`
call above) — `ollama.ts` does no client-side chunking of its own; the comment
in the file is explicit that server-side truncation happens "rather than"
splitting the text.

**A second, earlier safety net exists specifically for PDF indexing.**
`src/main/services/pdf/PdfIndexer.ts` slices any chunk text to
`EMBED_CHAR_CAP = 6000` **characters** before it ever reaches the embedding
provider — independent of, and applied before, whichever of the two paths
above eventually runs. Its own comment explains why: `AdaptiveChunker` can
occasionally emit a single oversized chunk (a paragraph longer than
`maxChunkSize`), and without this cap that one pathological chunk used to
abort the *entire* indexing run with an HTTP 500 ("input length exceeds the
context length") from Ollama — this cap is what prevents that, not
`truncate: true` alone.

**Oversized input, embedded-model path**: the chunk-and-average `chunkText()`
method shown below is real, but it belongs to `EmbeddedLLMClient.ts` (the
`node-llama-cpp` embedded provider), not Ollama — an earlier version of this
page misattributed it here. Its threshold is `EMBEDDING_MAX_CHUNK_LENGTH = 2000`
**characters**, not "8192 tokens":

```typescript
// EmbeddedLLMClient.ts — embedded provider only, not Ollama
private chunkText(text: string, maxLength: number): string[] {
  // Find sentence ending in last 200 characters
  const sentenceEndings = /[.!?;](?=\s|$)/g;
  // ... cut at last found period
}
// Chunks are embedded individually, then averaged into a single vector.
```

---

### Step 4: Multi-Strategy Indexing

**File**: `backend/core/vector-store/EnhancedVectorStore.ts`

Three indexes are created simultaneously:

#### Index 1: SQLite (database)

```sql
-- Real schema, VectorStore.ts:104-113 — table is pdf_chunks, not chunks;
-- no metadata/created_at columns; start_position/end_position instead
CREATE TABLE IF NOT EXISTS pdf_chunks (
  id TEXT PRIMARY KEY,
  document_id TEXT NOT NULL,
  content TEXT NOT NULL,
  page_number INTEGER NOT NULL,
  chunk_index INTEGER NOT NULL,
  start_position INTEGER NOT NULL,
  end_position INTEGER NOT NULL,
  embedding BLOB,
  FOREIGN KEY (document_id) REFERENCES pdf_documents(id) ON DELETE CASCADE
);

CREATE INDEX IF NOT EXISTS idx_chunks_document_id ON pdf_chunks(document_id);
CREATE INDEX IF NOT EXISTS idx_chunks_page_number ON pdf_chunks(page_number);
```

**Role**: Persistent storage + fallback for linear search

#### Index 2: HNSW (fast vector search)

**File**: `backend/core/vector-store/HNSWVectorStore.ts`

```typescript
import { HierarchicalNSW } from 'hnswlib-node';

const index = new HierarchicalNSW('cosine', 768); // 768 dimensions
index.initIndex(maxElements, M=16, efConstruction=100);

// Indexing
for (const chunk of chunks) {
  index.addPoint(chunk.embedding, chunk.id);
}

// Save to disk
index.writeIndexSync(indexPath);
```

**Parameters**:
- `M=16`: Connections per node (speed/precision trade-off)
- `efConstruction=100`: Construction effort
- `efSearch=50`: Search effort

**Complexity**:
- Construction: O(n log n)
- Search: O(log n)

**Memory footprint**: ~500 MB for 50k chunks (768 dims)

#### Index 3: BM25 (keyword search)

**File**: `backend/core/search/BM25Index.ts`

There is no hand-rolled inverted index — indexing delegates entirely to
`natural`'s `TfIdf` class, with BM25 scoring layered on top of it:

```typescript
import natural from 'natural';
const { TfIdf } = natural;

// 1. Delegate document storage/tokenization to natural's TfIdf
const tfidf = new TfIdf();
tfidf.addDocument(preprocessText(chunk.content));

// 2. BM25 scoring, computed from TfIdf's term/document frequencies
const idf = Math.log((totalDocs - docFreq + 0.5) / (docFreq + 0.5));
const score = idf * ((tf * (k1 + 1)) / (tf + k1 * (1 - b + b * (docLength / avgDocLength))));
```

**Parameters**:
- `k1=1.5`: Term frequency saturation
- `b=0.75`: Document length normalization

**Memory footprint**: ~100 MB for 50k chunks

---

## Search System

### Hybrid Search (Dense + Sparse)

**File**: `backend/core/search/HybridSearch.ts`

Hybrid search combines the advantages of two complementary approaches:

```
Query: "bloom taxonomy methodology"
        │
        ├────────────────────┬────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
  ┌──────────┐        ┌──────────┐        ┌──────────┐
  │  Dense   │        │  Sparse  │        │  Fusion  │
  │  (HNSW)  │        │  (BM25)  │        │  (RRF)   │
  └──────────┘        └──────────┘        └──────────┘
        │                    │                    │
        ▼                    ▼                    ▼
   Top-50 results       Top-50 results       Top-10 final
   (semantic)           (keywords)           (best of both)
```

#### Step 1: Dense Search (HNSW)

```typescript
// 1. Generate query embedding via the configured EmbeddingProvider
// (Ollama, embedded, or a cloud provider — see Embedded LLM Support below)
const queryEmbedding = await embeddingProvider.generateEmbedding(query);

// 2. HNSW search
const denseResults = await hnswStore.search(queryEmbedding, 50);
// Returns: [{ id, score }, ...]
```

**Advantage**: Understands semantic meaning ("active learning" ≈ "apprentissage actif")

#### Step 2: Sparse Search (BM25)

```typescript
// 1. Query tokenization
const queryTokens = tokenizer.tokenize(query.toLowerCase());

// 2. BM25 scoring
const sparseResults = bm25Index.search(queryTokens, 50);
// Returns: [{ id, score }, ...]
```

**Advantage**: Excellent for rare terms, proper nouns, acronyms ("BERTopic", "Piaget")

#### Step 3: RRF Fusion (Reciprocal Rank Fusion)

```typescript
function reciprocalRankFusion(
  denseResults: SearchResult[],
  sparseResults: SearchResult[],
  k: number = 60
): SearchResult[] {
  const scores = new Map<string, number>();

  // Dense search contribution (60%)
  for (let i = 0; i < denseResults.length; i++) {
    const chunkId = denseResults[i].id;
    const rrfScore = 0.6 / (k + i + 1);
    scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
  }

  // Sparse search contribution (40%)
  for (let i = 0; i < sparseResults.length; i++) {
    const chunkId = sparseResults[i].id;
    const rrfScore = 0.4 / (k + i + 1);
    scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
  }

  // Sort by final score
  return Array.from(scores.entries())
    .sort((a, b) => b[1] - a[1])
    .slice(0, 10)
    .map(([id, score]) => ({ id, score }));
}
```

**Weights**: 60% dense / 40% sparse (configurable)

**RRF Formula**: `RRF(d) = Σ weight_i / (k + rank_i(d))`

**Exact match boost**: `HybridSearch.ts` multiplies a chunk's RRF score by
`EXACT_MATCH_BOOST` when the query string appears verbatim in it — RRF
scores alone peak too low (top-1 ≈ 1/(K+1) ≈ 0.016) to otherwise outrank
strong semantic neighbours scoring 0.45–0.55.

### Performance

| Metric | Linear Search | HNSW Only | Hybrid (HNSW+BM25) |
|--------|--------------|-----------|-------------------|
| **Time** | 500ms | 15ms | 30ms |
| **Precision@10** | 65% | 75% | 80% |
| **Recall@10** | 45% | 55% | 60% |

---

## Assisted Generation (RAG)

### Complete Pipeline

```
User Query
    │
    ├─► 1. Generate embedding
    │       │
    │       ▼
    ├─► 2. Hybrid search (HNSW + BM25)
    │       │
    │       ▼
    ├─► 3. Retrieve top-K chunks (K=10)
    │       │
    │       ▼
    ├─► 4. Optional: Graph expansion
    │       │
    │       ▼
    ├─► 5. Build context
    │       │
    │       ▼
    └─► 6. Generate with LLM
            │
            ▼
        Response with citations
```

### Steps 1-3: Search (see previous section)

### Step 4: Graph Expansion — declared but dead

`useGraphContext`, `graphSimilarityThreshold`, and `additionalGraphDocs` are
declared in `backend/types/config.ts` and set by mode presets, and even
exposed in the RAG settings UI — but confirmed to have **zero consumers**
anywhere in `retrieval-service.ts` or `backend/core/rag/`. Turning this
setting on currently has no effect on retrieval; it isn't wired to
anything. Documented here as a known gap, not as working behavior — the
mechanism described in earlier versions of this page (graph-neighbor
lookup, cosine similarity expansion) does not exist in the code today.

### Step 5: Context Building

**File**: `src/main/services/fusion-chat-service.ts`, `formatContextAsSystemPrompt()`.
An earlier version of this page showed a fabricated Markdown-based builder
(`# Context (sources)`, `## [Title, p.X]` headers) that doesn't match the
real code at all — and contradicted this wiki's own
[Brainstorm guide](./1.11-Brainstorm-Mode-Guide.md#source-grounding),
which correctly describes the real format. The actual builder produces
numbered `TITRE`/`AUTEUR`/`EXTRAIT` blocks, in French, with an explicit
rule forcing the model to answer only from those fields:

```typescript
function formatContextAsSystemPrompt(hits: MultiSourceSearchResult[]): string {
  const SNIPPET_CHARS = 1500;
  const lines: string[] = [
    'Contexte extrait du corpus indexé (sources numérotées ci-dessous).',
    "RÈGLE : réponds UNIQUEMENT à partir des sources ci-dessous (champs TITRE, AUTEUR et EXTRAIT). Ne complète JAMAIS avec tes connaissances générales.",
    // ...two more rule lines (TITRE-as-valid-answer for identification
    // questions, explicit no-canned-phrase wording for "not found") —
    // omitted here for brevity, present in the real function.
    "Cite le numéro de la source [N] pour chaque affirmation.",
    '',
  ];
  hits.forEach((h, i) => {
    const title = h.document.title || h.document.id || 'Sans titre';
    const kind = h.sourceType === 'primary' ? 'archive'
      : h.sourceType === 'vault' ? 'note' : 'bibliographie';
    lines.push(`[${i + 1}] type : ${kind}`);
    lines.push(`TITRE : ${title}`);
    // ...AUTEUR (secondary/primary only, never for vault notes; `year` also
    // secondary-only), EXTRAIT (with a page number ONLY for secondary/PDF
    // hits — primary/archive hits get a bare "EXTRAIT :" with no locator at
    // all, not a photo number or item date as a substitute)
  });
  return lines.join('\n');
}
```

**Generated context example**:

```text
[1] type : bibliographie
TITRE : Active Learning in Higher Education
AUTEUR : Smith, 2019
EXTRAIT (p. 12) :
The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment...
```

### Step 6: LLM Generation

**Files**: `src/main/services/fusion-chat-service.ts` (transport, IPC `fusion:chat:*`) and `src/main/services/chat-engine.ts` (agent loop, retrieval hook, compaction). There is no separate English "You are a research assistant..." prompt template — the system message the model actually receives **is** the TITRE/AUTEUR/EXTRAIT block from Step 5 above, prepended to the conversation. The call goes through whichever provider is configured (`ProviderRegistry`), not a hardcoded Ollama call:

```typescript
const response = await provider.generate({
  messages: [
    { role: 'system', content: formatContextAsSystemPrompt(hits) },
    ...conversationHistory,
  ],
  stream: true,
});
```

**Streaming**: Response is sent token by token for better UX

**Stream Cancellation**: Users can cancel ongoing generation via a per-session `AbortController`.

### Context Compression

**File**: `backend/core/rag/ContextCompressor.ts`

**Wired into the retrieval path since RC4.** Before that it was reachable
from nothing but its own test: no service, IPC handler or chat path
imported it, and `retrieval-service.ts` never populated the field the rest
of the pipeline checks, so no context was ever compressed.

RC4 also fixed three ways it distorted the excerpts shown in the *Sources*
panel — which reads the compressed text, not the original. Markdown tables
were shredded (`|` doubled as the internal sentence boundary) and recomposed
into rows that never existed; sentences taken from far apart were joined
with no ellipsis; and the "keep the best two" fallback returned them in
score order, rewriting the text's chronology. Tables and code blocks now
pass through whole, cuts are marked `[…]`, and document order is preserved.


The design below is real (it's exactly what `ContextCompressor.ts`
implements) — this documents the *intended* algorithm, not something that
currently runs:

Sizes below are **characters**, not tokens — `ContextCompressor.ts`'s own
comments say so explicitly ("Skip compression for small contexts (< 10k
chars)").

```
┌─────────────────────────────────────────────────────────────┐
│              Context Compression Levels                      │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  No compression (context ≤ 10,000 chars)                      │
│    └── Passed through unchanged                               │
│                                                               │
│  Level 1: Light (context 15,000-25,000 chars)                 │
│    └── Semantic deduplication only (threshold 0.88)            │
│                                                               │
│  Level 2: Medium (context 25,000-35,000 chars)                 │
│    └── Deduplication (0.85) + relevant-sentence extraction     │
│                                                               │
│  Level 3: Aggressive (context > 35,000 chars)                  │
│    └── Aggressive deduplication (0.80) + full extraction       │
│                                                               │
└─────────────────────────────────────────────────────────────┘
```

Contexts between 10,000 and 15,000 characters would fall through all three
levels untouched even if this ran — there's a real gap between "skip" and
"Level 1", not a continuous scale.

**Algorithm, as implemented in the unused class**:
1. Calculate total context size in characters
2. Select compression level based on size thresholds
3. Apply relevance scoring to sentences (keyword + query similarity)
4. Preserve keyword-rich sentences
5. Return compressed context with reduction statistics

**`enableContextCompression`** is a per-mode config field (see
[RAG Configuration Defaults](#rag-configuration-defaults) below) that, per
the above, has no consumer either — toggling it in a mode has no observable
effect either way.

### RAG Explanation (Explainable AI)

**File**: `backend/types/chat-source.ts`

ClioDeck provides transparency about how RAG responses are generated. The
real shape is nested, not flat:

```typescript
interface RAGExplanation {
  search: {
    query: string;
    totalResults: number;
    searchDurationMs: number;
    cacheHit: boolean;
    sourceType: 'primary' | 'secondary' | 'both';
    documents: Array<{
      title: string;
      similarity: number;
      sourceType: string;
      chunkCount: number;
    }>;
  };
  compression?: {
    enabled: boolean;
    originalChunks: number;
    finalChunks: number;
    originalSize: number;
    finalSize: number;
    reductionPercent: number;
    strategy?: string;
  };
  graph?: {
    enabled: boolean;
    relatedDocsFound: number;
    documentTitles: string[];
  };
  llm: {
    provider: string;
    model: string;
    contextWindow: number;
    temperature: number;
    promptSize: number;
  };
  timing: {
    searchMs: number;
    compressionMs?: number;
    generationMs: number;
    totalMs: number;
  };
}
```

This information is displayed in the chat interface, allowing researchers to understand and evaluate the response context.

---

## External Integrations

### Zotero

**File**: `backend/integrations/zotero/ZoteroAPI.ts` (not `zotero-service.ts`, which only orchestrates — the actual `fetch()` calls and headers live here)

```typescript
// 1. Authentication
const headers = {
  'Zotero-API-Key': apiKey,
  'Zotero-API-Version': '3'
};

// 2. Retrieve collections
const collections = await fetch(
  `https://api.zotero.org/users/${userId}/collections`,
  { headers }
);

// 3. Retrieve items from a collection
const items = await fetch(
  `https://api.zotero.org/users/${userId}/collections/${collectionId}/items`,
  { headers }
);

// 4. BibTeX export
const bibtex = await fetch(
  `https://api.zotero.org/users/${userId}/collections/${collectionId}/items?format=bibtex`,
  { headers }
);

// 5. Download PDFs
for (const item of items) {
  if (item.data.linkMode === 'linked_file') {
    const pdf = await downloadAttachment(item.key);
    await savePDF(pdf, projectPath);
  }
}
```

### BibTeX

**Parsing**: `backend/core/bibliography/BibTeXParser.ts` — a 484-line
`BibTeXParser` class, not a standalone regex-based function. The
illustrative snippet in earlier versions of this page (a ~20-line
single-regex function) drastically understated it; there is no
`parseBibTeX` function anywhere in the codebase. Custom-field round-trip
(preserving fields BibTeX doesn't natively model) is handled here, matched
by `BibTeXExporter.ts` on the way back out.

### Topic Modeling (Python)

**Service**: `backend/python-services/topic-modeling/main.py`

**The Python service does not compute embeddings itself** — an earlier
version of this page showed it calling `SentenceTransformer(...).encode()`
internally, which contradicts the module's own docstring ("Reçoit des
embeddings **pré-calculés** depuis Electron/Ollama") and its request
schema, which requires `embeddings` as a mandatory input field
(`min_length=5`) alongside `documents` and `document_ids`. Embeddings come
from the TypeScript side, via whichever `EmbeddingProvider` is configured
(see Step 3 above) — `sentence-transformers` is in `requirements.txt` only
as a BERTopic transitive dependency, never imported directly in `main.py`.

```python
from bertopic import BERTopic
from fastapi import FastAPI
import numpy as np

app = FastAPI()

@app.post("/analyze")
async def analyze_corpus(request: AnalyzeRequest):
    # Embeddings arrive pre-computed — no SentenceTransformer call here
    embeddings_array = np.array(request.embeddings, dtype=np.float32)

    topic_model = BERTopic(min_topic_size=request.min_topic_size)
    topics, _ = topic_model.fit_transform(request.documents, embeddings_array)

    return {
        "topics": topics,               # per real AnalyzeResponse shape
        "topic_assignments": {...},     # document_id -> topic id
        "outliers": [...],
        "statistics": {...},
    }
```

**Communication**: TypeScript backend communicates via HTTP (port 8001)

```typescript
const response = await fetch('http://localhost:8001/analyze', {
  method: 'POST',
  body: JSON.stringify({
    embeddings,     // required — computed TypeScript-side first
    documents: chunks,
    document_ids: ids,
  })
});

// Real response shape: {topics, topic_assignments, outliers, statistics}
// — not {topics, topic_info} as an earlier version of this page showed.
const { topics, topic_assignments, outliers, statistics } = await response.json();
```

---

## Performance and Optimizations

### Memory Footprint (typical 50k chunks)

| Component | RAM | Description |
|-----------|-----|-------------|
| Electron | 1.5 GB | App + Chromium |
| HNSW index | 500 MB | 50k × 768 dims × 3x overhead |
| BM25 index | 100 MB | Inverted index + vocabulary |
| SQLite | 300 MB | Chunks + metadata |
| Embedding provider | 0–500 MB | Ollama's `nomic-embed-text` loads ~500 MB in RAM locally; the embedded model and cloud providers have a different footprint (embedded: bundled with the app; cloud: none, runs remotely) |
| **Total** | **~3 GB** | For an average project |

### Recommended Configuration

| Corpus Size | Min RAM | CPU | Indexing Time |
|-------------|---------|-----|---------------|
| 10-50 PDFs | 4 GB | Dual-core | 5-10 min |
| 50-200 PDFs | 8 GB | Quad-core | 20-40 min |
| 200-500 PDFs | 16 GB | 8+ cores | 1-2 hours |

### CPU Optimizations

**1. Parallel chunking**:
```typescript
// Process documents in parallel (max 4 simultaneous)
const batches = chunk(documents, 4);
for (const batch of batches) {
  await Promise.all(batch.map(doc => indexDocument(doc)));
}
```

**2. Batch embeddings**:
```typescript
// Generate embeddings in batches of 32 — illustrative strategy, not a
// literal citation; the real batched call is generateEmbeddings(texts)
// on the active EmbeddingProvider (see Step 3 above), not an
// ollama.batchEmbed() method (no such method exists).
for (let i = 0; i < chunks.length; i += 32) {
  const batch = chunks.slice(i, i + 32);
  const embeddings = await embeddingProvider.generateEmbeddings(batch);
}
```

**3. Background index building** — illustrative strategy; `HNSWVectorStore.ts` has no `rebuild()` method, this section describes a deferred-work pattern in general terms, not a real call:
```typescript
setTimeout(() => {
  // build/refresh the HNSW index here
}, 5000);
```

### Chunking Configurations

```typescript
const CHUNKING_CONFIGS = {
  // For modest machines (8 GB RAM)
  cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },

  // Balance performance/precision
  standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },

  // For maximum precision (16+ GB RAM)
  large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};
```

(Same `maxTokens: 8192` fabrication as above — real config has no such field, see the note there.)

---

## Key Architecture Files

```
backend/
├── core/
│   ├── analysis/
│   │   └── TopicModelingService.ts      # Thematic analysis + graph
│   ├── chunking/
│   │   ├── AdaptiveChunker.ts           # Structure-aware chunking
│   │   └── DocumentChunker.ts           # Simple chunking (fallback)
│   ├── history/
│   │   └── HistoryManager.ts            # Research journal
│   ├── llm/
│   │   └── providers/                   # Typed provider registry
│   │       ├── base.ts                  # LLMProvider / EmbeddingProvider contracts
│   │       ├── registry.ts              # Open factory map, 5 providers + embedded
│   │       ├── ollama.ts, embedded.ts, anthropic.ts,
│   │       │   openai-compatible.ts, mistral.ts, gemini.ts
│   │       ├── fallback.ts              # Ollama → embedded auto-fallback
│   │       └── cliodeck-config-adapter.ts
│   ├── pdf/
│   │   ├── PDFExtractor.ts              # PDF extraction (pdfjs) — the actual getDocument/getTextContent/getMetadata calls
│   │   └── PDFIndexer.ts                # Orchestrates extraction + indexing, delegates to PDFExtractor.ts
│   ├── search/
│   │   ├── BM25Index.ts                 # Sparse index (keywords)
│   │   └── HybridSearch.ts              # Dense + sparse fusion
│   └── vector-store/
│       ├── VectorStore.ts               # SQLite base store
│       ├── HNSWVectorStore.ts           # Fast HNSW index
│       └── EnhancedVectorStore.ts       # Unified wrapper
│
├── python-services/
│   └── topic-modeling/
│       ├── main.py                      # FastAPI service
│       ├── topic_analyzer.py            # BERTopic logic
│       └── requirements.txt             # Python dependencies
│
├── recipes/
│   ├── schema.ts, runner.ts             # ClioRecipes: YAML workflow steps
│   └── builtin/                         # Zotero review, Tropy analysis, etc.
│
├── types/
│   ├── config.ts                        # Configuration types
│   ├── book.ts                          # Book/chapters manifest + settings
│   └── pdf-document.ts                  # Document/chunk types
│
src/
├── editor/
│   ├── cm/                              # CodeMirror 6 state, fidelity, proposals
│   ├── facade.ts                        # Editor-agnostic abstraction (EditorFacade)
│   ├── proposals/                       # Adjudicable AI-proposal contract
│   └── slides.ts                        # Shared slide-splitting logic
│
├── main/
│   ├── services/
│   │   ├── fusion-chat-service.ts       # Unified chat transport (IPC fusion:chat:*)
│   │   ├── chat-engine.ts               # Shared agent loop, compaction, retrieval hook
│   │   ├── retrieval-service.ts         # Multi-source RAG (PDFs + Tropy + Obsidian + manuscript)
│   │   ├── manuscript-assembler.ts      # Book export assembly (per-chapter footnote IDs)
│   │   ├── manuscript-index-service.ts  # Manuscript as a 4th RAG corpus
│   │   ├── mcp-clients-service.ts       # External MCP server lifecycle
│   │   ├── usage-journal-service.ts     # AI usage journal sink (.cliodeck/journal.db)
│   │   ├── pdf-service.ts               # PDF management
│   │   ├── tropy-service.ts             # Primary sources (Tropy)
│   │   ├── project-manager.ts           # Project management
│   │   └── topic-modeling-service.ts    # Python service proxy
│   └── ipc/
│       └── handlers/                    # Electron IPC
│
└── renderer/
    ├── components/
    │   ├── Chat/                        # AssistantChat (single shell, variants full/panel)
    │   ├── Config/                      # Settings
    │   ├── Editor/                      # CodeMirror 6 editor
    │   ├── Brainstorm/                  # Chat + Ideas + Board + Graph tabs
    │   ├── PDFIndex/                    # PDF management
    │   └── Journal/                     # Research journal
    └── stores/
        └── editorStore.ts               # Zustand state
```

`backend/core/llm/OllamaClient.ts` and `src/main/services/chat-service.ts`,
which older versions of this page listed here, no longer exist — replaced by
the provider registry and `fusion-chat-service.ts`/`chat-engine.ts` above.

---

## Primary Sources Integration (Tropy)

### Architecture

`PrimarySourcesVectorStore` handles Tropy primary sources, but it stores its
SQLite data in the **shared** `.cliodeck/brain.db` (the same file PDFs and
research history use — see [Key Architecture Files](#key-architecture-files)
above), not a standalone database. Only the HNSW index is its own file:

```
┌─────────────────────────────────────────────────────────────┐
│                    Primary Sources Pipeline                   │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  Tropy Project (.tropy / .tpy)                                │
│         │                                                     │
│         ▼                                                     │
│  ┌──────────────┐                                            │
│  │ TropyReader  │ ──► Read-only access to SQLite database    │
│  └──────────────┘                                            │
│         │                                                     │
│         ▼                                                     │
│  ┌──────────────┐                                            │
│  │  TropySync   │ ──► Sync items, photos, notes, tags        │
│  └──────────────┘                                            │
│         │                                                     │
│         ├──► OCR Pipeline (Tesseract.js) for images          │
│         │                                                     │
│         ▼                                                     │
│  ┌────────────────────────┐                                  │
│  │ PrimarySourcesVectorStore │                               │
│  │   - SQLite: brain.db (shared)  │                          │
│  │   - HNSW: primary-hnsw.index   │                          │
│  │   - BM25: in-memory        │                               │
│  └────────────────────────┘                                  │
│                                                               │
└─────────────────────────────────────────────────────────────┘
```

### Data Flow

1. **TropyReader** opens the `.tpy` file in **read-only** mode
2. **TropySync** extracts items with metadata, notes, and photos
3. Transcription source priority, per item (`TropySync.ts`): **Tropy notes**
   first (if the item has any actual note content, not just metadata) →
   then an **external transcription** (Transkribus or similar, if a
   `transcriptionDirectory` is configured) → then **OCR** (Tesseract.js), if
   nothing else produced text and OCR is enabled. An earlier version of this
   page said "notes > OCR > manual" — wrong on two counts: the real
   second step is the external-transcription source, not OCR, and "manual"
   isn't part of this automatic chain at all — it's a separate, per-source
   *re-run* action, reachable from the source card since RC4, not a
   fallback this pipeline ever reaches on its own.
4. **`DocumentChunker`** (fixed-size) segments transcriptions —
   **hardcoded** in `tropy-service.ts`, not "the same as PDFs": PDFs default
   to `AdaptiveChunker` (structure-aware) via `useAdaptiveChunking: true` in
   `DEFAULT_CONFIG.rag`, and Tropy has no equivalent option to opt into it.
   Primary sources always get fixed-size chunking regardless of that
   setting.
5. The configured embedding provider generates embeddings (Ollama's `nomic-embed-text`, the embedded model, or a cloud provider)
6. **PrimarySourcesVectorStore** stores chunks with HNSW + BM25 indexes

### Unified Search

The snippet below is not a literal quote — an earlier version of this page
presented it as one (`// In retrieval-service.ts`, no "illustrative" caveat,
unlike other pseudo-code elsewhere on this page). The real code in
`retrieval-service.ts` is a per-source-type **partial-success** pattern, not
a naive merge-and-sort: secondary and primary searches each run in their own
try/catch, each tracking its own `ok`/`error`/`durationMs` outcome, and
results are pushed into a shared array as they succeed — one source type
failing doesn't drop the other's results, and there's no similarity-sort
merge step shown below actually happening at this layer:

```typescript
// Illustrative only — NOT a literal quote from retrieval-service.ts
const pdfResults = await vectorStore.search(embedding, topK);
const tropyResults = await tropyService.search(query, { topK });

// Merge and deduplicate
const allResults = [...pdfResults, ...tropyResults]
  .sort((a, b) => b.similarity - a.similarity)
  .slice(0, topK);
```

`tropyService.search()` itself runs its own internal hybrid HNSW+BM25 search
with RRF scoring and FR/EN multilingual query expansion — the same
expansion mechanism `SecondaryRetriever` uses for PDFs, reused via
`expandQueryToText()` rather than duplicated.

---

## Embedded LLM Support

ClioDeck supports embedded LLMs via `node-llama-cpp` for fully offline
generation *and* embeddings — see the [Embedded LLM Guide](./1.7-Embedded-LLM-Guide.md)
for the canonical, up-to-date description (model sizes, prompt format,
troubleshooting). That page is the one to update if this changes again;
this section only sketches where it sits in the architecture.

The embedded provider is one entry in the same typed `ProviderRegistry`
used by Ollama and the four cloud providers (`backend/core/llm/providers/`)
— there is no separate "manager" class layered on top, and no requirement
to have Ollama installed at all. `embedded` is a valid, independent value for **both** `generationProvider`
and `embeddingProvider` in config (`cliodeck-config-adapter.ts`). Both have
a UI control since RC4: Settings → LLM for generation, Settings →
Embeddings for the embedding provider and model. Before RC4, reaching
`embedded` for embeddings meant hand-editing the config file. See the
[Embedded LLM Guide](./1.7-Embedded-LLM-Guide.md).

Changing the embedding provider invalidates every existing index — vectors
from one model cannot be compared with another's, and the comparison raises
no error. Since RC4 the app says so before applying the change, and rebuilds
the provider registry immediately instead of waiting for a restart.
An `auto` mode also exists for each, which prefers Ollama when reachable
and falls back to the embedded model (`backend/core/llm/providers/fallback.ts`).
For embeddings specifically, the `auto` fallback only engages if the
embedded and Ollama embedding models share the same vector dimension
(`cliodeck-config-adapter.ts`'s `embeddingWithFallback()`) — mixing
dimensions would silently corrupt similarity search, so a mismatch is
left to fail loudly instead.

---

## RAG Configuration Defaults

Current default values live on the `rag` key of `DEFAULT_CONFIG` in
`backend/types/config.ts` — there is no separately-named `DEFAULT_RAG_CONFIG`
constant, just this nested object:

```typescript
const DEFAULT_CONFIG = {
  // ...
  rag: {
    // Search parameters
    topK: 10,                          // Number of chunks to retrieve
    similarityThreshold: 0.12,         // RRF-optimized (not cosine similarity)

    // Chunking
    chunkingConfig: 'cpuOptimized',    // 300 words, 50 overlap
    useAdaptiveChunking: true,

    // Indexes
    useHNSWIndex: true,
    useHybridSearch: true,             // HNSW + BM25

    // Quality
    enableQualityFiltering: true,
    enablePreprocessing: true,
    enableDeduplication: true,

    // Advanced (optional)
    useSemanticChunking: false,
  },
};
```

**`enableContextCompression` is not part of this object at all.** An earlier
version of this page listed it here as a single global default (`true`) —
in the real code it's a per-**mode** override, set in
`backend/core/modes/built-in-modes.ts`: `true` for `primary-source-analyst`,
`academic-writer`, and `methodology-assistant`; `false` for
`literature-review`, `critical-reviewer`, and `brainstorming`; unset for
`default-assistant` and `free-mode`. Before RC4 none of this mattered: the
class the flag toggles was never called, so the flag had no observable
effect whatever a mode set it to. It is wired now — see
[Context Compression](#context-compression) above.

### Similarity Threshold Note

The default `similarityThreshold` of 0.12 is optimized for RRF (Reciprocal Rank Fusion) scores, which are much smaller than raw cosine similarity scores. Typical RRF scores range from 0.001 to 0.02.

---

## Technical References

### Papers

- **HNSW**: [Efficient and robust approximate nearest neighbor search](https://arxiv.org/abs/1603.09320) (Malkov & Yashunin, 2016)
- **BM25**: [The Probabilistic Relevance Framework: BM25 and Beyond](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf) (Robertson & Zaragoza, 2009)
- **RRF**: [Reciprocal rank fusion outperforms condorcet](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) (Cormack et al., 2009)
- **RAG**: [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) (Lewis et al., 2020)

### Tools and Libraries

- [Ollama](https://ollama.ai/) - Local LLMs
- [node-llama-cpp](https://github.com/withcatai/node-llama-cpp) - Embedded LLM execution
- [hnswlib](https://github.com/nmslib/hnswlib) - HNSW implementation
- [natural](https://github.com/NaturalNode/natural) - NLP for Node.js
- [BERTopic](https://maartengr.github.io/BERTopic/) - Topic modeling
- [pdfjs-dist](https://mozilla.github.io/pdf.js/) - PDF rendering
- [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) - Synchronous SQLite
- [Tesseract.js](https://tesseract.projectnaptha.com/) - OCR for primary sources
- [Tropy](https://tropy.org/) - Primary sources management

---

**Version**: 1.0.0-rc.4