Skip to content

2. Technical Architecture

Jump to bottom
Frédéric Clavert edited this page Jul 26, 2026 · 19 revisions

This document describes the architecture and operation of ClioDeck's RAG (Retrieval-Augmented Generation) system, an academic writing assistant for historians and humanities researchers.

Table of Contents

Overview

ClioDeck is an Electron application that combines:

  • Frontend: React/TypeScript interface with a CodeMirror 6 editor (live markdown rendering)
  • Backend: Node.js/TypeScript services for indexing and search
  • Python Services: Topic modeling and textometric analysis
  • LLM: a typed provider registry — Ollama, embedded (node-llama-cpp), Anthropic, OpenAI-compatible, Mistral, Gemini — for embeddings and text generation
  • Storage: SQLite for data and vector indexes

Design Goals

  1. Local-first: All data remains on the user's machine
  2. Performance: Optimized for modest machines (8-16 GB RAM, CPU only)
  3. Academic: Complete traceability and verifiable citations
  4. Interoperability: Integration with Zotero, BibTeX, Markdown, PDF

Global Architecture

┌─────────────────────────────────────────────────────────────┐
│                    ClioDeck Electron App                      │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Frontend   │  │   Backend    │  │   Services   │      │
│  │  React/Vite  │◄─┤  TypeScript  │◄─┤    Python    │      │
│  │              │  │              │  │              │      │
│  │ • Editor     │  │ • PDF Index  │  │ • Topic      │      │
│  │ • Chat UI    │  │ • Vector DB  │  │   Modeling   │      │
│  │ • Config     │  │ • Search     │  │ • BERTopic   │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                            │                │                │
│                            ▼                ▼                │
│                    ┌──────────────┐  ┌──────────────┐      │
│                    │   Ollama     │  │   SQLite     │      │
│                    │   (Local)    │  │ + HNSW Index │      │
│                    └──────────────┘  └──────────────┘      │
│                                                               │
└─────────────────────────────────────────────────────────────┘

Key Technologies

Component Technology Role
UI Framework React 18 + TypeScript User interface
Editor CodeMirror 6 + Lezer Live markdown rendering, Pandoc dialect
Build Tool Vite Bundling and hot reload
Desktop Electron 40 Multi-platform desktop app
Database better-sqlite3 Synchronous local storage
Vector Index hnswlib-node Fast vector search
Sparse Index natural (BM25) Keyword search
LLM Typed provider registry Ollama, embedded, Anthropic, OpenAI-compatible, Mistral, Gemini
PDF Processing pdfjs-dist Text extraction
Bibliography BibTeX parsing Reference management
Topic Modeling Python + BERTopic Thematic analysis

Indexing Pipeline

The indexing pipeline transforms PDFs into queryable vector chunks.

Step 1: PDF Extraction

File: backend/core/pdf/PDFExtractor.ts (not PDFIndexer.ts, which only orchestrates and delegates here)

// 1. Load PDF with pdfjs
const pdf = await pdfjs.getDocument(pdfPath).promise;

// 2. Extract text page by page
const pages = [];
for (let i = 1; i <= pdf.numPages; i++) {
  const page = await pdf.getPage(i);
  const textContent = await page.getTextContent();
  const text = textContent.items.map(item => item.str).join(' ');
  pages.push({ pageNumber: pageNum, text });  // field is `text`, not `content` — PDFExtractor.ts:174-176
}

// 3. Extract metadata
const metadata = await pdf.getMetadata();
const title = metadata.info.Title || path.basename(pdfPath);

Output: Array of pages with raw text + document metadata

This code never runs directly inside the Electron main process. Real PDF indexing (src/main/services/pdf/PdfIndexer.ts, pdf-service.ts) calls extractPdfIsolated() (src/main/services/pdf-extract-isolated.ts), which spawns the system Node binary as a child process and runs PDFExtractor inside a separate worker script — a workaround for a pdfjs-dist crash on Electron's bundled Node. Three real, previously undocumented consequences for indexing a large PDF:

  • 120-second hard timeout per PDF. A pathological or very large scan that takes longer fails with "PDF extraction timed out after 120s" — there is no way to raise this from the UI.
  • Global sequential queue, one extraction at a time, app-wide. extractPdfIsolated() chains every call onto a single pending promise specifically to bound RAM — importing a large batch of PDFs (e.g. a big Zotero library) processes them strictly one by one, never in parallel, regardless of CPU cores available.
  • Requires a system Node 20+ binary on PATH (or at a few hardcoded Homebrew/nvm paths) separate from Electron's own bundled Node — if none is found, extraction fails with an explicit "requires a system Node.js binary" error rather than a silent hang.

A fourth consequence, real bug, verified against code: that 120-second window (or longer, queued behind other pending PDFs in the same sequential lane) is plenty of time for a user to switch ClioDeck projects. Unlike the Zotero/Tropy/Obsidian bugs found elsewhere in this audit (issues #33/#34/#35 — those misattribute data to the wrong project), PDF indexing does NOT have that problem: PdfIndexer (src/main/services/pdf/PdfIndexer.ts) receives its vectorStore through constructor injection, captured once, so an in-flight indexPDF() call keeps writing to the correct (original) project's store even after a switch — no cross-project data corruption. But pdf-service.ts's init(projectPath) — called on every project switch — unconditionally does this.vectorStore.close() on the previous project's store before setting up the new one, with no check for an in-flight indexing operation still using that exact same store instance. PdfIndexer.indexPDF() writes to this.vectorStore again after the (possibly 120-second) extraction finishes (setDocumentCollections(...), plus the embedding/chunk-storage steps inside this.backend.indexPDF(...)) — against a better-sqlite3 handle that's by then closed. The practical effect: switching projects while a PDF was indexing made the job throw once it tried to write its results. Fixed in RC4 — in-flight indexing is drained before the store closes. The manuscript index service, written in the same cycle, reproduced the same defect and was fixed with it.

Step 2: Adaptive Chunking

Files:

  • backend/core/chunking/AdaptiveChunker.ts (recommended)
  • backend/core/chunking/DocumentChunker.ts (fallback)
  • backend/core/chunking/SemanticChunker.ts (advanced semantic-aware chunking)
  • backend/core/chunking/ChunkDeduplicator.ts (removes near-duplicate chunks)
  • backend/core/chunking/ChunkQualityScorer.ts (scores chunk quality for filtering)
  • backend/core/chunking/EmbeddingCache.ts (caches embeddings for performance)

Structure Detection

The adaptive chunker automatically detects document structure:

// Header detection — patterns as they actually are in AdaptiveChunker.ts,
// stricter than earlier versions of this page implied
const patterns = {
  markdown: /^#{1,6}\s+(.+)/,                          // # Title, ## Subtitle
  numbered: /^(\d+(?:\.\d+)*)\.\s+([A-Z][^.]{2,50})$/, // "1. Introduction" — needs a trailing dot and a capitalized 2-50 char title
  uppercase: /^([A-Z][A-Z\s]{5,50})$/,                 // ALL CAPS, 6-51 chars, also checked against a known-header allowlist (ABSTRACT, INTRODUCTION, …)
  roman: /^([IVX]+)\.\s+([A-Z][^.]{2,50})$/            // "I. Introduction" — same capitalized-title constraint as `numbered`, not a bare "I. anything" match
};

// Automatic classification
const sectionTypes = {
  abstract: /abstract|résumé/i,
  introduction: /introduction/i,
  methodology: /method|méthodologie/i,
  results: /results|résultats/i,
  discussion: /discussion/i,
  conclusion: /conclusion/i,
  references: /references|bibliographie/i
};

Chunk Creation

Principles:

  1. Respect sentence boundaries: Chunks always end with ., !, ? or ;
  2. Document context: Each chunk includes title and section as prefix
  3. Skip references: Bibliography section is not indexed
  4. Smart overlap: Overlaps occur on complete sentences

Generated chunk example:

[Doc: Active Learning Strategies in Higher Education | Section: Methodology]

The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment. Students were divided
into control and experimental groups. The control group followed traditional
lecture-based instruction, while the experimental group participated in
collaborative problem-solving activities.

Configuration:

interface ChunkingConfig {
  maxChunkSize: number;     // Target max size in words (300-800)
  overlapSize: number;      // Overlap in words (50-100)
  minChunkSize: number;     // Minimum words, avoids tiny leftover chunks
}

const CHUNKING_CONFIGS = {
  cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },
  standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },
  large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};

There is no maxTokens field anywhere in this config — an earlier version of this page invented one; token limits for embedding models are handled separately, not as part of chunking config.

Output: Array of chunks with metadata

interface DocumentChunk {
  id: string;                 // UUID
  documentId: string;         // Document reference
  content: string;            // Chunk text (with context)
  pageNumber: number;         // Source page
  chunkIndex: number;         // Position in document
  startPosition: number;      // Character offset, start
  endPosition: number;        // Character offset, end
  // No `embedding` field here — embeddings live in the DB row
  // (pdf_chunks.embedding) or the vector store, not on this object.
  metadata?: {
    sectionTitle?: string;    // "Methodology", "Results"
    sectionType?: string;     // "methodology", "results"
    sectionLevel?: number;    // 1, 2, 3...
  };
}

Step 3: Embedding Generation

File: backend/core/llm/providers/ollama.ts (one of six embedding providers behind the typed registry — see Embedded LLM Support below for the others)

// 1. Connect to Ollama — the MODERN /api/embed endpoint, batched, not the
// legacy /api/embeddings single-prompt endpoint an earlier version of this
// page showed. ollama.ts's own comment explains why: "the legacy
// /api/embeddings endpoint ignores `truncate` and 500s — which is exactly
// the PDF-indexing crash this replaces."
const response = await fetch('http://localhost:11434/api/embed', {
  method: 'POST',
  body: JSON.stringify({
    model: 'nomic-embed-text',  // 768 dimensions
    input: [chunkContent, /* ...more chunks, batched */],
    truncate: true
  })
});

// 2. Retrieve vectors (one per input, in order)
const { embeddings } = await response.json();
// embeddings: number[][] (768 dimensions each)

Model used: nomic-embed-text

  • Dimensions: 768
  • Max context: 8192 tokens (model-dependent — nomic-embed-text itself advertises 2048 and ignores larger num_ctx values)
  • Size: ~274 MB
  • Performance: ~100 embeddings/second on average CPU

Oversized input, Ollama path: inputs that overflow the model's context are truncated server-side by Ollama itself (truncate: true in the /api/embed call above) — ollama.ts does no client-side chunking of its own; the comment in the file is explicit that server-side truncation happens "rather than" splitting the text.

A second, earlier safety net exists specifically for PDF indexing. src/main/services/pdf/PdfIndexer.ts slices any chunk text to EMBED_CHAR_CAP = 6000 characters before it ever reaches the embedding provider — independent of, and applied before, whichever of the two paths above eventually runs. Its own comment explains why: AdaptiveChunker can occasionally emit a single oversized chunk (a paragraph longer than maxChunkSize), and without this cap that one pathological chunk used to abort the entire indexing run with an HTTP 500 ("input length exceeds the context length") from Ollama — this cap is what prevents that, not truncate: true alone.

Oversized input, embedded-model path: the chunk-and-average chunkText() method shown below is real, but it belongs to EmbeddedLLMClient.ts (the node-llama-cpp embedded provider), not Ollama — an earlier version of this page misattributed it here. Its threshold is EMBEDDING_MAX_CHUNK_LENGTH = 2000 characters, not "8192 tokens":

// EmbeddedLLMClient.ts — embedded provider only, not Ollama
private chunkText(text: string, maxLength: number): string[] {
  // Find sentence ending in last 200 characters
  const sentenceEndings = /[.!?;](?=\s|$)/g;
  // ... cut at last found period
}
// Chunks are embedded individually, then averaged into a single vector.

Step 4: Multi-Strategy Indexing

File: backend/core/vector-store/EnhancedVectorStore.ts

Three indexes are created simultaneously:

Index 1: SQLite (database)

-- Real schema, VectorStore.ts:104-113 — table is pdf_chunks, not chunks;
-- no metadata/created_at columns; start_position/end_position instead
CREATE TABLE IF NOT EXISTS pdf_chunks (
  id TEXT PRIMARY KEY,
  document_id TEXT NOT NULL,
  content TEXT NOT NULL,
  page_number INTEGER NOT NULL,
  chunk_index INTEGER NOT NULL,
  start_position INTEGER NOT NULL,
  end_position INTEGER NOT NULL,
  embedding BLOB,
  FOREIGN KEY (document_id) REFERENCES pdf_documents(id) ON DELETE CASCADE
);

CREATE INDEX IF NOT EXISTS idx_chunks_document_id ON pdf_chunks(document_id);
CREATE INDEX IF NOT EXISTS idx_chunks_page_number ON pdf_chunks(page_number);

Role: Persistent storage + fallback for linear search

Index 2: HNSW (fast vector search)

File: backend/core/vector-store/HNSWVectorStore.ts

import { HierarchicalNSW } from 'hnswlib-node';

const index = new HierarchicalNSW('cosine', 768); // 768 dimensions
index.initIndex(maxElements, M=16, efConstruction=100);

// Indexing
for (const chunk of chunks) {
  index.addPoint(chunk.embedding, chunk.id);
}

// Save to disk
index.writeIndexSync(indexPath);

Parameters:

  • M=16: Connections per node (speed/precision trade-off)
  • efConstruction=100: Construction effort
  • efSearch=50: Search effort

Complexity:

  • Construction: O(n log n)
  • Search: O(log n)

Memory footprint: ~500 MB for 50k chunks (768 dims)

Index 3: BM25 (keyword search)

File: backend/core/search/BM25Index.ts

There is no hand-rolled inverted index — indexing delegates entirely to natural's TfIdf class, with BM25 scoring layered on top of it:

import natural from 'natural';
const { TfIdf } = natural;

// 1. Delegate document storage/tokenization to natural's TfIdf
const tfidf = new TfIdf();
tfidf.addDocument(preprocessText(chunk.content));

// 2. BM25 scoring, computed from TfIdf's term/document frequencies
const idf = Math.log((totalDocs - docFreq + 0.5) / (docFreq + 0.5));
const score = idf * ((tf * (k1 + 1)) / (tf + k1 * (1 - b + b * (docLength / avgDocLength))));

Parameters:

  • k1=1.5: Term frequency saturation
  • b=0.75: Document length normalization

Memory footprint: ~100 MB for 50k chunks

Search System

Hybrid Search (Dense + Sparse)

File: backend/core/search/HybridSearch.ts

Hybrid search combines the advantages of two complementary approaches:

Query: "bloom taxonomy methodology"
        │
        ├────────────────────┬────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
  ┌──────────┐        ┌──────────┐        ┌──────────┐
  │  Dense   │        │  Sparse  │        │  Fusion  │
  │  (HNSW)  │        │  (BM25)  │        │  (RRF)   │
  └──────────┘        └──────────┘        └──────────┘
        │                    │                    │
        ▼                    ▼                    ▼
   Top-50 results       Top-50 results       Top-10 final
   (semantic)           (keywords)           (best of both)

Step 1: Dense Search (HNSW)

// 1. Generate query embedding via the configured EmbeddingProvider
// (Ollama, embedded, or a cloud provider — see Embedded LLM Support below)
const queryEmbedding = await embeddingProvider.generateEmbedding(query);

// 2. HNSW search
const denseResults = await hnswStore.search(queryEmbedding, 50);
// Returns: [{ id, score }, ...]

Advantage: Understands semantic meaning ("active learning" ≈ "apprentissage actif")

Step 2: Sparse Search (BM25)

// 1. Query tokenization
const queryTokens = tokenizer.tokenize(query.toLowerCase());

// 2. BM25 scoring
const sparseResults = bm25Index.search(queryTokens, 50);
// Returns: [{ id, score }, ...]

Advantage: Excellent for rare terms, proper nouns, acronyms ("BERTopic", "Piaget")

Step 3: RRF Fusion (Reciprocal Rank Fusion)

function reciprocalRankFusion(
  denseResults: SearchResult[],
  sparseResults: SearchResult[],
  k: number = 60
): SearchResult[] {
  const scores = new Map<string, number>();

  // Dense search contribution (60%)
  for (let i = 0; i < denseResults.length; i++) {
    const chunkId = denseResults[i].id;
    const rrfScore = 0.6 / (k + i + 1);
    scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
  }

  // Sparse search contribution (40%)
  for (let i = 0; i < sparseResults.length; i++) {
    const chunkId = sparseResults[i].id;
    const rrfScore = 0.4 / (k + i + 1);
    scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
  }

  // Sort by final score
  return Array.from(scores.entries())
    .sort((a, b) => b[1] - a[1])
    .slice(0, 10)
    .map(([id, score]) => ({ id, score }));
}

Weights: 60% dense / 40% sparse (configurable)

RRF Formula: RRF(d) = Σ weight_i / (k + rank_i(d))

Exact match boost: HybridSearch.ts multiplies a chunk's RRF score by EXACT_MATCH_BOOST when the query string appears verbatim in it — RRF scores alone peak too low (top-1 ≈ 1/(K+1) ≈ 0.016) to otherwise outrank strong semantic neighbours scoring 0.45–0.55.

Performance

Metric Linear Search HNSW Only Hybrid (HNSW+BM25)
Time 500ms 15ms 30ms
Precision@10 65% 75% 80%
Recall@10 45% 55% 60%

Assisted Generation (RAG)

Complete Pipeline

User Query
    │
    ├─► 1. Generate embedding
    │       │
    │       ▼
    ├─► 2. Hybrid search (HNSW + BM25)
    │       │
    │       ▼
    ├─► 3. Retrieve top-K chunks (K=10)
    │       │
    │       ▼
    ├─► 4. Optional: Graph expansion
    │       │
    │       ▼
    ├─► 5. Build context
    │       │
    │       ▼
    └─► 6. Generate with LLM
            │
            ▼
        Response with citations

Steps 1-3: Search (see previous section)

Step 4: Graph Expansion — declared but dead

useGraphContext, graphSimilarityThreshold, and additionalGraphDocs are declared in backend/types/config.ts and set by mode presets, and even exposed in the RAG settings UI — but confirmed to have zero consumers anywhere in retrieval-service.ts or backend/core/rag/. Turning this setting on currently has no effect on retrieval; it isn't wired to anything. Documented here as a known gap, not as working behavior — the mechanism described in earlier versions of this page (graph-neighbor lookup, cosine similarity expansion) does not exist in the code today.

Step 5: Context Building

File: src/main/services/fusion-chat-service.ts, formatContextAsSystemPrompt(). An earlier version of this page showed a fabricated Markdown-based builder (# Context (sources), ## [Title, p.X] headers) that doesn't match the real code at all — and contradicted this wiki's own Brainstorm guide, which correctly describes the real format. The actual builder produces numbered TITRE/AUTEUR/EXTRAIT blocks, in French, with an explicit rule forcing the model to answer only from those fields:

function formatContextAsSystemPrompt(hits: MultiSourceSearchResult[]): string {
  const SNIPPET_CHARS = 1500;
  const lines: string[] = [
    'Contexte extrait du corpus indexé (sources numérotées ci-dessous).',
    "RÈGLE : réponds UNIQUEMENT à partir des sources ci-dessous (champs TITRE, AUTEUR et EXTRAIT). Ne complète JAMAIS avec tes connaissances générales.",
    // ...two more rule lines (TITRE-as-valid-answer for identification
    // questions, explicit no-canned-phrase wording for "not found") —
    // omitted here for brevity, present in the real function.
    "Cite le numéro de la source [N] pour chaque affirmation.",
    '',
  ];
  hits.forEach((h, i) => {
    const title = h.document.title || h.document.id || 'Sans titre';
    const kind = h.sourceType === 'primary' ? 'archive'
      : h.sourceType === 'vault' ? 'note' : 'bibliographie';
    lines.push(`[${i + 1}] type : ${kind}`);
    lines.push(`TITRE : ${title}`);
    // ...AUTEUR (secondary/primary only, never for vault notes; `year` also
    // secondary-only), EXTRAIT (with a page number ONLY for secondary/PDF
    // hits — primary/archive hits get a bare "EXTRAIT :" with no locator at
    // all, not a photo number or item date as a substitute)
  });
  return lines.join('\n');
}

Generated context example:

[1] type : bibliographie
TITRE : Active Learning in Higher Education
AUTEUR : Smith, 2019
EXTRAIT (p. 12) :
The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment...

Step 6: LLM Generation

Files: src/main/services/fusion-chat-service.ts (transport, IPC fusion:chat:*) and src/main/services/chat-engine.ts (agent loop, retrieval hook, compaction). There is no separate English "You are a research assistant..." prompt template — the system message the model actually receives is the TITRE/AUTEUR/EXTRAIT block from Step 5 above, prepended to the conversation. The call goes through whichever provider is configured (ProviderRegistry), not a hardcoded Ollama call:

const response = await provider.generate({
  messages: [
    { role: 'system', content: formatContextAsSystemPrompt(hits) },
    ...conversationHistory,
  ],
  stream: true,
});

Streaming: Response is sent token by token for better UX

Stream Cancellation: Users can cancel ongoing generation via a per-session AbortController.

Context Compression

File: backend/core/rag/ContextCompressor.ts

Wired into the retrieval path since RC4. Before that it was reachable from nothing but its own test: no service, IPC handler or chat path imported it, and retrieval-service.ts never populated the field the rest of the pipeline checks, so no context was ever compressed.

RC4 also fixed three ways it distorted the excerpts shown in the Sources panel — which reads the compressed text, not the original. Markdown tables were shredded (| doubled as the internal sentence boundary) and recomposed into rows that never existed; sentences taken from far apart were joined with no ellipsis; and the "keep the best two" fallback returned them in score order, rewriting the text's chronology. Tables and code blocks now pass through whole, cuts are marked […], and document order is preserved.

The design below is real (it's exactly what ContextCompressor.ts implements) — this documents the intended algorithm, not something that currently runs:

Sizes below are characters, not tokens — ContextCompressor.ts's own comments say so explicitly ("Skip compression for small contexts (< 10k chars)").

┌─────────────────────────────────────────────────────────────┐
│              Context Compression Levels                      │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  No compression (context ≤ 10,000 chars)                      │
│    └── Passed through unchanged                               │
│                                                               │
│  Level 1: Light (context 15,000-25,000 chars)                 │
│    └── Semantic deduplication only (threshold 0.88)            │
│                                                               │
│  Level 2: Medium (context 25,000-35,000 chars)                 │
│    └── Deduplication (0.85) + relevant-sentence extraction     │
│                                                               │
│  Level 3: Aggressive (context > 35,000 chars)                  │
│    └── Aggressive deduplication (0.80) + full extraction       │
│                                                               │
└─────────────────────────────────────────────────────────────┘

Contexts between 10,000 and 15,000 characters would fall through all three levels untouched even if this ran — there's a real gap between "skip" and "Level 1", not a continuous scale.

Algorithm, as implemented in the unused class:

  1. Calculate total context size in characters
  2. Select compression level based on size thresholds
  3. Apply relevance scoring to sentences (keyword + query similarity)
  4. Preserve keyword-rich sentences
  5. Return compressed context with reduction statistics

enableContextCompression is a per-mode config field (see RAG Configuration Defaults below) that, per the above, has no consumer either — toggling it in a mode has no observable effect either way.

RAG Explanation (Explainable AI)

File: backend/types/chat-source.ts

ClioDeck provides transparency about how RAG responses are generated. The real shape is nested, not flat:

interface RAGExplanation {
  search: {
    query: string;
    totalResults: number;
    searchDurationMs: number;
    cacheHit: boolean;
    sourceType: 'primary' | 'secondary' | 'both';
    documents: Array<{
      title: string;
      similarity: number;
      sourceType: string;
      chunkCount: number;
    }>;
  };
  compression?: {
    enabled: boolean;
    originalChunks: number;
    finalChunks: number;
    originalSize: number;
    finalSize: number;
    reductionPercent: number;
    strategy?: string;
  };
  graph?: {
    enabled: boolean;
    relatedDocsFound: number;
    documentTitles: string[];
  };
  llm: {
    provider: string;
    model: string;
    contextWindow: number;
    temperature: number;
    promptSize: number;
  };
  timing: {
    searchMs: number;
    compressionMs?: number;
    generationMs: number;
    totalMs: number;
  };
}

This information is displayed in the chat interface, allowing researchers to understand and evaluate the response context.

External Integrations

Zotero

File: backend/integrations/zotero/ZoteroAPI.ts (not zotero-service.ts, which only orchestrates — the actual fetch() calls and headers live here)

// 1. Authentication
const headers = {
  'Zotero-API-Key': apiKey,
  'Zotero-API-Version': '3'
};

// 2. Retrieve collections
const collections = await fetch(
  `https://api.zotero.org/users/${userId}/collections`,
  { headers }
);

// 3. Retrieve items from a collection
const items = await fetch(
  `https://api.zotero.org/users/${userId}/collections/${collectionId}/items`,
  { headers }
);

// 4. BibTeX export
const bibtex = await fetch(
  `https://api.zotero.org/users/${userId}/collections/${collectionId}/items?format=bibtex`,
  { headers }
);

// 5. Download PDFs
for (const item of items) {
  if (item.data.linkMode === 'linked_file') {
    const pdf = await downloadAttachment(item.key);
    await savePDF(pdf, projectPath);
  }
}

BibTeX

Parsing: backend/core/bibliography/BibTeXParser.ts — a 484-line BibTeXParser class, not a standalone regex-based function. The illustrative snippet in earlier versions of this page (a ~20-line single-regex function) drastically understated it; there is no parseBibTeX function anywhere in the codebase. Custom-field round-trip (preserving fields BibTeX doesn't natively model) is handled here, matched by BibTeXExporter.ts on the way back out.

Topic Modeling (Python)

Service: backend/python-services/topic-modeling/main.py

The Python service does not compute embeddings itself — an earlier version of this page showed it calling SentenceTransformer(...).encode() internally, which contradicts the module's own docstring ("Reçoit des embeddings pré-calculés depuis Electron/Ollama") and its request schema, which requires embeddings as a mandatory input field (min_length=5) alongside documents and document_ids. Embeddings come from the TypeScript side, via whichever EmbeddingProvider is configured (see Step 3 above) — sentence-transformers is in requirements.txt only as a BERTopic transitive dependency, never imported directly in main.py.

from bertopic import BERTopic
from fastapi import FastAPI
import numpy as np

app = FastAPI()

@app.post("/analyze")
async def analyze_corpus(request: AnalyzeRequest):
    # Embeddings arrive pre-computed — no SentenceTransformer call here
    embeddings_array = np.array(request.embeddings, dtype=np.float32)

    topic_model = BERTopic(min_topic_size=request.min_topic_size)
    topics, _ = topic_model.fit_transform(request.documents, embeddings_array)

    return {
        "topics": topics,               # per real AnalyzeResponse shape
        "topic_assignments": {...},     # document_id -> topic id
        "outliers": [...],
        "statistics": {...},
    }

Communication: TypeScript backend communicates via HTTP (port 8001)

const response = await fetch('http://localhost:8001/analyze', {
  method: 'POST',
  body: JSON.stringify({
    embeddings,     // required — computed TypeScript-side first
    documents: chunks,
    document_ids: ids,
  })
});

// Real response shape: {topics, topic_assignments, outliers, statistics}
// — not {topics, topic_info} as an earlier version of this page showed.
const { topics, topic_assignments, outliers, statistics } = await response.json();

Performance and Optimizations

Memory Footprint (typical 50k chunks)

Component RAM Description
Electron 1.5 GB App + Chromium
HNSW index 500 MB 50k × 768 dims × 3x overhead
BM25 index 100 MB Inverted index + vocabulary
SQLite 300 MB Chunks + metadata
Embedding provider 0–500 MB Ollama's nomic-embed-text loads ~500 MB in RAM locally; the embedded model and cloud providers have a different footprint (embedded: bundled with the app; cloud: none, runs remotely)
Total ~3 GB For an average project

Recommended Configuration

Corpus Size Min RAM CPU Indexing Time
10-50 PDFs 4 GB Dual-core 5-10 min
50-200 PDFs 8 GB Quad-core 20-40 min
200-500 PDFs 16 GB 8+ cores 1-2 hours

CPU Optimizations

1. Parallel chunking:

// Process documents in parallel (max 4 simultaneous)
const batches = chunk(documents, 4);
for (const batch of batches) {
  await Promise.all(batch.map(doc => indexDocument(doc)));
}

2. Batch embeddings:

// Generate embeddings in batches of 32 — illustrative strategy, not a
// literal citation; the real batched call is generateEmbeddings(texts)
// on the active EmbeddingProvider (see Step 3 above), not an
// ollama.batchEmbed() method (no such method exists).
for (let i = 0; i < chunks.length; i += 32) {
  const batch = chunks.slice(i, i + 32);
  const embeddings = await embeddingProvider.generateEmbeddings(batch);
}

3. Background index building — illustrative strategy; HNSWVectorStore.ts has no rebuild() method, this section describes a deferred-work pattern in general terms, not a real call:

setTimeout(() => {
  // build/refresh the HNSW index here
}, 5000);

Chunking Configurations

const CHUNKING_CONFIGS = {
  // For modest machines (8 GB RAM)
  cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },

  // Balance performance/precision
  standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },

  // For maximum precision (16+ GB RAM)
  large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};

(Same maxTokens: 8192 fabrication as above — real config has no such field, see the note there.)

Key Architecture Files

backend/
├── core/
│   ├── analysis/
│   │   └── TopicModelingService.ts      # Thematic analysis + graph
│   ├── chunking/
│   │   ├── AdaptiveChunker.ts           # Structure-aware chunking
│   │   └── DocumentChunker.ts           # Simple chunking (fallback)
│   ├── history/
│   │   └── HistoryManager.ts            # Research journal
│   ├── llm/
│   │   └── providers/                   # Typed provider registry
│   │       ├── base.ts                  # LLMProvider / EmbeddingProvider contracts
│   │       ├── registry.ts              # Open factory map, 5 providers + embedded
│   │       ├── ollama.ts, embedded.ts, anthropic.ts,
│   │       │   openai-compatible.ts, mistral.ts, gemini.ts
│   │       ├── fallback.ts              # Ollama → embedded auto-fallback
│   │       └── cliodeck-config-adapter.ts
│   ├── pdf/
│   │   ├── PDFExtractor.ts              # PDF extraction (pdfjs) — the actual getDocument/getTextContent/getMetadata calls
│   │   └── PDFIndexer.ts                # Orchestrates extraction + indexing, delegates to PDFExtractor.ts
│   ├── search/
│   │   ├── BM25Index.ts                 # Sparse index (keywords)
│   │   └── HybridSearch.ts              # Dense + sparse fusion
│   └── vector-store/
│       ├── VectorStore.ts               # SQLite base store
│       ├── HNSWVectorStore.ts           # Fast HNSW index
│       └── EnhancedVectorStore.ts       # Unified wrapper
│
├── python-services/
│   └── topic-modeling/
│       ├── main.py                      # FastAPI service
│       ├── topic_analyzer.py            # BERTopic logic
│       └── requirements.txt             # Python dependencies
│
├── recipes/
│   ├── schema.ts, runner.ts             # ClioRecipes: YAML workflow steps
│   └── builtin/                         # Zotero review, Tropy analysis, etc.
│
├── types/
│   ├── config.ts                        # Configuration types
│   ├── book.ts                          # Book/chapters manifest + settings
│   └── pdf-document.ts                  # Document/chunk types
│
src/
├── editor/
│   ├── cm/                              # CodeMirror 6 state, fidelity, proposals
│   ├── facade.ts                        # Editor-agnostic abstraction (EditorFacade)
│   ├── proposals/                       # Adjudicable AI-proposal contract
│   └── slides.ts                        # Shared slide-splitting logic
│
├── main/
│   ├── services/
│   │   ├── fusion-chat-service.ts       # Unified chat transport (IPC fusion:chat:*)
│   │   ├── chat-engine.ts               # Shared agent loop, compaction, retrieval hook
│   │   ├── retrieval-service.ts         # Multi-source RAG (PDFs + Tropy + Obsidian + manuscript)
│   │   ├── manuscript-assembler.ts      # Book export assembly (per-chapter footnote IDs)
│   │   ├── manuscript-index-service.ts  # Manuscript as a 4th RAG corpus
│   │   ├── mcp-clients-service.ts       # External MCP server lifecycle
│   │   ├── usage-journal-service.ts     # AI usage journal sink (.cliodeck/journal.db)
│   │   ├── pdf-service.ts               # PDF management
│   │   ├── tropy-service.ts             # Primary sources (Tropy)
│   │   ├── project-manager.ts           # Project management
│   │   └── topic-modeling-service.ts    # Python service proxy
│   └── ipc/
│       └── handlers/                    # Electron IPC
│
└── renderer/
    ├── components/
    │   ├── Chat/                        # AssistantChat (single shell, variants full/panel)
    │   ├── Config/                      # Settings
    │   ├── Editor/                      # CodeMirror 6 editor
    │   ├── Brainstorm/                  # Chat + Ideas + Board + Graph tabs
    │   ├── PDFIndex/                    # PDF management
    │   └── Journal/                     # Research journal
    └── stores/
        └── editorStore.ts               # Zustand state

backend/core/llm/OllamaClient.ts and src/main/services/chat-service.ts, which older versions of this page listed here, no longer exist — replaced by the provider registry and fusion-chat-service.ts/chat-engine.ts above.

Primary Sources Integration (Tropy)

Architecture

PrimarySourcesVectorStore handles Tropy primary sources, but it stores its SQLite data in the shared .cliodeck/brain.db (the same file PDFs and research history use — see Key Architecture Files above), not a standalone database. Only the HNSW index is its own file:

┌─────────────────────────────────────────────────────────────┐
│                    Primary Sources Pipeline                   │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  Tropy Project (.tropy / .tpy)                                │
│         │                                                     │
│         ▼                                                     │
│  ┌──────────────┐                                            │
│  │ TropyReader  │ ──► Read-only access to SQLite database    │
│  └──────────────┘                                            │
│         │                                                     │
│         ▼                                                     │
│  ┌──────────────┐                                            │
│  │  TropySync   │ ──► Sync items, photos, notes, tags        │
│  └──────────────┘                                            │
│         │                                                     │
│         ├──► OCR Pipeline (Tesseract.js) for images          │
│         │                                                     │
│         ▼                                                     │
│  ┌────────────────────────┐                                  │
│  │ PrimarySourcesVectorStore │                               │
│  │   - SQLite: brain.db (shared)  │                          │
│  │   - HNSW: primary-hnsw.index   │                          │
│  │   - BM25: in-memory        │                               │
│  └────────────────────────┘                                  │
│                                                               │
└─────────────────────────────────────────────────────────────┘

Data Flow

  1. TropyReader opens the .tpy file in read-only mode
  2. TropySync extracts items with metadata, notes, and photos
  3. Transcription source priority, per item (TropySync.ts): Tropy notes first (if the item has any actual note content, not just metadata) → then an external transcription (Transkribus or similar, if a transcriptionDirectory is configured) → then OCR (Tesseract.js), if nothing else produced text and OCR is enabled. An earlier version of this page said "notes > OCR > manual" — wrong on two counts: the real second step is the external-transcription source, not OCR, and "manual" isn't part of this automatic chain at all — it's a separate, per-source re-run action, reachable from the source card since RC4, not a fallback this pipeline ever reaches on its own.
  4. DocumentChunker (fixed-size) segments transcriptions — hardcoded in tropy-service.ts, not "the same as PDFs": PDFs default to AdaptiveChunker (structure-aware) via useAdaptiveChunking: true in DEFAULT_CONFIG.rag, and Tropy has no equivalent option to opt into it. Primary sources always get fixed-size chunking regardless of that setting.
  5. The configured embedding provider generates embeddings (Ollama's nomic-embed-text, the embedded model, or a cloud provider)
  6. PrimarySourcesVectorStore stores chunks with HNSW + BM25 indexes

Unified Search

The snippet below is not a literal quote — an earlier version of this page presented it as one (// In retrieval-service.ts, no "illustrative" caveat, unlike other pseudo-code elsewhere on this page). The real code in retrieval-service.ts is a per-source-type partial-success pattern, not a naive merge-and-sort: secondary and primary searches each run in their own try/catch, each tracking its own ok/error/durationMs outcome, and results are pushed into a shared array as they succeed — one source type failing doesn't drop the other's results, and there's no similarity-sort merge step shown below actually happening at this layer:

// Illustrative only — NOT a literal quote from retrieval-service.ts
const pdfResults = await vectorStore.search(embedding, topK);
const tropyResults = await tropyService.search(query, { topK });

// Merge and deduplicate
const allResults = [...pdfResults, ...tropyResults]
  .sort((a, b) => b.similarity - a.similarity)
  .slice(0, topK);

tropyService.search() itself runs its own internal hybrid HNSW+BM25 search with RRF scoring and FR/EN multilingual query expansion — the same expansion mechanism SecondaryRetriever uses for PDFs, reused via expandQueryToText() rather than duplicated.

Embedded LLM Support

ClioDeck supports embedded LLMs via node-llama-cpp for fully offline generation and embeddings — see the Embedded LLM Guide for the canonical, up-to-date description (model sizes, prompt format, troubleshooting). That page is the one to update if this changes again; this section only sketches where it sits in the architecture.

The embedded provider is one entry in the same typed ProviderRegistry used by Ollama and the four cloud providers (backend/core/llm/providers/) — there is no separate "manager" class layered on top, and no requirement to have Ollama installed at all. embedded is a valid, independent value for both generationProvider and embeddingProvider in config (cliodeck-config-adapter.ts). Both have a UI control since RC4: Settings → LLM for generation, Settings → Embeddings for the embedding provider and model. Before RC4, reaching embedded for embeddings meant hand-editing the config file. See the Embedded LLM Guide.

Changing the embedding provider invalidates every existing index — vectors from one model cannot be compared with another's, and the comparison raises no error. Since RC4 the app says so before applying the change, and rebuilds the provider registry immediately instead of waiting for a restart. An auto mode also exists for each, which prefers Ollama when reachable and falls back to the embedded model (backend/core/llm/providers/fallback.ts). For embeddings specifically, the auto fallback only engages if the embedded and Ollama embedding models share the same vector dimension (cliodeck-config-adapter.ts's embeddingWithFallback()) — mixing dimensions would silently corrupt similarity search, so a mismatch is left to fail loudly instead.

RAG Configuration Defaults

Current default values live on the rag key of DEFAULT_CONFIG in backend/types/config.ts — there is no separately-named DEFAULT_RAG_CONFIG constant, just this nested object:

const DEFAULT_CONFIG = {
  // ...
  rag: {
    // Search parameters
    topK: 10,                          // Number of chunks to retrieve
    similarityThreshold: 0.12,         // RRF-optimized (not cosine similarity)

    // Chunking
    chunkingConfig: 'cpuOptimized',    // 300 words, 50 overlap
    useAdaptiveChunking: true,

    // Indexes
    useHNSWIndex: true,
    useHybridSearch: true,             // HNSW + BM25

    // Quality
    enableQualityFiltering: true,
    enablePreprocessing: true,
    enableDeduplication: true,

    // Advanced (optional)
    useSemanticChunking: false,
  },
};

enableContextCompression is not part of this object at all. An earlier version of this page listed it here as a single global default (true) — in the real code it's a per-mode override, set in backend/core/modes/built-in-modes.ts: true for primary-source-analyst, academic-writer, and methodology-assistant; false for literature-review, critical-reviewer, and brainstorming; unset for default-assistant and free-mode. Before RC4 none of this mattered: the class the flag toggles was never called, so the flag had no observable effect whatever a mode set it to. It is wired now — see Context Compression above.

Similarity Threshold Note

The default similarityThreshold of 0.12 is optimized for RRF (Reciprocal Rank Fusion) scores, which are much smaller than raw cosine similarity scores. Typical RRF scores range from 0.001 to 0.02.

Technical References

Papers

Tools and Libraries

Version: 1.0.0-rc.4

Clone this wiki locally