Skip to content

2. Technical Architecture

Jump to bottom
Frédéric Clavert edited this page Jul 24, 2026 · 19 revisions

This document describes the architecture and operation of ClioDeck's RAG (Retrieval-Augmented Generation) system, an academic writing assistant for historians and humanities researchers.

Table of Contents

Overview

ClioDeck is an Electron application that combines:

  • Frontend: React/TypeScript interface with a CodeMirror 6 editor (live markdown rendering)
  • Backend: Node.js/TypeScript services for indexing and search
  • Python Services: Topic modeling and textometric analysis
  • LLM: a typed provider registry — Ollama, embedded (node-llama-cpp), Anthropic, OpenAI-compatible, Mistral, Gemini — for embeddings and text generation
  • Storage: SQLite for data and vector indexes

Design Goals

  1. Local-first: All data remains on the user's machine
  2. Performance: Optimized for modest machines (8-16 GB RAM, CPU only)
  3. Academic: Complete traceability and verifiable citations
  4. Interoperability: Integration with Zotero, BibTeX, Markdown, PDF

Global Architecture

┌─────────────────────────────────────────────────────────────┐
│                    ClioDeck Electron App                      │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Frontend   │  │   Backend    │  │   Services   │      │
│  │  React/Vite  │◄─┤  TypeScript  │◄─┤    Python    │      │
│  │              │  │              │  │              │      │
│  │ • Editor     │  │ • PDF Index  │  │ • Topic      │      │
│  │ • Chat UI    │  │ • Vector DB  │  │   Modeling   │      │
│  │ • Config     │  │ • Search     │  │ • BERTopic   │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                            │                │                │
│                            ▼                ▼                │
│                    ┌──────────────┐  ┌──────────────┐      │
│                    │   Ollama     │  │   SQLite     │      │
│                    │   (Local)    │  │ + HNSW Index │      │
│                    └──────────────┘  └──────────────┘      │
│                                                               │
└─────────────────────────────────────────────────────────────┘

Key Technologies

Component Technology Role
UI Framework React 18 + TypeScript User interface
Editor CodeMirror 6 + Lezer Live markdown rendering, Pandoc dialect
Build Tool Vite Bundling and hot reload
Desktop Electron 40 Multi-platform desktop app
Database better-sqlite3 Synchronous local storage
Vector Index hnswlib-node Fast vector search
Sparse Index natural (BM25) Keyword search
LLM Typed provider registry Ollama, embedded, Anthropic, OpenAI-compatible, Mistral, Gemini
PDF Processing pdfjs-dist Text extraction
Bibliography BibTeX parsing Reference management
Topic Modeling Python + BERTopic Thematic analysis

Indexing Pipeline

The indexing pipeline transforms PDFs into queryable vector chunks.

Step 1: PDF Extraction

File: backend/core/pdf/PDFExtractor.ts (not PDFIndexer.ts, which only orchestrates and delegates here)

// 1. Load PDF with pdfjs
const pdf = await pdfjs.getDocument(pdfPath).promise;

// 2. Extract text page by page
const pages = [];
for (let i = 1; i <= pdf.numPages; i++) {
  const page = await pdf.getPage(i);
  const textContent = await page.getTextContent();
  const text = textContent.items.map(item => item.str).join(' ');
  pages.push({ pageNumber: pageNum, text });  // field is `text`, not `content` — PDFExtractor.ts:174-176
}

// 3. Extract metadata
const metadata = await pdf.getMetadata();
const title = metadata.info.Title || path.basename(pdfPath);

Output: Array of pages with raw text + document metadata

This code never runs directly inside the Electron main process. Real PDF indexing (src/main/services/pdf/PdfIndexer.ts, pdf-service.ts) calls extractPdfIsolated() (src/main/services/pdf-extract-isolated.ts), which spawns the system Node binary as a child process and runs PDFExtractor inside a separate worker script — a workaround for a pdfjs-dist crash on Electron's bundled Node. Three real, previously undocumented consequences for indexing a large PDF:

  • 120-second hard timeout per PDF. A pathological or very large scan that takes longer fails with "PDF extraction timed out after 120s" — there is no way to raise this from the UI.
  • Global sequential queue, one extraction at a time, app-wide. extractPdfIsolated() chains every call onto a single pending promise specifically to bound RAM — importing a large batch of PDFs (e.g. a big Zotero library) processes them strictly one by one, never in parallel, regardless of CPU cores available.
  • Requires a system Node 20+ binary on PATH (or at a few hardcoded Homebrew/nvm paths) separate from Electron's own bundled Node — if none is found, extraction fails with an explicit "requires a system Node.js binary" error rather than a silent hang.

Step 2: Adaptive Chunking

Files:

  • backend/core/chunking/AdaptiveChunker.ts (recommended)
  • backend/core/chunking/DocumentChunker.ts (fallback)
  • backend/core/chunking/SemanticChunker.ts (advanced semantic-aware chunking)
  • backend/core/chunking/ChunkDeduplicator.ts (removes near-duplicate chunks)
  • backend/core/chunking/ChunkQualityScorer.ts (scores chunk quality for filtering)
  • backend/core/chunking/EmbeddingCache.ts (caches embeddings for performance)

Structure Detection

The adaptive chunker automatically detects document structure:

// Header detection — patterns as they actually are in AdaptiveChunker.ts,
// stricter than earlier versions of this page implied
const patterns = {
  markdown: /^#{1,6}\s+(.+)/,                          // # Title, ## Subtitle
  numbered: /^(\d+(?:\.\d+)*)\.\s+([A-Z][^.]{2,50})$/, // "1. Introduction" — needs a trailing dot and a capitalized 2-50 char title
  uppercase: /^([A-Z][A-Z\s]{5,50})$/,                 // ALL CAPS, 6-51 chars, also checked against a known-header allowlist (ABSTRACT, INTRODUCTION, …)
  roman: /^([IVX]+)\.\s+([A-Z][^.]{2,50})$/            // "I. Introduction" — same capitalized-title constraint as `numbered`, not a bare "I. anything" match
};

// Automatic classification
const sectionTypes = {
  abstract: /abstract|résumé/i,
  introduction: /introduction/i,
  methodology: /method|méthodologie/i,
  results: /results|résultats/i,
  discussion: /discussion/i,
  conclusion: /conclusion/i,
  references: /references|bibliographie/i
};

Chunk Creation

Principles:

  1. Respect sentence boundaries: Chunks always end with ., !, ? or ;
  2. Document context: Each chunk includes title and section as prefix
  3. Skip references: Bibliography section is not indexed
  4. Smart overlap: Overlaps occur on complete sentences

Generated chunk example:

[Doc: Active Learning Strategies in Higher Education | Section: Methodology]

The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment. Students were divided
into control and experimental groups. The control group followed traditional
lecture-based instruction, while the experimental group participated in
collaborative problem-solving activities.

Configuration:

interface ChunkingConfig {
  maxChunkSize: number;     // Target max size in words (300-800)
  overlapSize: number;      // Overlap in words (50-100)
  minChunkSize: number;     // Minimum words, avoids tiny leftover chunks
}

const CHUNKING_CONFIGS = {
  cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },
  standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },
  large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};

There is no maxTokens field anywhere in this config — an earlier version of this page invented one; token limits for embedding models are handled separately, not as part of chunking config.

Output: Array of chunks with metadata

interface DocumentChunk {
  id: string;                 // UUID
  documentId: string;         // Document reference
  content: string;            // Chunk text (with context)
  pageNumber: number;         // Source page
  chunkIndex: number;         // Position in document
  startPosition: number;      // Character offset, start
  endPosition: number;        // Character offset, end
  // No `embedding` field here — embeddings live in the DB row
  // (pdf_chunks.embedding) or the vector store, not on this object.
  metadata?: {
    sectionTitle?: string;    // "Methodology", "Results"
    sectionType?: string;     // "methodology", "results"
    sectionLevel?: number;    // 1, 2, 3...
  };
}

Step 3: Embedding Generation

File: backend/core/llm/providers/ollama.ts (one of six embedding providers behind the typed registry — see Embedded LLM Support below for the others)

// 1. Connect to Ollama — the MODERN /api/embed endpoint, batched, not the
// legacy /api/embeddings single-prompt endpoint an earlier version of this
// page showed. ollama.ts's own comment explains why: "the legacy
// /api/embeddings endpoint ignores `truncate` and 500s — which is exactly
// the PDF-indexing crash this replaces."
const response = await fetch('http://localhost:11434/api/embed', {
  method: 'POST',
  body: JSON.stringify({
    model: 'nomic-embed-text',  // 768 dimensions
    input: [chunkContent, /* ...more chunks, batched */],
    truncate: true
  })
});

// 2. Retrieve vectors (one per input, in order)
const { embeddings } = await response.json();
// embeddings: number[][] (768 dimensions each)

Model used: nomic-embed-text

  • Dimensions: 768
  • Max context: 8192 tokens (model-dependent — nomic-embed-text itself advertises 2048 and ignores larger num_ctx values)
  • Size: ~274 MB
  • Performance: ~100 embeddings/second on average CPU

Oversized input, Ollama path: inputs that overflow the model's context are truncated server-side by Ollama itself (truncate: true in the /api/embed call above) — ollama.ts does no client-side chunking of its own; the comment in the file is explicit that server-side truncation happens "rather than" splitting the text.

A second, earlier safety net exists specifically for PDF indexing. src/main/services/pdf/PdfIndexer.ts slices any chunk text to EMBED_CHAR_CAP = 6000 characters before it ever reaches the embedding provider — independent of, and applied before, whichever of the two paths above eventually runs. Its own comment explains why: AdaptiveChunker can occasionally emit a single oversized chunk (a paragraph longer than maxChunkSize), and without this cap that one pathological chunk used to abort the entire indexing run with an HTTP 500 ("input length exceeds the context length") from Ollama — this cap is what prevents that, not truncate: true alone.

Oversized input, embedded-model path: the chunk-and-average chunkText() method shown below is real, but it belongs to EmbeddedLLMClient.ts (the node-llama-cpp embedded provider), not Ollama — an earlier version of this page misattributed it here. Its threshold is EMBEDDING_MAX_CHUNK_LENGTH = 2000 characters, not "8192 tokens":

// EmbeddedLLMClient.ts — embedded provider only, not Ollama
private chunkText(text: string, maxLength: number): string[] {
  // Find sentence ending in last 200 characters
  const sentenceEndings = /[.!?;](?=\s|$)/g;
  // ... cut at last found period
}
// Chunks are embedded individually, then averaged into a single vector.

Step 4: Multi-Strategy Indexing

File: backend/core/vector-store/EnhancedVectorStore.ts

Three indexes are created simultaneously:

Index 1: SQLite (database)

-- Real schema, VectorStore.ts:104-113 — table is pdf_chunks, not chunks;
-- no metadata/created_at columns; start_position/end_position instead
CREATE TABLE IF NOT EXISTS pdf_chunks (
  id TEXT PRIMARY KEY,
  document_id TEXT NOT NULL,
  content TEXT NOT NULL,
  page_number INTEGER NOT NULL,
  chunk_index INTEGER NOT NULL,
  start_position INTEGER NOT NULL,
  end_position INTEGER NOT NULL,
  embedding BLOB,
  FOREIGN KEY (document_id) REFERENCES pdf_documents(id) ON DELETE CASCADE
);

CREATE INDEX IF NOT EXISTS idx_chunks_document_id ON pdf_chunks(document_id);
CREATE INDEX IF NOT EXISTS idx_chunks_page_number ON pdf_chunks(page_number);

Role: Persistent storage + fallback for linear search

Index 2: HNSW (fast vector search)

File: backend/core/vector-store/HNSWVectorStore.ts

import { HierarchicalNSW } from 'hnswlib-node';

const index = new HierarchicalNSW('cosine', 768); // 768 dimensions
index.initIndex(maxElements, M=16, efConstruction=100);

// Indexing
for (const chunk of chunks) {
  index.addPoint(chunk.embedding, chunk.id);
}

// Save to disk
index.writeIndexSync(indexPath);

Parameters:

  • M=16: Connections per node (speed/precision trade-off)
  • efConstruction=100: Construction effort
  • efSearch=50: Search effort

Complexity:

  • Construction: O(n log n)
  • Search: O(log n)

Memory footprint: ~500 MB for 50k chunks (768 dims)

Index 3: BM25 (keyword search)

File: backend/core/search/BM25Index.ts

There is no hand-rolled inverted index — indexing delegates entirely to natural's TfIdf class, with BM25 scoring layered on top of it:

import natural from 'natural';
const { TfIdf } = natural;

// 1. Delegate document storage/tokenization to natural's TfIdf
const tfidf = new TfIdf();
tfidf.addDocument(preprocessText(chunk.content));

// 2. BM25 scoring, computed from TfIdf's term/document frequencies
const idf = Math.log((totalDocs - docFreq + 0.5) / (docFreq + 0.5));
const score = idf * ((tf * (k1 + 1)) / (tf + k1 * (1 - b + b * (docLength / avgDocLength))));

Parameters:

  • k1=1.5: Term frequency saturation
  • b=0.75: Document length normalization

Memory footprint: ~100 MB for 50k chunks

Search System

Hybrid Search (Dense + Sparse)

File: backend/core/search/HybridSearch.ts

Hybrid search combines the advantages of two complementary approaches:

Query: "bloom taxonomy methodology"
        │
        ├────────────────────┬────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
  ┌──────────┐        ┌──────────┐        ┌──────────┐
  │  Dense   │        │  Sparse  │        │  Fusion  │
  │  (HNSW)  │        │  (BM25)  │        │  (RRF)   │
  └──────────┘        └──────────┘        └──────────┘
        │                    │                    │
        ▼                    ▼                    ▼
   Top-50 results       Top-50 results       Top-10 final
   (semantic)           (keywords)           (best of both)

Step 1: Dense Search (HNSW)

// 1. Generate query embedding via the configured EmbeddingProvider
// (Ollama, embedded, or a cloud provider — see Embedded LLM Support below)
const queryEmbedding = await embeddingProvider.generateEmbedding(query);

// 2. HNSW search
const denseResults = await hnswStore.search(queryEmbedding, 50);
// Returns: [{ id, score }, ...]

Advantage: Understands semantic meaning ("active learning" ≈ "apprentissage actif")

Step 2: Sparse Search (BM25)

// 1. Query tokenization
const queryTokens = tokenizer.tokenize(query.toLowerCase());

// 2. BM25 scoring
const sparseResults = bm25Index.search(queryTokens, 50);
// Returns: [{ id, score }, ...]

Advantage: Excellent for rare terms, proper nouns, acronyms ("BERTopic", "Piaget")

Step 3: RRF Fusion (Reciprocal Rank Fusion)

function reciprocalRankFusion(
  denseResults: SearchResult[],
  sparseResults: SearchResult[],
  k: number = 60
): SearchResult[] {
  const scores = new Map<string, number>();

  // Dense search contribution (60%)
  for (let i = 0; i < denseResults.length; i++) {
    const chunkId = denseResults[i].id;
    const rrfScore = 0.6 / (k + i + 1);
    scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
  }

  // Sparse search contribution (40%)
  for (let i = 0; i < sparseResults.length; i++) {
    const chunkId = sparseResults[i].id;
    const rrfScore = 0.4 / (k + i + 1);
    scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
  }

  // Sort by final score
  return Array.from(scores.entries())
    .sort((a, b) => b[1] - a[1])
    .slice(0, 10)
    .map(([id, score]) => ({ id, score }));
}

Weights: 60% dense / 40% sparse (configurable)

RRF Formula: RRF(d) = Σ weight_i / (k + rank_i(d))

Exact match boost: HybridSearch.ts multiplies a chunk's RRF score by EXACT_MATCH_BOOST when the query string appears verbatim in it — RRF scores alone peak too low (top-1 ≈ 1/(K+1) ≈ 0.016) to otherwise outrank strong semantic neighbours scoring 0.45–0.55.

Performance

Metric Linear Search HNSW Only Hybrid (HNSW+BM25)
Time 500ms 15ms 30ms
Precision@10 65% 75% 80%
Recall@10 45% 55% 60%

Assisted Generation (RAG)

Complete Pipeline

User Query
    │
    ├─► 1. Generate embedding
    │       │
    │       ▼
    ├─► 2. Hybrid search (HNSW + BM25)
    │       │
    │       ▼
    ├─► 3. Retrieve top-K chunks (K=10)
    │       │
    │       ▼
    ├─► 4. Optional: Graph expansion
    │       │
    │       ▼
    ├─► 5. Build context
    │       │
    │       ▼
    └─► 6. Generate with LLM
            │
            ▼
        Response with citations

Steps 1-3: Search (see previous section)

Step 4: Graph Expansion — declared but dead

useGraphContext, graphSimilarityThreshold, and additionalGraphDocs are declared in backend/types/config.ts and set by mode presets, and even exposed in the RAG settings UI — but confirmed to have zero consumers anywhere in retrieval-service.ts or backend/core/rag/. Turning this setting on currently has no effect on retrieval; it isn't wired to anything. Documented here as a known gap, not as working behavior — the mechanism described in earlier versions of this page (graph-neighbor lookup, cosine similarity expansion) does not exist in the code today.

Step 5: Context Building

File: src/main/services/fusion-chat-service.ts, formatContextAsSystemPrompt(). An earlier version of this page showed a fabricated Markdown-based builder (# Context (sources), ## [Title, p.X] headers) that doesn't match the real code at all — and contradicted this wiki's own Brainstorm guide, which correctly describes the real format. The actual builder produces numbered TITRE/AUTEUR/EXTRAIT blocks, in French, with an explicit rule forcing the model to answer only from those fields:

function formatContextAsSystemPrompt(hits: MultiSourceSearchResult[]): string {
  const SNIPPET_CHARS = 1500;
  const lines: string[] = [
    'Contexte extrait du corpus indexé (sources numérotées ci-dessous).',
    "RÈGLE : réponds UNIQUEMENT à partir des sources ci-dessous (champs TITRE, AUTEUR et EXTRAIT). Ne complète JAMAIS avec tes connaissances générales.",
    // ...two more rule lines (TITRE-as-valid-answer for identification
    // questions, explicit no-canned-phrase wording for "not found") —
    // omitted here for brevity, present in the real function.
    "Cite le numéro de la source [N] pour chaque affirmation.",
    '',
  ];
  hits.forEach((h, i) => {
    const title = h.document.title || h.document.id || 'Sans titre';
    const kind = h.sourceType === 'primary' ? 'archive'
      : h.sourceType === 'vault' ? 'note' : 'bibliographie';
    lines.push(`[${i + 1}] type : ${kind}`);
    lines.push(`TITRE : ${title}`);
    // ...AUTEUR (if present), EXTRAIT (with page number for secondary sources)
  });
  return lines.join('\n');
}

Generated context example:

[1] type : bibliographie
TITRE : Active Learning in Higher Education
AUTEUR : Smith, 2019
EXTRAIT (p. 12) :
The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment...

Step 6: LLM Generation

Files: src/main/services/fusion-chat-service.ts (transport, IPC fusion:chat:*) and src/main/services/chat-engine.ts (agent loop, retrieval hook, compaction). There is no separate English "You are a research assistant..." prompt template — the system message the model actually receives is the TITRE/AUTEUR/EXTRAIT block from Step 5 above, prepended to the conversation. The call goes through whichever provider is configured (ProviderRegistry), not a hardcoded Ollama call:

const response = await provider.generate({
  messages: [
    { role: 'system', content: formatContextAsSystemPrompt(hits) },
    ...conversationHistory,
  ],
  stream: true,
});

Streaming: Response is sent token by token for better UX

Stream Cancellation: Users can cancel ongoing generation via a per-session AbortController.

Context Compression — declared but dead

File: backend/core/rag/ContextCompressor.ts

This class is never called by the running app. grep -rln "ContextCompressor" --include="*.ts" . returns only the class's own file and its own test (ContextCompressor.test.ts) — no service, IPC handler, or chat path imports it. Filed as issue #28. Confirmed from the consumer side too: chat-engine.ts reads result.explanation.compression and only fires its onCompressionStats hook when that field is truthy, but retrieval-service.ts (the only place RAGExplanation objects are actually built) never assigns a compression key to them — so that field is always undefined in practice, and the "compressing" phase in chat-engine.ts never becomes observable. Whatever context a query retrieves is sent to the LLM as-is, uncompressed, regardless of size.

The design below is real (it's exactly what ContextCompressor.ts implements) — this documents the intended algorithm, not something that currently runs:

Sizes below are characters, not tokens — ContextCompressor.ts's own comments say so explicitly ("Skip compression for small contexts (< 10k chars)").

┌─────────────────────────────────────────────────────────────┐
│           Context Compression Levels (not wired up)           │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  No compression (context ≤ 10,000 chars)                      │
│    └── Passed through unchanged                               │
│                                                               │
│  Level 1: Light (context 15,000-25,000 chars)                 │
│    └── Semantic deduplication only (threshold 0.88)            │
│                                                               │
│  Level 2: Medium (context 25,000-35,000 chars)                 │
│    └── Deduplication (0.85) + relevant-sentence extraction     │
│                                                               │
│  Level 3: Aggressive (context > 35,000 chars)                  │
│    └── Aggressive deduplication (0.80) + full extraction       │
│                                                               │
└─────────────────────────────────────────────────────────────┘

Contexts between 10,000 and 15,000 characters would fall through all three levels untouched even if this ran — there's a real gap between "skip" and "Level 1", not a continuous scale.

Algorithm, as implemented in the unused class:

  1. Calculate total context size in characters
  2. Select compression level based on size thresholds
  3. Apply relevance scoring to sentences (keyword + query similarity)
  4. Preserve keyword-rich sentences
  5. Return compressed context with reduction statistics

enableContextCompression is a per-mode config field (see RAG Configuration Defaults below) that, per the above, has no consumer either — toggling it in a mode has no observable effect either way.

RAG Explanation (Explainable AI)

File: backend/types/chat-source.ts

ClioDeck provides transparency about how RAG responses are generated. The real shape is nested, not flat:

interface RAGExplanation {
  search: {
    query: string;
    totalResults: number;
    searchDurationMs: number;
    cacheHit: boolean;
    sourceType: 'primary' | 'secondary' | 'both';
    documents: Array<{
      title: string;
      similarity: number;
      sourceType: string;
      chunkCount: number;
    }>;
  };
  compression?: {
    enabled: boolean;
    originalChunks: number;
    finalChunks: number;
    originalSize: number;
    finalSize: number;
    reductionPercent: number;
    strategy?: string;
  };
  graph?: {
    enabled: boolean;
    relatedDocsFound: number;
    documentTitles: string[];
  };
  llm: {
    provider: string;
    model: string;
    contextWindow: number;
    temperature: number;
    promptSize: number;
  };
  timing: {
    searchMs: number;
    compressionMs?: number;
    generationMs: number;
    totalMs: number;
  };
}

This information is displayed in the chat interface, allowing researchers to understand and evaluate the response context.

External Integrations

Zotero

File: backend/integrations/zotero/ZoteroAPI.ts (not zotero-service.ts, which only orchestrates — the actual fetch() calls and headers live here)

// 1. Authentication
const headers = {
  'Zotero-API-Key': apiKey,
  'Zotero-API-Version': '3'
};

// 2. Retrieve collections
const collections = await fetch(
  `https://api.zotero.org/users/${userId}/collections`,
  { headers }
);

// 3. Retrieve items from a collection
const items = await fetch(
  `https://api.zotero.org/users/${userId}/collections/${collectionId}/items`,
  { headers }
);

// 4. BibTeX export
const bibtex = await fetch(
  `https://api.zotero.org/users/${userId}/collections/${collectionId}/items?format=bibtex`,
  { headers }
);

// 5. Download PDFs
for (const item of items) {
  if (item.data.linkMode === 'linked_file') {
    const pdf = await downloadAttachment(item.key);
    await savePDF(pdf, projectPath);
  }
}

BibTeX

Parsing: backend/core/bibliography/BibTeXParser.ts — a 484-line BibTeXParser class, not a standalone regex-based function. The illustrative snippet in earlier versions of this page (a ~20-line single-regex function) drastically understated it; there is no parseBibTeX function anywhere in the codebase. Custom-field round-trip (preserving fields BibTeX doesn't natively model) is handled here, matched by BibTeXExporter.ts on the way back out.

Topic Modeling (Python)

Service: backend/python-services/topic-modeling/main.py

The Python service does not compute embeddings itself — an earlier version of this page showed it calling SentenceTransformer(...).encode() internally, which contradicts the module's own docstring ("Reçoit des embeddings pré-calculés depuis Electron/Ollama") and its request schema, which requires embeddings as a mandatory input field (min_length=5) alongside documents and document_ids. Embeddings come from the TypeScript side, via whichever EmbeddingProvider is configured (see Step 3 above) — sentence-transformers is in requirements.txt only as a BERTopic transitive dependency, never imported directly in main.py.

from bertopic import BERTopic
from fastapi import FastAPI
import numpy as np

app = FastAPI()

@app.post("/analyze")
async def analyze_corpus(request: AnalyzeRequest):
    # Embeddings arrive pre-computed — no SentenceTransformer call here
    embeddings_array = np.array(request.embeddings, dtype=np.float32)

    topic_model = BERTopic(min_topic_size=request.min_topic_size)
    topics, _ = topic_model.fit_transform(request.documents, embeddings_array)

    return {
        "topics": topics,               # per real AnalyzeResponse shape
        "topic_assignments": {...},     # document_id -> topic id
        "outliers": [...],
        "statistics": {...},
    }

Communication: TypeScript backend communicates via HTTP (port 8001)

const response = await fetch('http://localhost:8001/analyze', {
  method: 'POST',
  body: JSON.stringify({
    embeddings,     // required — computed TypeScript-side first
    documents: chunks,
    document_ids: ids,
  })
});

// Real response shape: {topics, topic_assignments, outliers, statistics}
// — not {topics, topic_info} as an earlier version of this page showed.
const { topics, topic_assignments, outliers, statistics } = await response.json();

Performance and Optimizations

Memory Footprint (typical 50k chunks)

Component RAM Description
Electron 1.5 GB App + Chromium
HNSW index 500 MB 50k × 768 dims × 3x overhead
BM25 index 100 MB Inverted index + vocabulary
SQLite 300 MB Chunks + metadata
Embedding provider 0–500 MB Ollama's nomic-embed-text loads ~500 MB in RAM locally; the embedded model and cloud providers have a different footprint (embedded: bundled with the app; cloud: none, runs remotely)
Total ~3 GB For an average project

Recommended Configuration

Corpus Size Min RAM CPU Indexing Time
10-50 PDFs 4 GB Dual-core 5-10 min
50-200 PDFs 8 GB Quad-core 20-40 min
200-500 PDFs 16 GB 8+ cores 1-2 hours

CPU Optimizations

1. Parallel chunking:

// Process documents in parallel (max 4 simultaneous)
const batches = chunk(documents, 4);
for (const batch of batches) {
  await Promise.all(batch.map(doc => indexDocument(doc)));
}

2. Batch embeddings:

// Generate embeddings in batches of 32 — illustrative strategy, not a
// literal citation; the real batched call is generateEmbeddings(texts)
// on the active EmbeddingProvider (see Step 3 above), not an
// ollama.batchEmbed() method (no such method exists).
for (let i = 0; i < chunks.length; i += 32) {
  const batch = chunks.slice(i, i + 32);
  const embeddings = await embeddingProvider.generateEmbeddings(batch);
}

3. Background index building — illustrative strategy; HNSWVectorStore.ts has no rebuild() method, this section describes a deferred-work pattern in general terms, not a real call:

setTimeout(() => {
  // build/refresh the HNSW index here
}, 5000);

Chunking Configurations

const CHUNKING_CONFIGS = {
  // For modest machines (8 GB RAM)
  cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },

  // Balance performance/precision
  standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },

  // For maximum precision (16+ GB RAM)
  large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};

(Same maxTokens: 8192 fabrication as above — real config has no such field, see the note there.)

Key Architecture Files

backend/
├── core/
│   ├── analysis/
│   │   └── TopicModelingService.ts      # Thematic analysis + graph
│   ├── chunking/
│   │   ├── AdaptiveChunker.ts           # Structure-aware chunking
│   │   └── DocumentChunker.ts           # Simple chunking (fallback)
│   ├── history/
│   │   └── HistoryManager.ts            # Research journal
│   ├── llm/
│   │   └── providers/                   # Typed provider registry
│   │       ├── base.ts                  # LLMProvider / EmbeddingProvider contracts
│   │       ├── registry.ts              # Open factory map, 5 providers + embedded
│   │       ├── ollama.ts, embedded.ts, anthropic.ts,
│   │       │   openai-compatible.ts, mistral.ts, gemini.ts
│   │       ├── fallback.ts              # Ollama → embedded auto-fallback
│   │       └── cliodeck-config-adapter.ts
│   ├── pdf/
│   │   ├── PDFExtractor.ts              # PDF extraction (pdfjs) — the actual getDocument/getTextContent/getMetadata calls
│   │   └── PDFIndexer.ts                # Orchestrates extraction + indexing, delegates to PDFExtractor.ts
│   ├── search/
│   │   ├── BM25Index.ts                 # Sparse index (keywords)
│   │   └── HybridSearch.ts              # Dense + sparse fusion
│   └── vector-store/
│       ├── VectorStore.ts               # SQLite base store
│       ├── HNSWVectorStore.ts           # Fast HNSW index
│       └── EnhancedVectorStore.ts       # Unified wrapper
│
├── python-services/
│   └── topic-modeling/
│       ├── main.py                      # FastAPI service
│       ├── topic_analyzer.py            # BERTopic logic
│       └── requirements.txt             # Python dependencies
│
├── recipes/
│   ├── schema.ts, runner.ts             # ClioRecipes: YAML workflow steps
│   └── builtin/                         # Zotero review, Tropy analysis, etc.
│
├── types/
│   ├── config.ts                        # Configuration types
│   ├── book.ts                          # Book/chapters manifest + settings
│   └── pdf-document.ts                  # Document/chunk types
│
src/
├── editor/
│   ├── cm/                              # CodeMirror 6 state, fidelity, proposals
│   ├── facade.ts                        # Editor-agnostic abstraction (EditorFacade)
│   ├── proposals/                       # Adjudicable AI-proposal contract
│   └── slides.ts                        # Shared slide-splitting logic
│
├── main/
│   ├── services/
│   │   ├── fusion-chat-service.ts       # Unified chat transport (IPC fusion:chat:*)
│   │   ├── chat-engine.ts               # Shared agent loop, compaction, retrieval hook
│   │   ├── retrieval-service.ts         # Multi-source RAG (PDFs + Tropy + Obsidian + manuscript)
│   │   ├── manuscript-assembler.ts      # Book export assembly (per-chapter footnote IDs)
│   │   ├── manuscript-index-service.ts  # Manuscript as a 4th RAG corpus
│   │   ├── mcp-clients-service.ts       # External MCP server lifecycle
│   │   ├── usage-journal-service.ts     # AI usage journal sink (.cliodeck/journal.db)
│   │   ├── pdf-service.ts               # PDF management
│   │   ├── tropy-service.ts             # Primary sources (Tropy)
│   │   ├── project-manager.ts           # Project management
│   │   └── topic-modeling-service.ts    # Python service proxy
│   └── ipc/
│       └── handlers/                    # Electron IPC
│
└── renderer/
    ├── components/
    │   ├── Chat/                        # AssistantChat (single shell, variants full/panel)
    │   ├── Config/                      # Settings
    │   ├── Editor/                      # CodeMirror 6 editor
    │   ├── Brainstorm/                  # Chat + Ideas + Board + Graph tabs
    │   ├── PDFIndex/                    # PDF management
    │   └── Journal/                     # Research journal
    └── stores/
        └── editorStore.ts               # Zustand state

backend/core/llm/OllamaClient.ts and src/main/services/chat-service.ts, which older versions of this page listed here, no longer exist — replaced by the provider registry and fusion-chat-service.ts/chat-engine.ts above.

Primary Sources Integration (Tropy)

Architecture

PrimarySourcesVectorStore handles Tropy primary sources, but it stores its SQLite data in the shared .cliodeck/brain.db (the same file PDFs and research history use — see Key Architecture Files above), not a standalone database. Only the HNSW index is its own file:

┌─────────────────────────────────────────────────────────────┐
│                    Primary Sources Pipeline                   │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  Tropy Project (.tropy / .tpy)                                │
│         │                                                     │
│         ▼                                                     │
│  ┌──────────────┐                                            │
│  │ TropyReader  │ ──► Read-only access to SQLite database    │
│  └──────────────┘                                            │
│         │                                                     │
│         ▼                                                     │
│  ┌──────────────┐                                            │
│  │  TropySync   │ ──► Sync items, photos, notes, tags        │
│  └──────────────┘                                            │
│         │                                                     │
│         ├──► OCR Pipeline (Tesseract.js) for images          │
│         │                                                     │
│         ▼                                                     │
│  ┌────────────────────────┐                                  │
│  │ PrimarySourcesVectorStore │                               │
│  │   - SQLite: brain.db (shared)  │                          │
│  │   - HNSW: primary-hnsw.index   │                          │
│  │   - BM25: in-memory        │                               │
│  └────────────────────────┘                                  │
│                                                               │
└─────────────────────────────────────────────────────────────┘

Data Flow

  1. TropyReader opens the .tpy file in read-only mode
  2. TropySync extracts items with metadata, notes, and photos
  3. Notes are collected as transcriptions (Tropy notes > OCR > manual)
  4. DocumentChunker segments transcriptions (same as PDFs)
  5. The configured embedding provider generates embeddings (Ollama's nomic-embed-text, the embedded model, or a cloud provider)
  6. PrimarySourcesVectorStore stores chunks with HNSW + BM25 indexes

Unified Search

When querying, both stores are searched and results are merged:

// In retrieval-service.ts
const pdfResults = await vectorStore.search(embedding, topK);
const tropyResults = await tropyService.search(query, { topK });

// Merge and deduplicate
const allResults = [...pdfResults, ...tropyResults]
  .sort((a, b) => b.similarity - a.similarity)
  .slice(0, topK);

Embedded LLM Support

ClioDeck supports embedded LLMs via node-llama-cpp for fully offline generation and embeddings — see the Embedded LLM Guide for the canonical, up-to-date description (model sizes, prompt format, troubleshooting). That page is the one to update if this changes again; this section only sketches where it sits in the architecture.

The embedded provider is one entry in the same typed ProviderRegistry used by Ollama and the four cloud providers (backend/core/llm/providers/) — there is no separate "manager" class layered on top, and no requirement to have Ollama installed at all. embedded is a valid, independent value for both generationProvider and embeddingProvider in config (cliodeck-config-adapter.ts) — but only generationProvider has a UI control, in Settings → LLM. embeddingProvider has no UI anywhere; reaching embedded for embeddings today means hand-editing the config file. See the Embedded LLM Guide and issue #18 — an earlier version of this page wrongly implied both were UI-selectable. An auto mode also exists for each, which prefers Ollama when reachable and falls back to the embedded model (backend/core/llm/providers/fallback.ts). For embeddings specifically, the auto fallback only engages if the embedded and Ollama embedding models share the same vector dimension (cliodeck-config-adapter.ts's embeddingWithFallback()) — mixing dimensions would silently corrupt similarity search, so a mismatch is left to fail loudly instead.

RAG Configuration Defaults

Current default values live on the rag key of DEFAULT_CONFIG in backend/types/config.ts — there is no separately-named DEFAULT_RAG_CONFIG constant, just this nested object:

const DEFAULT_CONFIG = {
  // ...
  rag: {
    // Search parameters
    topK: 10,                          // Number of chunks to retrieve
    similarityThreshold: 0.12,         // RRF-optimized (not cosine similarity)

    // Chunking
    chunkingConfig: 'cpuOptimized',    // 300 words, 50 overlap
    useAdaptiveChunking: true,

    // Indexes
    useHNSWIndex: true,
    useHybridSearch: true,             // HNSW + BM25

    // Quality
    enableQualityFiltering: true,
    enablePreprocessing: true,
    enableDeduplication: true,

    // Advanced (optional)
    useSemanticChunking: false,
  },
};

enableContextCompression is not part of this object at all. An earlier version of this page listed it here as a single global default (true) — in the real code it's a per-mode override, set in backend/core/modes/built-in-modes.ts: true for primary-source-analyst, academic-writer, and methodology-assistant; false for literature-review, critical-reviewer, and brainstorming; unset for default-assistant and free-mode. None of this matters in practice: as Context Compression above explains, the class this flag would toggle is never called by anything — there is no call site to fall back to, and the flag has zero observable effect regardless of which mode sets it or to what.

Similarity Threshold Note

The default similarityThreshold of 0.12 is optimized for RRF (Reciprocal Rank Fusion) scores, which are much smaller than raw cosine similarity scores. Typical RRF scores range from 0.001 to 0.02.

Technical References

Papers

Tools and Libraries

Version: 1.0.0-rc.3

Clone this wiki locally