-
Notifications
You must be signed in to change notification settings - Fork 0
2. Technical Architecture
This document describes the architecture and operation of ClioDeck's RAG (Retrieval-Augmented Generation) system, an academic writing assistant for historians and humanities researchers.
- Overview
- Global Architecture
- Indexing Pipeline
- Search System
- Assisted Generation (RAG)
- External Integrations
- Performance and Optimizations
ClioDeck is an Electron application that combines:
- Frontend: React/TypeScript interface with a CodeMirror 6 editor (live markdown rendering)
- Backend: Node.js/TypeScript services for indexing and search
- Python Services: Topic modeling and textometric analysis
-
LLM: a typed provider registry — Ollama, embedded (
node-llama-cpp), Anthropic, OpenAI-compatible, Mistral, Gemini — for embeddings and text generation - Storage: SQLite for data and vector indexes
- Local-first: All data remains on the user's machine
- Performance: Optimized for modest machines (8-16 GB RAM, CPU only)
- Academic: Complete traceability and verifiable citations
- Interoperability: Integration with Zotero, BibTeX, Markdown, PDF
┌─────────────────────────────────────────────────────────────┐
│ ClioDeck Electron App │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Frontend │ │ Backend │ │ Services │ │
│ │ React/Vite │◄─┤ TypeScript │◄─┤ Python │ │
│ │ │ │ │ │ │ │
│ │ • Editor │ │ • PDF Index │ │ • Topic │ │
│ │ • Chat UI │ │ • Vector DB │ │ Modeling │ │
│ │ • Config │ │ • Search │ │ • BERTopic │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Ollama │ │ SQLite │ │
│ │ (Local) │ │ + HNSW Index │ │
│ └──────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
| Component | Technology | Role |
|---|---|---|
| UI Framework | React 18 + TypeScript | User interface |
| Editor | CodeMirror 6 + Lezer | Live markdown rendering, Pandoc dialect |
| Build Tool | Vite | Bundling and hot reload |
| Desktop | Electron 40 | Multi-platform desktop app |
| Database | better-sqlite3 | Synchronous local storage |
| Vector Index | hnswlib-node | Fast vector search |
| Sparse Index | natural (BM25) | Keyword search |
| LLM | Typed provider registry | Ollama, embedded, Anthropic, OpenAI-compatible, Mistral, Gemini |
| PDF Processing | pdfjs-dist | Text extraction |
| Bibliography | BibTeX parsing | Reference management |
| Topic Modeling | Python + BERTopic | Thematic analysis |
The indexing pipeline transforms PDFs into queryable vector chunks.
File: backend/core/pdf/PDFExtractor.ts (not PDFIndexer.ts, which only orchestrates and delegates here)
// 1. Load PDF with pdfjs
const pdf = await pdfjs.getDocument(pdfPath).promise;
// 2. Extract text page by page
const pages = [];
for (let i = 1; i <= pdf.numPages; i++) {
const page = await pdf.getPage(i);
const textContent = await page.getTextContent();
const text = textContent.items.map(item => item.str).join(' ');
pages.push({ pageNumber: pageNum, text }); // field is `text`, not `content` — PDFExtractor.ts:174-176
}
// 3. Extract metadata
const metadata = await pdf.getMetadata();
const title = metadata.info.Title || path.basename(pdfPath);Output: Array of pages with raw text + document metadata
Files:
-
backend/core/chunking/AdaptiveChunker.ts(recommended) -
backend/core/chunking/DocumentChunker.ts(fallback) -
backend/core/chunking/SemanticChunker.ts(advanced semantic-aware chunking) -
backend/core/chunking/ChunkDeduplicator.ts(removes near-duplicate chunks) -
backend/core/chunking/ChunkQualityScorer.ts(scores chunk quality for filtering) -
backend/core/chunking/EmbeddingCache.ts(caches embeddings for performance)
The adaptive chunker automatically detects document structure:
// Header detection — patterns as they actually are in AdaptiveChunker.ts,
// stricter than earlier versions of this page implied
const patterns = {
markdown: /^#{1,6}\s+(.+)/, // # Title, ## Subtitle
numbered: /^(\d+(?:\.\d+)*)\.\s+([A-Z][^.]{2,50})$/, // "1. Introduction" — needs a trailing dot and a capitalized 2-50 char title
uppercase: /^([A-Z][A-Z\s]{5,50})$/, // ALL CAPS, 6-51 chars, also checked against a known-header allowlist (ABSTRACT, INTRODUCTION, …)
roman: /^([IVX]+)\.\s+([A-Z][^.]{2,50})$/ // "I. Introduction" — same capitalized-title constraint as `numbered`, not a bare "I. anything" match
};
// Automatic classification
const sectionTypes = {
abstract: /abstract|résumé/i,
introduction: /introduction/i,
methodology: /method|méthodologie/i,
results: /results|résultats/i,
discussion: /discussion/i,
conclusion: /conclusion/i,
references: /references|bibliographie/i
};Principles:
-
Respect sentence boundaries: Chunks always end with
.,!,?or; - Document context: Each chunk includes title and section as prefix
- Skip references: Bibliography section is not indexed
- Smart overlap: Overlaps occur on complete sentences
Generated chunk example:
[Doc: Active Learning Strategies in Higher Education | Section: Methodology]
The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment. Students were divided
into control and experimental groups. The control group followed traditional
lecture-based instruction, while the experimental group participated in
collaborative problem-solving activities.
Configuration:
interface ChunkingConfig {
maxChunkSize: number; // Target max size in words (300-800)
overlapSize: number; // Overlap in words (50-100)
minChunkSize: number; // Minimum words, avoids tiny leftover chunks
}
const CHUNKING_CONFIGS = {
cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },
standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },
large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};There is no maxTokens field anywhere in this config — an earlier version
of this page invented one; token limits for embedding models are handled
separately, not as part of chunking config.
Output: Array of chunks with metadata
interface DocumentChunk {
id: string; // UUID
documentId: string; // Document reference
content: string; // Chunk text (with context)
pageNumber: number; // Source page
chunkIndex: number; // Position in document
startPosition: number; // Character offset, start
endPosition: number; // Character offset, end
// No `embedding` field here — embeddings live in the DB row
// (pdf_chunks.embedding) or the vector store, not on this object.
metadata?: {
sectionTitle?: string; // "Methodology", "Results"
sectionType?: string; // "methodology", "results"
sectionLevel?: number; // 1, 2, 3...
};
}File: backend/core/llm/providers/ollama.ts (one of six embedding providers behind the typed registry — see Embedded LLM Support below for the others)
// 1. Connect to Ollama — the MODERN /api/embed endpoint, batched, not the
// legacy /api/embeddings single-prompt endpoint an earlier version of this
// page showed. ollama.ts's own comment explains why: "the legacy
// /api/embeddings endpoint ignores `truncate` and 500s — which is exactly
// the PDF-indexing crash this replaces."
const response = await fetch('http://localhost:11434/api/embed', {
method: 'POST',
body: JSON.stringify({
model: 'nomic-embed-text', // 768 dimensions
input: [chunkContent, /* ...more chunks, batched */],
truncate: true
})
});
// 2. Retrieve vectors (one per input, in order)
const { embeddings } = await response.json();
// embeddings: number[][] (768 dimensions each)Model used: nomic-embed-text
- Dimensions: 768
- Max context: 8192 tokens (model-dependent —
nomic-embed-textitself advertises 2048 and ignores largernum_ctxvalues) - Size: ~274 MB
- Performance: ~100 embeddings/second on average CPU
Oversized input, Ollama path: inputs that overflow the model's context are
truncated server-side by Ollama itself (truncate: true in the /api/embed
call above) — ollama.ts does no client-side chunking of its own; the comment
in the file is explicit that server-side truncation happens "rather than"
splitting the text.
Oversized input, embedded-model path: the chunk-and-average chunkText()
method shown below is real, but it belongs to EmbeddedLLMClient.ts (the
node-llama-cpp embedded provider), not Ollama — an earlier version of this
page misattributed it here. Its threshold is EMBEDDING_MAX_CHUNK_LENGTH = 2000
characters, not "8192 tokens":
// EmbeddedLLMClient.ts — embedded provider only, not Ollama
private chunkText(text: string, maxLength: number): string[] {
// Find sentence ending in last 200 characters
const sentenceEndings = /[.!?;](?=\s|$)/g;
// ... cut at last found period
}
// Chunks are embedded individually, then averaged into a single vector.File: backend/core/vector-store/EnhancedVectorStore.ts
Three indexes are created simultaneously:
-- Real schema, VectorStore.ts:104-113 — table is pdf_chunks, not chunks;
-- no metadata/created_at columns; start_position/end_position instead
CREATE TABLE IF NOT EXISTS pdf_chunks (
id TEXT PRIMARY KEY,
document_id TEXT NOT NULL,
content TEXT NOT NULL,
page_number INTEGER NOT NULL,
chunk_index INTEGER NOT NULL,
start_position INTEGER NOT NULL,
end_position INTEGER NOT NULL,
embedding BLOB,
FOREIGN KEY (document_id) REFERENCES pdf_documents(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_chunks_document_id ON pdf_chunks(document_id);
CREATE INDEX IF NOT EXISTS idx_chunks_page_number ON pdf_chunks(page_number);Role: Persistent storage + fallback for linear search
File: backend/core/vector-store/HNSWVectorStore.ts
import { HierarchicalNSW } from 'hnswlib-node';
const index = new HierarchicalNSW('cosine', 768); // 768 dimensions
index.initIndex(maxElements, M=16, efConstruction=100);
// Indexing
for (const chunk of chunks) {
index.addPoint(chunk.embedding, chunk.id);
}
// Save to disk
index.writeIndexSync(indexPath);Parameters:
-
M=16: Connections per node (speed/precision trade-off) -
efConstruction=100: Construction effort -
efSearch=50: Search effort
Complexity:
- Construction: O(n log n)
- Search: O(log n)
Memory footprint: ~500 MB for 50k chunks (768 dims)
File: backend/core/search/BM25Index.ts
There is no hand-rolled inverted index — indexing delegates entirely to
natural's TfIdf class, with BM25 scoring layered on top of it:
import natural from 'natural';
const { TfIdf } = natural;
// 1. Delegate document storage/tokenization to natural's TfIdf
const tfidf = new TfIdf();
tfidf.addDocument(preprocessText(chunk.content));
// 2. BM25 scoring, computed from TfIdf's term/document frequencies
const idf = Math.log((totalDocs - docFreq + 0.5) / (docFreq + 0.5));
const score = idf * ((tf * (k1 + 1)) / (tf + k1 * (1 - b + b * (docLength / avgDocLength))));Parameters:
-
k1=1.5: Term frequency saturation -
b=0.75: Document length normalization
Memory footprint: ~100 MB for 50k chunks
File: backend/core/search/HybridSearch.ts
Hybrid search combines the advantages of two complementary approaches:
Query: "bloom taxonomy methodology"
│
├────────────────────┬────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Dense │ │ Sparse │ │ Fusion │
│ (HNSW) │ │ (BM25) │ │ (RRF) │
└──────────┘ └──────────┘ └──────────┘
│ │ │
▼ ▼ ▼
Top-50 results Top-50 results Top-10 final
(semantic) (keywords) (best of both)
// 1. Generate query embedding via the configured EmbeddingProvider
// (Ollama, embedded, or a cloud provider — see Embedded LLM Support below)
const queryEmbedding = await embeddingProvider.generateEmbedding(query);
// 2. HNSW search
const denseResults = await hnswStore.search(queryEmbedding, 50);
// Returns: [{ id, score }, ...]Advantage: Understands semantic meaning ("active learning" ≈ "apprentissage actif")
// 1. Query tokenization
const queryTokens = tokenizer.tokenize(query.toLowerCase());
// 2. BM25 scoring
const sparseResults = bm25Index.search(queryTokens, 50);
// Returns: [{ id, score }, ...]Advantage: Excellent for rare terms, proper nouns, acronyms ("BERTopic", "Piaget")
function reciprocalRankFusion(
denseResults: SearchResult[],
sparseResults: SearchResult[],
k: number = 60
): SearchResult[] {
const scores = new Map<string, number>();
// Dense search contribution (60%)
for (let i = 0; i < denseResults.length; i++) {
const chunkId = denseResults[i].id;
const rrfScore = 0.6 / (k + i + 1);
scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
}
// Sparse search contribution (40%)
for (let i = 0; i < sparseResults.length; i++) {
const chunkId = sparseResults[i].id;
const rrfScore = 0.4 / (k + i + 1);
scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
}
// Sort by final score
return Array.from(scores.entries())
.sort((a, b) => b[1] - a[1])
.slice(0, 10)
.map(([id, score]) => ({ id, score }));
}Weights: 60% dense / 40% sparse (configurable)
RRF Formula: RRF(d) = Σ weight_i / (k + rank_i(d))
Exact match boost: HybridSearch.ts multiplies a chunk's RRF score by
EXACT_MATCH_BOOST when the query string appears verbatim in it — RRF
scores alone peak too low (top-1 ≈ 1/(K+1) ≈ 0.016) to otherwise outrank
strong semantic neighbours scoring 0.45–0.55.
| Metric | Linear Search | HNSW Only | Hybrid (HNSW+BM25) |
|---|---|---|---|
| Time | 500ms | 15ms | 30ms |
| Precision@10 | 65% | 75% | 80% |
| Recall@10 | 45% | 55% | 60% |
User Query
│
├─► 1. Generate embedding
│ │
│ ▼
├─► 2. Hybrid search (HNSW + BM25)
│ │
│ ▼
├─► 3. Retrieve top-K chunks (K=10)
│ │
│ ▼
├─► 4. Optional: Graph expansion
│ │
│ ▼
├─► 5. Build context
│ │
│ ▼
└─► 6. Generate with LLM
│
▼
Response with citations
useGraphContext, graphSimilarityThreshold, and additionalGraphDocs are
declared in backend/types/config.ts and set by mode presets, and even
exposed in the RAG settings UI — but confirmed to have zero consumers
anywhere in retrieval-service.ts or backend/core/rag/. Turning this
setting on currently has no effect on retrieval; it isn't wired to
anything. Documented here as a known gap, not as working behavior — the
mechanism described in earlier versions of this page (graph-neighbor
lookup, cosine similarity expansion) does not exist in the code today.
File: src/main/services/fusion-chat-service.ts, formatContextAsSystemPrompt().
An earlier version of this page showed a fabricated Markdown-based builder
(# Context (sources), ## [Title, p.X] headers) that doesn't match the
real code at all — and contradicted this wiki's own
Brainstorm guide,
which correctly describes the real format. The actual builder produces
numbered TITRE/AUTEUR/EXTRAIT blocks, in French, with an explicit
rule forcing the model to answer only from those fields:
function formatContextAsSystemPrompt(hits: MultiSourceSearchResult[]): string {
const SNIPPET_CHARS = 1500;
const lines: string[] = [
'Contexte extrait du corpus indexé (sources numérotées ci-dessous).',
"RÈGLE : réponds UNIQUEMENT à partir des sources ci-dessous (champs TITRE, AUTEUR et EXTRAIT). Ne complète JAMAIS avec tes connaissances générales.",
// ...two more rule lines (TITRE-as-valid-answer for identification
// questions, explicit no-canned-phrase wording for "not found") —
// omitted here for brevity, present in the real function.
"Cite le numéro de la source [N] pour chaque affirmation.",
'',
];
hits.forEach((h, i) => {
const title = h.document.title || h.document.id || 'Sans titre';
const kind = h.sourceType === 'primary' ? 'archive'
: h.sourceType === 'vault' ? 'note' : 'bibliographie';
lines.push(`[${i + 1}] type : ${kind}`);
lines.push(`TITRE : ${title}`);
// ...AUTEUR (if present), EXTRAIT (with page number for secondary sources)
});
return lines.join('\n');
}Generated context example:
[1] type : bibliographie
TITRE : Active Learning in Higher Education
AUTEUR : Smith, 2019
EXTRAIT (p. 12) :
The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment...
Files: src/main/services/fusion-chat-service.ts (transport, IPC fusion:chat:*) and src/main/services/chat-engine.ts (agent loop, retrieval hook, compaction). There is no separate English "You are a research assistant..." prompt template — the system message the model actually receives is the TITRE/AUTEUR/EXTRAIT block from Step 5 above, prepended to the conversation. The call goes through whichever provider is configured (ProviderRegistry), not a hardcoded Ollama call:
const response = await provider.generate({
messages: [
{ role: 'system', content: formatContextAsSystemPrompt(hits) },
...conversationHistory,
],
stream: true,
});Streaming: Response is sent token by token for better UX
Stream Cancellation: Users can cancel ongoing generation via a per-session AbortController.
File: backend/core/rag/ContextCompressor.ts
This class is never called by the running app. grep -rln "ContextCompressor" --include="*.ts" . returns only the class's own file
and its own test (ContextCompressor.test.ts) — no service, IPC handler,
or chat path imports it. Filed as issue #28.
Confirmed from the consumer side too:
chat-engine.ts reads result.explanation.compression and only fires its
onCompressionStats hook when that field is truthy, but retrieval-service.ts
(the only place RAGExplanation objects are actually built) never assigns
a compression key to them — so that field is always undefined in
practice, and the "compressing" phase in chat-engine.ts never becomes
observable. Whatever context a query retrieves is sent to the LLM as-is,
uncompressed, regardless of size.
The design below is real (it's exactly what ContextCompressor.ts
implements) — this documents the intended algorithm, not something that
currently runs:
Sizes below are characters, not tokens — ContextCompressor.ts's own
comments say so explicitly ("Skip compression for small contexts (< 10k
chars)").
┌─────────────────────────────────────────────────────────────┐
│ Context Compression Levels (not wired up) │
├─────────────────────────────────────────────────────────────┤
│ │
│ No compression (context ≤ 10,000 chars) │
│ └── Passed through unchanged │
│ │
│ Level 1: Light (context 15,000-25,000 chars) │
│ └── Semantic deduplication only (threshold 0.88) │
│ │
│ Level 2: Medium (context 25,000-35,000 chars) │
│ └── Deduplication (0.85) + relevant-sentence extraction │
│ │
│ Level 3: Aggressive (context > 35,000 chars) │
│ └── Aggressive deduplication (0.80) + full extraction │
│ │
└─────────────────────────────────────────────────────────────┘
Contexts between 10,000 and 15,000 characters would fall through all three levels untouched even if this ran — there's a real gap between "skip" and "Level 1", not a continuous scale.
Algorithm, as implemented in the unused class:
- Calculate total context size in characters
- Select compression level based on size thresholds
- Apply relevance scoring to sentences (keyword + query similarity)
- Preserve keyword-rich sentences
- Return compressed context with reduction statistics
enableContextCompression is a per-mode config field (see
RAG Configuration Defaults below) that, per
the above, has no consumer either — toggling it in a mode has no observable
effect either way.
File: backend/types/chat-source.ts
ClioDeck provides transparency about how RAG responses are generated. The real shape is nested, not flat:
interface RAGExplanation {
search: {
query: string;
totalResults: number;
searchDurationMs: number;
cacheHit: boolean;
sourceType: 'primary' | 'secondary' | 'both';
documents: Array<{
title: string;
similarity: number;
sourceType: string;
chunkCount: number;
}>;
};
compression?: {
enabled: boolean;
originalChunks: number;
finalChunks: number;
originalSize: number;
finalSize: number;
reductionPercent: number;
strategy?: string;
};
graph?: {
enabled: boolean;
relatedDocsFound: number;
documentTitles: string[];
};
llm: {
provider: string;
model: string;
contextWindow: number;
temperature: number;
promptSize: number;
};
timing: {
searchMs: number;
compressionMs?: number;
generationMs: number;
totalMs: number;
};
}This information is displayed in the chat interface, allowing researchers to understand and evaluate the response context.
File: backend/integrations/zotero/ZoteroAPI.ts (not zotero-service.ts, which only orchestrates — the actual fetch() calls and headers live here)
// 1. Authentication
const headers = {
'Zotero-API-Key': apiKey,
'Zotero-API-Version': '3'
};
// 2. Retrieve collections
const collections = await fetch(
`https://api.zotero.org/users/${userId}/collections`,
{ headers }
);
// 3. Retrieve items from a collection
const items = await fetch(
`https://api.zotero.org/users/${userId}/collections/${collectionId}/items`,
{ headers }
);
// 4. BibTeX export
const bibtex = await fetch(
`https://api.zotero.org/users/${userId}/collections/${collectionId}/items?format=bibtex`,
{ headers }
);
// 5. Download PDFs
for (const item of items) {
if (item.data.linkMode === 'linked_file') {
const pdf = await downloadAttachment(item.key);
await savePDF(pdf, projectPath);
}
}Parsing: backend/core/bibliography/BibTeXParser.ts — a 484-line
BibTeXParser class, not a standalone regex-based function. The
illustrative snippet in earlier versions of this page (a ~20-line
single-regex function) drastically understated it; there is no
parseBibTeX function anywhere in the codebase. Custom-field round-trip
(preserving fields BibTeX doesn't natively model) is handled here, matched
by BibTeXExporter.ts on the way back out.
Service: backend/python-services/topic-modeling/main.py
The Python service does not compute embeddings itself — an earlier
version of this page showed it calling SentenceTransformer(...).encode()
internally, which contradicts the module's own docstring ("Reçoit des
embeddings pré-calculés depuis Electron/Ollama") and its request
schema, which requires embeddings as a mandatory input field
(min_length=5) alongside documents and document_ids. Embeddings come
from the TypeScript side, via whichever EmbeddingProvider is configured
(see Step 3 above) — sentence-transformers is in requirements.txt only
as a BERTopic transitive dependency, never imported directly in main.py.
from bertopic import BERTopic
from fastapi import FastAPI
import numpy as np
app = FastAPI()
@app.post("/analyze")
async def analyze_corpus(request: AnalyzeRequest):
# Embeddings arrive pre-computed — no SentenceTransformer call here
embeddings_array = np.array(request.embeddings, dtype=np.float32)
topic_model = BERTopic(min_topic_size=request.min_topic_size)
topics, _ = topic_model.fit_transform(request.documents, embeddings_array)
return {
"topics": topics, # per real AnalyzeResponse shape
"topic_assignments": {...}, # document_id -> topic id
"outliers": [...],
"statistics": {...},
}Communication: TypeScript backend communicates via HTTP (port 8001)
const response = await fetch('http://localhost:8001/analyze', {
method: 'POST',
body: JSON.stringify({
embeddings, // required — computed TypeScript-side first
documents: chunks,
document_ids: ids,
})
});
// Real response shape: {topics, topic_assignments, outliers, statistics}
// — not {topics, topic_info} as an earlier version of this page showed.
const { topics, topic_assignments, outliers, statistics } = await response.json();| Component | RAM | Description |
|---|---|---|
| Electron | 1.5 GB | App + Chromium |
| HNSW index | 500 MB | 50k × 768 dims × 3x overhead |
| BM25 index | 100 MB | Inverted index + vocabulary |
| SQLite | 300 MB | Chunks + metadata |
| Embedding provider | 0–500 MB | Ollama's nomic-embed-text loads ~500 MB in RAM locally; the embedded model and cloud providers have a different footprint (embedded: bundled with the app; cloud: none, runs remotely) |
| Total | ~3 GB | For an average project |
| Corpus Size | Min RAM | CPU | Indexing Time |
|---|---|---|---|
| 10-50 PDFs | 4 GB | Dual-core | 5-10 min |
| 50-200 PDFs | 8 GB | Quad-core | 20-40 min |
| 200-500 PDFs | 16 GB | 8+ cores | 1-2 hours |
1. Parallel chunking:
// Process documents in parallel (max 4 simultaneous)
const batches = chunk(documents, 4);
for (const batch of batches) {
await Promise.all(batch.map(doc => indexDocument(doc)));
}2. Batch embeddings:
// Generate embeddings in batches of 32 — illustrative strategy, not a
// literal citation; the real batched call is generateEmbeddings(texts)
// on the active EmbeddingProvider (see Step 3 above), not an
// ollama.batchEmbed() method (no such method exists).
for (let i = 0; i < chunks.length; i += 32) {
const batch = chunks.slice(i, i + 32);
const embeddings = await embeddingProvider.generateEmbeddings(batch);
}3. Background index building — illustrative strategy; HNSWVectorStore.ts has no rebuild() method, this section describes a deferred-work pattern in general terms, not a real call:
setTimeout(() => {
// build/refresh the HNSW index here
}, 5000);const CHUNKING_CONFIGS = {
// For modest machines (8 GB RAM)
cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },
// Balance performance/precision
standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },
// For maximum precision (16+ GB RAM)
large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};(Same maxTokens: 8192 fabrication as above — real config has no such field, see the note there.)
backend/
├── core/
│ ├── analysis/
│ │ └── TopicModelingService.ts # Thematic analysis + graph
│ ├── chunking/
│ │ ├── AdaptiveChunker.ts # Structure-aware chunking
│ │ └── DocumentChunker.ts # Simple chunking (fallback)
│ ├── history/
│ │ └── HistoryManager.ts # Research journal
│ ├── llm/
│ │ └── providers/ # Typed provider registry
│ │ ├── base.ts # LLMProvider / EmbeddingProvider contracts
│ │ ├── registry.ts # Open factory map, 5 providers + embedded
│ │ ├── ollama.ts, embedded.ts, anthropic.ts,
│ │ │ openai-compatible.ts, mistral.ts, gemini.ts
│ │ ├── fallback.ts # Ollama → embedded auto-fallback
│ │ └── cliodeck-config-adapter.ts
│ ├── pdf/
│ │ ├── PDFExtractor.ts # PDF extraction (pdfjs) — the actual getDocument/getTextContent/getMetadata calls
│ │ └── PDFIndexer.ts # Orchestrates extraction + indexing, delegates to PDFExtractor.ts
│ ├── search/
│ │ ├── BM25Index.ts # Sparse index (keywords)
│ │ └── HybridSearch.ts # Dense + sparse fusion
│ └── vector-store/
│ ├── VectorStore.ts # SQLite base store
│ ├── HNSWVectorStore.ts # Fast HNSW index
│ └── EnhancedVectorStore.ts # Unified wrapper
│
├── python-services/
│ └── topic-modeling/
│ ├── main.py # FastAPI service
│ ├── topic_analyzer.py # BERTopic logic
│ └── requirements.txt # Python dependencies
│
├── recipes/
│ ├── schema.ts, runner.ts # ClioRecipes: YAML workflow steps
│ └── builtin/ # Zotero review, Tropy analysis, etc.
│
├── types/
│ ├── config.ts # Configuration types
│ ├── book.ts # Book/chapters manifest + settings
│ └── pdf-document.ts # Document/chunk types
│
src/
├── editor/
│ ├── cm/ # CodeMirror 6 state, fidelity, proposals
│ ├── facade.ts # Editor-agnostic abstraction (EditorFacade)
│ ├── proposals/ # Adjudicable AI-proposal contract
│ └── slides.ts # Shared slide-splitting logic
│
├── main/
│ ├── services/
│ │ ├── fusion-chat-service.ts # Unified chat transport (IPC fusion:chat:*)
│ │ ├── chat-engine.ts # Shared agent loop, compaction, retrieval hook
│ │ ├── retrieval-service.ts # Multi-source RAG (PDFs + Tropy + Obsidian + manuscript)
│ │ ├── manuscript-assembler.ts # Book export assembly (per-chapter footnote IDs)
│ │ ├── manuscript-index-service.ts # Manuscript as a 4th RAG corpus
│ │ ├── mcp-clients-service.ts # External MCP server lifecycle
│ │ ├── usage-journal-service.ts # AI usage journal sink (.cliodeck/journal.db)
│ │ ├── pdf-service.ts # PDF management
│ │ ├── tropy-service.ts # Primary sources (Tropy)
│ │ ├── project-manager.ts # Project management
│ │ └── topic-modeling-service.ts # Python service proxy
│ └── ipc/
│ └── handlers/ # Electron IPC
│
└── renderer/
├── components/
│ ├── Chat/ # AssistantChat (single shell, variants full/panel)
│ ├── Config/ # Settings
│ ├── Editor/ # CodeMirror 6 editor
│ ├── Brainstorm/ # Chat + Ideas + Board + Graph tabs
│ ├── PDFIndex/ # PDF management
│ └── Journal/ # Research journal
└── stores/
└── editorStore.ts # Zustand state
backend/core/llm/OllamaClient.ts and src/main/services/chat-service.ts,
which older versions of this page listed here, no longer exist — replaced by
the provider registry and fusion-chat-service.ts/chat-engine.ts above.
PrimarySourcesVectorStore handles Tropy primary sources, but it stores its
SQLite data in the shared .cliodeck/brain.db (the same file PDFs and
research history use — see Key Architecture Files
above), not a standalone database. Only the HNSW index is its own file:
┌─────────────────────────────────────────────────────────────┐
│ Primary Sources Pipeline │
├─────────────────────────────────────────────────────────────┤
│ │
│ Tropy Project (.tropy / .tpy) │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ TropyReader │ ──► Read-only access to SQLite database │
│ └──────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ TropySync │ ──► Sync items, photos, notes, tags │
│ └──────────────┘ │
│ │ │
│ ├──► OCR Pipeline (Tesseract.js) for images │
│ │ │
│ ▼ │
│ ┌────────────────────────┐ │
│ │ PrimarySourcesVectorStore │ │
│ │ - SQLite: brain.db (shared) │ │
│ │ - HNSW: primary-hnsw.index │ │
│ │ - BM25: in-memory │ │
│ └────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
-
TropyReader opens the
.tpyfile in read-only mode - TropySync extracts items with metadata, notes, and photos
- Notes are collected as transcriptions (Tropy notes > OCR > manual)
- DocumentChunker segments transcriptions (same as PDFs)
- The configured embedding provider generates embeddings (Ollama's
nomic-embed-text, the embedded model, or a cloud provider) - PrimarySourcesVectorStore stores chunks with HNSW + BM25 indexes
When querying, both stores are searched and results are merged:
// In retrieval-service.ts
const pdfResults = await vectorStore.search(embedding, topK);
const tropyResults = await tropyService.search(query, { topK });
// Merge and deduplicate
const allResults = [...pdfResults, ...tropyResults]
.sort((a, b) => b.similarity - a.similarity)
.slice(0, topK);ClioDeck supports embedded LLMs via node-llama-cpp for fully offline
generation and embeddings — see the Embedded LLM Guide
for the canonical, up-to-date description (model sizes, prompt format,
troubleshooting). That page is the one to update if this changes again;
this section only sketches where it sits in the architecture.
The embedded provider is one entry in the same typed ProviderRegistry
used by Ollama and the four cloud providers (backend/core/llm/providers/)
— there is no separate "manager" class layered on top, and no requirement
to have Ollama installed at all. embedded is a valid, independent value for both generationProvider
and embeddingProvider in config (cliodeck-config-adapter.ts) — but
only generationProvider has a UI control, in Settings → LLM.
embeddingProvider has no UI anywhere; reaching embedded for embeddings
today means hand-editing the config file. See the
Embedded LLM Guide and
issue #18 — an
earlier version of this page wrongly implied both were UI-selectable.
An auto mode also exists for each, which prefers Ollama when reachable
and falls back to the embedded model (backend/core/llm/providers/fallback.ts).
For embeddings specifically, the auto fallback only engages if the
embedded and Ollama embedding models share the same vector dimension
(cliodeck-config-adapter.ts's embeddingWithFallback()) — mixing
dimensions would silently corrupt similarity search, so a mismatch is
left to fail loudly instead.
Current default values live on the rag key of DEFAULT_CONFIG in
backend/types/config.ts — there is no separately-named DEFAULT_RAG_CONFIG
constant, just this nested object:
const DEFAULT_CONFIG = {
// ...
rag: {
// Search parameters
topK: 10, // Number of chunks to retrieve
similarityThreshold: 0.12, // RRF-optimized (not cosine similarity)
// Chunking
chunkingConfig: 'cpuOptimized', // 300 words, 50 overlap
useAdaptiveChunking: true,
// Indexes
useHNSWIndex: true,
useHybridSearch: true, // HNSW + BM25
// Quality
enableQualityFiltering: true,
enablePreprocessing: true,
enableDeduplication: true,
// Advanced (optional)
useSemanticChunking: false,
},
};enableContextCompression is not part of this object at all. An earlier
version of this page listed it here as a single global default (true) —
in the real code it's a per-mode override, set in
backend/core/modes/built-in-modes.ts: true for primary-source-analyst,
academic-writer, and methodology-assistant; false for
literature-review, critical-reviewer, and brainstorming; unset for
default-assistant and free-mode. None of this matters in practice: as
Context Compression above
explains, the class this flag would toggle is never called by anything —
there is no call site to fall back to, and the flag has zero observable
effect regardless of which mode sets it or to what.
The default similarityThreshold of 0.12 is optimized for RRF (Reciprocal Rank Fusion) scores, which are much smaller than raw cosine similarity scores. Typical RRF scores range from 0.001 to 0.02.
- HNSW: Efficient and robust approximate nearest neighbor search (Malkov & Yashunin, 2016)
- BM25: The Probabilistic Relevance Framework: BM25 and Beyond (Robertson & Zaragoza, 2009)
- RRF: Reciprocal rank fusion outperforms condorcet (Cormack et al., 2009)
- RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
- Ollama - Local LLMs
- node-llama-cpp - Embedded LLM execution
- hnswlib - HNSW implementation
- natural - NLP for Node.js
- BERTopic - Topic modeling
- pdfjs-dist - PDF rendering
- better-sqlite3 - Synchronous SQLite
- Tesseract.js - OCR for primary sources
- Tropy - Primary sources management
Version: 1.0.0-rc.3