-
Notifications
You must be signed in to change notification settings - Fork 0
2. Technical Architecture
This document describes the architecture and operation of ClioDeck's RAG (Retrieval-Augmented Generation) system, an academic writing assistant for historians and humanities researchers.
- Overview
- Global Architecture
- Indexing Pipeline
- Search System
- Assisted Generation (RAG)
- External Integrations
- Performance and Optimizations
ClioDeck is an Electron application that combines:
- Frontend: React/TypeScript interface with a CodeMirror 6 editor (live markdown rendering)
- Backend: Node.js/TypeScript services for indexing and search
- Python Services: Topic modeling and textometric analysis
-
LLM: a typed provider registry — Ollama, embedded (
node-llama-cpp), Anthropic, OpenAI-compatible, Mistral, Gemini — for embeddings and text generation - Storage: SQLite for data and vector indexes
- Local-first: All data remains on the user's machine
- Performance: Optimized for modest machines (8-16 GB RAM, CPU only)
- Academic: Complete traceability and verifiable citations
- Interoperability: Integration with Zotero, BibTeX, Markdown, PDF
┌─────────────────────────────────────────────────────────────┐
│ ClioDeck Electron App │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Frontend │ │ Backend │ │ Services │ │
│ │ React/Vite │◄─┤ TypeScript │◄─┤ Python │ │
│ │ │ │ │ │ │ │
│ │ • Editor │ │ • PDF Index │ │ • Topic │ │
│ │ • Chat UI │ │ • Vector DB │ │ Modeling │ │
│ │ • Config │ │ • Search │ │ • BERTopic │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Ollama │ │ SQLite │ │
│ │ (Local) │ │ + HNSW Index │ │
│ └──────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
| Component | Technology | Role |
|---|---|---|
| UI Framework | React 18 + TypeScript | User interface |
| Editor | CodeMirror 6 + Lezer | Live markdown rendering, Pandoc dialect |
| Build Tool | Vite | Bundling and hot reload |
| Desktop | Electron 40 | Multi-platform desktop app |
| Database | better-sqlite3 | Synchronous local storage |
| Vector Index | hnswlib-node | Fast vector search |
| Sparse Index | natural (BM25) | Keyword search |
| LLM | Typed provider registry | Ollama, embedded, Anthropic, OpenAI-compatible, Mistral, Gemini |
| PDF Processing | pdfjs-dist | Text extraction |
| Bibliography | BibTeX parsing | Reference management |
| Topic Modeling | Python + BERTopic | Thematic analysis |
The indexing pipeline transforms PDFs into queryable vector chunks.
File: backend/core/pdf/PDFExtractor.ts (not PDFIndexer.ts, which only orchestrates and delegates here)
// 1. Load PDF with pdfjs
const pdf = await pdfjs.getDocument(pdfPath).promise;
// 2. Extract text page by page
const pages = [];
for (let i = 1; i <= pdf.numPages; i++) {
const page = await pdf.getPage(i);
const textContent = await page.getTextContent();
const text = textContent.items.map(item => item.str).join(' ');
pages.push({ pageNumber: pageNum, text }); // field is `text`, not `content` — PDFExtractor.ts:174-176
}
// 3. Extract metadata
const metadata = await pdf.getMetadata();
const title = metadata.info.Title || path.basename(pdfPath);Output: Array of pages with raw text + document metadata
Files:
-
backend/core/chunking/AdaptiveChunker.ts(recommended) -
backend/core/chunking/DocumentChunker.ts(fallback) -
backend/core/chunking/SemanticChunker.ts(advanced semantic-aware chunking) -
backend/core/chunking/ChunkDeduplicator.ts(removes near-duplicate chunks) -
backend/core/chunking/ChunkQualityScorer.ts(scores chunk quality for filtering) -
backend/core/chunking/EmbeddingCache.ts(caches embeddings for performance)
The adaptive chunker automatically detects document structure:
// Header detection — patterns as they actually are in AdaptiveChunker.ts,
// stricter than earlier versions of this page implied
const patterns = {
markdown: /^#{1,6}\s+(.+)/, // # Title, ## Subtitle
numbered: /^(\d+(?:\.\d+)*)\.\s+([A-Z][^.]{2,50})$/, // "1. Introduction" — needs a trailing dot and a capitalized 2-50 char title
uppercase: /^([A-Z][A-Z\s]{5,50})$/, // ALL CAPS, 6-51 chars, also checked against a known-header allowlist (ABSTRACT, INTRODUCTION, …)
roman: /^([IVX]+)\.\s+([A-Z][^.]{2,50})$/ // "I. Introduction" — same capitalized-title constraint as `numbered`, not a bare "I. anything" match
};
// Automatic classification
const sectionTypes = {
abstract: /abstract|résumé/i,
introduction: /introduction/i,
methodology: /method|méthodologie/i,
results: /results|résultats/i,
discussion: /discussion/i,
conclusion: /conclusion/i,
references: /references|bibliographie/i
};Principles:
-
Respect sentence boundaries: Chunks always end with
.,!,?or; - Document context: Each chunk includes title and section as prefix
- Skip references: Bibliography section is not indexed
- Smart overlap: Overlaps occur on complete sentences
Generated chunk example:
[Doc: Active Learning Strategies in Higher Education | Section: Methodology]
The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment. Students were divided
into control and experimental groups. The control group followed traditional
lecture-based instruction, while the experimental group participated in
collaborative problem-solving activities.
Configuration:
interface ChunkingConfig {
maxChunkSize: number; // Target max size in words (300-800)
overlapSize: number; // Overlap in words (50-100)
minChunkSize: number; // Minimum words, avoids tiny leftover chunks
}
const CHUNKING_CONFIGS = {
cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },
standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },
large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};There is no maxTokens field anywhere in this config — an earlier version
of this page invented one; token limits for embedding models are handled
separately, not as part of chunking config.
Output: Array of chunks with metadata
interface DocumentChunk {
id: string; // UUID
documentId: string; // Document reference
content: string; // Chunk text (with context)
pageNumber: number; // Source page
chunkIndex: number; // Position in document
startPosition: number; // Character offset, start
endPosition: number; // Character offset, end
// No `embedding` field here — embeddings live in the DB row
// (pdf_chunks.embedding) or the vector store, not on this object.
metadata?: {
sectionTitle?: string; // "Methodology", "Results"
sectionType?: string; // "methodology", "results"
sectionLevel?: number; // 1, 2, 3...
};
}File: backend/core/llm/providers/ollama.ts (one of six embedding providers behind the typed registry — see Embedded LLM Support below for the others)
// 1. Connect to Ollama — the MODERN /api/embed endpoint, batched, not the
// legacy /api/embeddings single-prompt endpoint an earlier version of this
// page showed. ollama.ts's own comment explains why: "the legacy
// /api/embeddings endpoint ignores `truncate` and 500s — which is exactly
// the PDF-indexing crash this replaces."
const response = await fetch('http://localhost:11434/api/embed', {
method: 'POST',
body: JSON.stringify({
model: 'nomic-embed-text', // 768 dimensions
input: [chunkContent, /* ...more chunks, batched */],
truncate: true
})
});
// 2. Retrieve vectors (one per input, in order)
const { embeddings } = await response.json();
// embeddings: number[][] (768 dimensions each)Model used: nomic-embed-text
- Dimensions: 768
- Max context: 8192 tokens (model-dependent —
nomic-embed-textitself advertises 2048 and ignores largernum_ctxvalues) - Size: ~274 MB
- Performance: ~100 embeddings/second on average CPU
Oversized input, Ollama path: inputs that overflow the model's context are
truncated server-side by Ollama itself (truncate: true in the /api/embed
call above) — ollama.ts does no client-side chunking of its own; the comment
in the file is explicit that server-side truncation happens "rather than"
splitting the text.
Oversized input, embedded-model path: the chunk-and-average chunkText()
method shown below is real, but it belongs to EmbeddedLLMClient.ts (the
node-llama-cpp embedded provider), not Ollama — an earlier version of this
page misattributed it here. Its threshold is EMBEDDING_MAX_CHUNK_LENGTH = 2000
characters, not "8192 tokens":
// EmbeddedLLMClient.ts — embedded provider only, not Ollama
private chunkText(text: string, maxLength: number): string[] {
// Find sentence ending in last 200 characters
const sentenceEndings = /[.!?;](?=\s|$)/g;
// ... cut at last found period
}
// Chunks are embedded individually, then averaged into a single vector.File: backend/core/vector-store/EnhancedVectorStore.ts
Three indexes are created simultaneously:
-- Real schema, VectorStore.ts:104-113 — table is pdf_chunks, not chunks;
-- no metadata/created_at columns; start_position/end_position instead
CREATE TABLE IF NOT EXISTS pdf_chunks (
id TEXT PRIMARY KEY,
document_id TEXT NOT NULL,
content TEXT NOT NULL,
page_number INTEGER NOT NULL,
chunk_index INTEGER NOT NULL,
start_position INTEGER NOT NULL,
end_position INTEGER NOT NULL,
embedding BLOB,
FOREIGN KEY (document_id) REFERENCES pdf_documents(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_chunks_document_id ON pdf_chunks(document_id);
CREATE INDEX IF NOT EXISTS idx_chunks_page_number ON pdf_chunks(page_number);Role: Persistent storage + fallback for linear search
File: backend/core/vector-store/HNSWVectorStore.ts
import { HierarchicalNSW } from 'hnswlib-node';
const index = new HierarchicalNSW('cosine', 768); // 768 dimensions
index.initIndex(maxElements, M=16, efConstruction=100);
// Indexing
for (const chunk of chunks) {
index.addPoint(chunk.embedding, chunk.id);
}
// Save to disk
index.writeIndexSync(indexPath);Parameters:
-
M=16: Connections per node (speed/precision trade-off) -
efConstruction=100: Construction effort -
efSearch=50: Search effort
Complexity:
- Construction: O(n log n)
- Search: O(log n)
Memory footprint: ~500 MB for 50k chunks (768 dims)
File: backend/core/search/BM25Index.ts
There is no hand-rolled inverted index — indexing delegates entirely to
natural's TfIdf class, with BM25 scoring layered on top of it:
import natural from 'natural';
const { TfIdf } = natural;
// 1. Delegate document storage/tokenization to natural's TfIdf
const tfidf = new TfIdf();
tfidf.addDocument(preprocessText(chunk.content));
// 2. BM25 scoring, computed from TfIdf's term/document frequencies
const idf = Math.log((totalDocs - docFreq + 0.5) / (docFreq + 0.5));
const score = idf * ((tf * (k1 + 1)) / (tf + k1 * (1 - b + b * (docLength / avgDocLength))));Parameters:
-
k1=1.5: Term frequency saturation -
b=0.75: Document length normalization
Memory footprint: ~100 MB for 50k chunks
File: backend/core/search/HybridSearch.ts
Hybrid search combines the advantages of two complementary approaches:
Query: "bloom taxonomy methodology"
│
├────────────────────┬────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Dense │ │ Sparse │ │ Fusion │
│ (HNSW) │ │ (BM25) │ │ (RRF) │
└──────────┘ └──────────┘ └──────────┘
│ │ │
▼ ▼ ▼
Top-50 results Top-50 results Top-10 final
(semantic) (keywords) (best of both)
// 1. Generate query embedding via the configured EmbeddingProvider
// (Ollama, embedded, or a cloud provider — see Embedded LLM Support below)
const queryEmbedding = await embeddingProvider.generateEmbedding(query);
// 2. HNSW search
const denseResults = await hnswStore.search(queryEmbedding, 50);
// Returns: [{ id, score }, ...]Advantage: Understands semantic meaning ("active learning" ≈ "apprentissage actif")
// 1. Query tokenization
const queryTokens = tokenizer.tokenize(query.toLowerCase());
// 2. BM25 scoring
const sparseResults = bm25Index.search(queryTokens, 50);
// Returns: [{ id, score }, ...]Advantage: Excellent for rare terms, proper nouns, acronyms ("BERTopic", "Piaget")
function reciprocalRankFusion(
denseResults: SearchResult[],
sparseResults: SearchResult[],
k: number = 60
): SearchResult[] {
const scores = new Map<string, number>();
// Dense search contribution (60%)
for (let i = 0; i < denseResults.length; i++) {
const chunkId = denseResults[i].id;
const rrfScore = 0.6 / (k + i + 1);
scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
}
// Sparse search contribution (40%)
for (let i = 0; i < sparseResults.length; i++) {
const chunkId = sparseResults[i].id;
const rrfScore = 0.4 / (k + i + 1);
scores.set(chunkId, (scores.get(chunkId) || 0) + rrfScore);
}
// Sort by final score
return Array.from(scores.entries())
.sort((a, b) => b[1] - a[1])
.slice(0, 10)
.map(([id, score]) => ({ id, score }));
}Weights: 60% dense / 40% sparse (configurable)
RRF Formula: RRF(d) = Σ weight_i / (k + rank_i(d))
Exact match boost: HybridSearch.ts multiplies a chunk's RRF score by
EXACT_MATCH_BOOST when the query string appears verbatim in it — RRF
scores alone peak too low (top-1 ≈ 1/(K+1) ≈ 0.016) to otherwise outrank
strong semantic neighbours scoring 0.45–0.55.
| Metric | Linear Search | HNSW Only | Hybrid (HNSW+BM25) |
|---|---|---|---|
| Time | 500ms | 15ms | 30ms |
| Precision@10 | 65% | 75% | 80% |
| Recall@10 | 45% | 55% | 60% |
User Query
│
├─► 1. Generate embedding
│ │
│ ▼
├─► 2. Hybrid search (HNSW + BM25)
│ │
│ ▼
├─► 3. Retrieve top-K chunks (K=10)
│ │
│ ▼
├─► 4. Optional: Graph expansion
│ │
│ ▼
├─► 5. Build context
│ │
│ ▼
└─► 6. Generate with LLM
│
▼
Response with citations
useGraphContext, graphSimilarityThreshold, and additionalGraphDocs are
declared in backend/types/config.ts and set by mode presets, and even
exposed in the RAG settings UI — but confirmed to have zero consumers
anywhere in retrieval-service.ts or backend/core/rag/. Turning this
setting on currently has no effect on retrieval; it isn't wired to
anything. Documented here as a known gap, not as working behavior — the
mechanism described in earlier versions of this page (graph-neighbor
lookup, cosine similarity expansion) does not exist in the code today.
File: src/main/services/fusion-chat-service.ts, formatContextAsSystemPrompt().
An earlier version of this page showed a fabricated Markdown-based builder
(# Context (sources), ## [Title, p.X] headers) that doesn't match the
real code at all — and contradicted this wiki's own
Brainstorm guide,
which correctly describes the real format. The actual builder produces
numbered TITRE/AUTEUR/EXTRAIT blocks, in French, with an explicit
rule forcing the model to answer only from those fields:
function formatContextAsSystemPrompt(hits: MultiSourceSearchResult[]): string {
const SNIPPET_CHARS = 1500;
const lines: string[] = [
'Contexte extrait du corpus indexé (sources numérotées ci-dessous).',
"RÈGLE : réponds UNIQUEMENT à partir des sources ci-dessous (champs TITRE, AUTEUR et EXTRAIT). Ne complète JAMAIS avec tes connaissances générales.",
// ...two more rule lines (TITRE-as-valid-answer for identification
// questions, explicit no-canned-phrase wording for "not found") —
// omitted here for brevity, present in the real function.
"Cite le numéro de la source [N] pour chaque affirmation.",
'',
];
hits.forEach((h, i) => {
const title = h.document.title || h.document.id || 'Sans titre';
const kind = h.sourceType === 'primary' ? 'archive'
: h.sourceType === 'vault' ? 'note' : 'bibliographie';
lines.push(`[${i + 1}] type : ${kind}`);
lines.push(`TITRE : ${title}`);
// ...AUTEUR (if present), EXTRAIT (with page number for secondary sources)
});
return lines.join('\n');
}Generated context example:
[1] type : bibliographie
TITRE : Active Learning in Higher Education
AUTEUR : Smith, 2019
EXTRAIT (p. 12) :
The intervention consisted of three phases: pre-assessment, active learning
activities using Bloom's taxonomy, and post-assessment...
Files: src/main/services/fusion-chat-service.ts (transport, IPC fusion:chat:*) and src/main/services/chat-engine.ts (agent loop, retrieval hook, compaction). There is no separate English "You are a research assistant..." prompt template — the system message the model actually receives is the TITRE/AUTEUR/EXTRAIT block from Step 5 above, prepended to the conversation. The call goes through whichever provider is configured (ProviderRegistry), not a hardcoded Ollama call:
const response = await provider.generate({
messages: [
{ role: 'system', content: formatContextAsSystemPrompt(hits) },
...conversationHistory,
],
stream: true,
});Streaming: Response is sent token by token for better UX
Stream Cancellation: Users can cancel ongoing generation via a per-session AbortController.
File: backend/core/rag/ContextCompressor.ts
When the retrieved context exceeds the LLM's context window, ClioDeck applies intelligent compression:
Sizes below are characters, not tokens — ContextCompressor.ts's own
comments say so explicitly ("Skip compression for small contexts (< 10k
chars)").
┌─────────────────────────────────────────────────────────────┐
│ Context Compression Levels │
├─────────────────────────────────────────────────────────────┤
│ │
│ No compression (context ≤ 10,000 chars) │
│ └── Passed through unchanged │
│ │
│ Level 1: Light (context 15,000-25,000 chars) │
│ └── Semantic deduplication only (threshold 0.88) │
│ │
│ Level 2: Medium (context 25,000-35,000 chars) │
│ └── Deduplication (0.85) + relevant-sentence extraction │
│ │
│ Level 3: Aggressive (context > 35,000 chars) │
│ └── Aggressive deduplication (0.80) + full extraction │
│ │
└─────────────────────────────────────────────────────────────┘
Contexts between 10,000 and 15,000 characters fall through all three levels untouched — there's a real gap between "skip" and "Level 1", not a continuous scale.
Algorithm:
- Calculate total context size in characters
- Select compression level based on size thresholds
- Apply relevance scoring to sentences (keyword + query similarity)
- Preserve keyword-rich sentences
- Return compressed context with reduction statistics
Configuration:
-
enableContextCompression: boolean(default: true) - Automatic level selection based on context size
File: backend/types/chat-source.ts
ClioDeck provides transparency about how RAG responses are generated. The real shape is nested, not flat:
interface RAGExplanation {
search: {
query: string;
totalResults: number;
searchDurationMs: number;
cacheHit: boolean;
sourceType: 'primary' | 'secondary' | 'both';
documents: Array<{
title: string;
similarity: number;
sourceType: string;
chunkCount: number;
}>;
};
compression?: {
enabled: boolean;
originalChunks: number;
finalChunks: number;
originalSize: number;
finalSize: number;
reductionPercent: number;
strategy?: string;
};
graph?: {
enabled: boolean;
relatedDocsFound: number;
documentTitles: string[];
};
llm: {
provider: string;
model: string;
contextWindow: number;
temperature: number;
promptSize: number;
};
timing: {
searchMs: number;
compressionMs?: number;
generationMs: number;
totalMs: number;
};
}This information is displayed in the chat interface, allowing researchers to understand and evaluate the response context.
File: backend/integrations/zotero/ZoteroAPI.ts (not zotero-service.ts, which only orchestrates — the actual fetch() calls and headers live here)
// 1. Authentication
const headers = {
'Zotero-API-Key': apiKey,
'Zotero-API-Version': '3'
};
// 2. Retrieve collections
const collections = await fetch(
`https://api.zotero.org/users/${userId}/collections`,
{ headers }
);
// 3. Retrieve items from a collection
const items = await fetch(
`https://api.zotero.org/users/${userId}/collections/${collectionId}/items`,
{ headers }
);
// 4. BibTeX export
const bibtex = await fetch(
`https://api.zotero.org/users/${userId}/collections/${collectionId}/items?format=bibtex`,
{ headers }
);
// 5. Download PDFs
for (const item of items) {
if (item.data.linkMode === 'linked_file') {
const pdf = await downloadAttachment(item.key);
await savePDF(pdf, projectPath);
}
}Parsing: backend/core/bibliography/BibTeXParser.ts — a 484-line
BibTeXParser class, not a standalone regex-based function. The
illustrative snippet in earlier versions of this page (a ~20-line
single-regex function) drastically understated it; there is no
parseBibTeX function anywhere in the codebase. Custom-field round-trip
(preserving fields BibTeX doesn't natively model) is handled here, matched
by BibTeXExporter.ts on the way back out.
Service: backend/python-services/topic-modeling/main.py
The Python service does not compute embeddings itself — an earlier
version of this page showed it calling SentenceTransformer(...).encode()
internally, which contradicts the module's own docstring ("Reçoit des
embeddings pré-calculés depuis Electron/Ollama") and its request
schema, which requires embeddings as a mandatory input field
(min_length=5) alongside documents and document_ids. Embeddings come
from the TypeScript side, via whichever EmbeddingProvider is configured
(see Step 3 above) — sentence-transformers is in requirements.txt only
as a BERTopic transitive dependency, never imported directly in main.py.
from bertopic import BERTopic
from fastapi import FastAPI
import numpy as np
app = FastAPI()
@app.post("/analyze")
async def analyze_corpus(request: AnalyzeRequest):
# Embeddings arrive pre-computed — no SentenceTransformer call here
embeddings_array = np.array(request.embeddings, dtype=np.float32)
topic_model = BERTopic(min_topic_size=request.min_topic_size)
topics, _ = topic_model.fit_transform(request.documents, embeddings_array)
return {
"topics": topics, # per real AnalyzeResponse shape
"topic_assignments": {...}, # document_id -> topic id
"outliers": [...],
"statistics": {...},
}Communication: TypeScript backend communicates via HTTP (port 8001)
const response = await fetch('http://localhost:8001/analyze', {
method: 'POST',
body: JSON.stringify({
embeddings, // required — computed TypeScript-side first
documents: chunks,
document_ids: ids,
})
});
// Real response shape: {topics, topic_assignments, outliers, statistics}
// — not {topics, topic_info} as an earlier version of this page showed.
const { topics, topic_assignments, outliers, statistics } = await response.json();| Component | RAM | Description |
|---|---|---|
| Electron | 1.5 GB | App + Chromium |
| HNSW index | 500 MB | 50k × 768 dims × 3x overhead |
| BM25 index | 100 MB | Inverted index + vocabulary |
| SQLite | 300 MB | Chunks + metadata |
| Embedding provider | 0–500 MB | Ollama's nomic-embed-text loads ~500 MB in RAM locally; the embedded model and cloud providers have a different footprint (embedded: bundled with the app; cloud: none, runs remotely) |
| Total | ~3 GB | For an average project |
| Corpus Size | Min RAM | CPU | Indexing Time |
|---|---|---|---|
| 10-50 PDFs | 4 GB | Dual-core | 5-10 min |
| 50-200 PDFs | 8 GB | Quad-core | 20-40 min |
| 200-500 PDFs | 16 GB | 8+ cores | 1-2 hours |
1. Parallel chunking:
// Process documents in parallel (max 4 simultaneous)
const batches = chunk(documents, 4);
for (const batch of batches) {
await Promise.all(batch.map(doc => indexDocument(doc)));
}2. Batch embeddings:
// Generate embeddings in batches of 32 — illustrative strategy, not a
// literal citation; the real batched call is generateEmbeddings(texts)
// on the active EmbeddingProvider (see Step 3 above), not an
// ollama.batchEmbed() method (no such method exists).
for (let i = 0; i < chunks.length; i += 32) {
const batch = chunks.slice(i, i + 32);
const embeddings = await embeddingProvider.generateEmbeddings(batch);
}3. Background index building — illustrative strategy; HNSWVectorStore.ts has no rebuild() method, this section describes a deferred-work pattern in general terms, not a real call:
setTimeout(() => {
// build/refresh the HNSW index here
}, 5000);const CHUNKING_CONFIGS = {
// For modest machines (8 GB RAM)
cpuOptimized: { maxChunkSize: 300, overlapSize: 50, minChunkSize: 50 },
// Balance performance/precision
standard: { maxChunkSize: 500, overlapSize: 75, minChunkSize: 100 },
// For maximum precision (16+ GB RAM)
large: { maxChunkSize: 800, overlapSize: 100, minChunkSize: 150 }
};(Same maxTokens: 8192 fabrication as above — real config has no such field, see the note there.)
backend/
├── core/
│ ├── analysis/
│ │ └── TopicModelingService.ts # Thematic analysis + graph
│ ├── chunking/
│ │ ├── AdaptiveChunker.ts # Structure-aware chunking
│ │ └── DocumentChunker.ts # Simple chunking (fallback)
│ ├── history/
│ │ └── HistoryManager.ts # Research journal
│ ├── llm/
│ │ └── providers/ # Typed provider registry
│ │ ├── base.ts # LLMProvider / EmbeddingProvider contracts
│ │ ├── registry.ts # Open factory map, 5 providers + embedded
│ │ ├── ollama.ts, embedded.ts, anthropic.ts,
│ │ │ openai-compatible.ts, mistral.ts, gemini.ts
│ │ ├── fallback.ts # Ollama → embedded auto-fallback
│ │ └── cliodeck-config-adapter.ts
│ ├── pdf/
│ │ ├── PDFExtractor.ts # PDF extraction (pdfjs) — the actual getDocument/getTextContent/getMetadata calls
│ │ └── PDFIndexer.ts # Orchestrates extraction + indexing, delegates to PDFExtractor.ts
│ ├── search/
│ │ ├── BM25Index.ts # Sparse index (keywords)
│ │ └── HybridSearch.ts # Dense + sparse fusion
│ └── vector-store/
│ ├── VectorStore.ts # SQLite base store
│ ├── HNSWVectorStore.ts # Fast HNSW index
│ └── EnhancedVectorStore.ts # Unified wrapper
│
├── python-services/
│ └── topic-modeling/
│ ├── main.py # FastAPI service
│ ├── topic_analyzer.py # BERTopic logic
│ └── requirements.txt # Python dependencies
│
├── recipes/
│ ├── schema.ts, runner.ts # ClioRecipes: YAML workflow steps
│ └── builtin/ # Zotero review, Tropy analysis, etc.
│
├── types/
│ ├── config.ts # Configuration types
│ ├── book.ts # Book/chapters manifest + settings
│ └── pdf-document.ts # Document/chunk types
│
src/
├── editor/
│ ├── cm/ # CodeMirror 6 state, fidelity, proposals
│ ├── facade.ts # Editor-agnostic abstraction (EditorFacade)
│ ├── proposals/ # Adjudicable AI-proposal contract
│ └── slides.ts # Shared slide-splitting logic
│
├── main/
│ ├── services/
│ │ ├── fusion-chat-service.ts # Unified chat transport (IPC fusion:chat:*)
│ │ ├── chat-engine.ts # Shared agent loop, compaction, retrieval hook
│ │ ├── retrieval-service.ts # Multi-source RAG (PDFs + Tropy + Obsidian + manuscript)
│ │ ├── manuscript-assembler.ts # Book export assembly (per-chapter footnote IDs)
│ │ ├── manuscript-index-service.ts # Manuscript as a 4th RAG corpus
│ │ ├── mcp-clients-service.ts # External MCP server lifecycle
│ │ ├── usage-journal-service.ts # AI usage journal sink (.cliodeck/journal.db)
│ │ ├── pdf-service.ts # PDF management
│ │ ├── tropy-service.ts # Primary sources (Tropy)
│ │ ├── project-manager.ts # Project management
│ │ └── topic-modeling-service.ts # Python service proxy
│ └── ipc/
│ └── handlers/ # Electron IPC
│
└── renderer/
├── components/
│ ├── Chat/ # AssistantChat (single shell, variants full/panel)
│ ├── Config/ # Settings
│ ├── Editor/ # CodeMirror 6 editor
│ ├── Brainstorm/ # Chat + Ideas + Board + Graph tabs
│ ├── PDFIndex/ # PDF management
│ └── Journal/ # Research journal
└── stores/
└── editorStore.ts # Zustand state
backend/core/llm/OllamaClient.ts and src/main/services/chat-service.ts,
which older versions of this page listed here, no longer exist — replaced by
the provider registry and fusion-chat-service.ts/chat-engine.ts above.
PrimarySourcesVectorStore handles Tropy primary sources, but it stores its
SQLite data in the shared .cliodeck/brain.db (the same file PDFs and
research history use — see Key Architecture Files
above), not a standalone database. Only the HNSW index is its own file:
┌─────────────────────────────────────────────────────────────┐
│ Primary Sources Pipeline │
├─────────────────────────────────────────────────────────────┤
│ │
│ Tropy Project (.tropy / .tpy) │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ TropyReader │ ──► Read-only access to SQLite database │
│ └──────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ TropySync │ ──► Sync items, photos, notes, tags │
│ └──────────────┘ │
│ │ │
│ ├──► OCR Pipeline (Tesseract.js) for images │
│ │ │
│ ▼ │
│ ┌────────────────────────┐ │
│ │ PrimarySourcesVectorStore │ │
│ │ - SQLite: brain.db (shared) │ │
│ │ - HNSW: primary-hnsw.index │ │
│ │ - BM25: in-memory │ │
│ └────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
-
TropyReader opens the
.tpyfile in read-only mode - TropySync extracts items with metadata, notes, and photos
- Notes are collected as transcriptions (Tropy notes > OCR > manual)
- DocumentChunker segments transcriptions (same as PDFs)
- The configured embedding provider generates embeddings (Ollama's
nomic-embed-text, the embedded model, or a cloud provider) - PrimarySourcesVectorStore stores chunks with HNSW + BM25 indexes
When querying, both stores are searched and results are merged:
// In retrieval-service.ts
const pdfResults = await vectorStore.search(embedding, topK);
const tropyResults = await tropyService.search(query, { topK });
// Merge and deduplicate
const allResults = [...pdfResults, ...tropyResults]
.sort((a, b) => b.similarity - a.similarity)
.slice(0, topK);ClioDeck supports embedded LLMs via node-llama-cpp for fully offline
generation and embeddings — see the Embedded LLM Guide
for the canonical, up-to-date description (model sizes, prompt format,
troubleshooting). That page is the one to update if this changes again;
this section only sketches where it sits in the architecture.
The embedded provider is one entry in the same typed ProviderRegistry
used by Ollama and the four cloud providers (backend/core/llm/providers/)
— there is no separate "manager" class layered on top, and no requirement
to have Ollama installed at all. embedded can be selected independently
for both generation and embeddings from Settings → LLM
(generationProvider / embeddingProvider, cliodeck-config-adapter.ts).
An auto mode also exists, which prefers Ollama when reachable and falls
back to the embedded model (backend/core/llm/providers/fallback.ts) —
but that's a convenience default, not the only way to reach the embedded
path.
Current default values (backend/types/config.ts) — the values below are
still accurate as of v1.0.0-rc.3, verified fresh; only this section's old
"beta.2" framing was stale:
const DEFAULT_RAG_CONFIG = {
// Search parameters
topK: 10, // Number of chunks to retrieve
similarityThreshold: 0.12, // RRF-optimized (not cosine similarity)
// Chunking
chunkingConfig: 'cpuOptimized', // 300 words, 50 overlap
useAdaptiveChunking: true,
// Indexes
useHNSWIndex: true,
useHybridSearch: true, // HNSW + BM25
// Quality
enableQualityFiltering: true,
enablePreprocessing: true,
enableDeduplication: true,
// Advanced (optional)
useSemanticChunking: false,
enableContextCompression: true // Default: true
};The default similarityThreshold of 0.12 is optimized for RRF (Reciprocal Rank Fusion) scores, which are much smaller than raw cosine similarity scores. Typical RRF scores range from 0.001 to 0.02.
- HNSW: Efficient and robust approximate nearest neighbor search (Malkov & Yashunin, 2016)
- BM25: The Probabilistic Relevance Framework: BM25 and Beyond (Robertson & Zaragoza, 2009)
- RRF: Reciprocal rank fusion outperforms condorcet (Cormack et al., 2009)
- RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
- Ollama - Local LLMs
- node-llama-cpp - Embedded LLM execution
- hnswlib - HNSW implementation
- natural - NLP for Node.js
- BERTopic - Topic modeling
- pdfjs-dist - PDF rendering
- better-sqlite3 - Synchronous SQLite
- Tesseract.js - OCR for primary sources
- Tropy - Primary sources management
Version: 1.0.0-rc.3