v0.10.0

Oasis v0.10.0 is a RAG-focused release — multimodal image embedding, production-grade SQLite performance, ingest checkpoint & resume, OpenAI-compatible embeddings, and a comprehensive wave of RAG pipeline hardening across retrieval, graph extraction, and storage layers.

---

✨ Highlights

🖼️ Multimodal Image Embedding

New MultimodalEmbeddingProvider interface enables cross-modal retrieval — text queries find matching images in the same vector space. provider/openaicompat.Embedding implements both text and multimodal embedding for OpenAI, vLLM, Ollama, and compatible APIs. ingest.WithImageEmbedding creates image chunks from extracted DOCX/PDF images. BlobStore interface stores large image bytes externally.

💾 Ingest Checkpoint & Resume

Pipeline persists progress after each stage (extracting → chunking → enriching → embedding → storing → graphing). ResumeIngest(ctx, checkpointID) picks up from the last completed stage. IngestBatch/ResumeBatch for multi-document ingestion with per-document tracking. WithBatchConcurrency(n) for parallel processing. Cross-document extraction also supports resume via ResumeCrossDocExtraction.

⚡ SQLite Production Performance

WAL mode + connection pool (4 conns, busy_timeout=5000) — readers no longer block writers. Batched INSERTs (3001→33 statements for 1000 chunks). Min-heap vector search O(N log K) with pre-computed norms. Binary embedding storage (~5x smaller I/O). In-memory vector index with lazy loading. Bounded index with FIFO eviction (WithMaxVecEntries). Batch search (SearchChunksBatch) for cross-document extraction.

🕸️ Graph-RAG Improvements

Bidirectional graph traversal (BidirectionalGraphStore). Document-aware extraction (WithGraphDocContext). Semantic batch parallelism (3x faster with worker pool). O(N·K) batch construction replacing O(N²). Edge descriptions with graph context on retrieval results. Improved directionality guidance for asymmetric relations. Temperature 0 for deterministic extraction.

🔗 OpenAI-Compatible Embedding Provider

provider/openaicompat.NewEmbedding — works with OpenAI, vLLM, Ollama, Together, Mistral. provider/resolve.EmbeddingProvider now resolves all OpenAI-compatible providers automatically.

📊 Retrieval Pipeline Fixes

RRF scores normalized to [0,1] — fixes WithMinRetrievalScore and ScoreReranker silently dropping all results. HybridRetriever runs vector + keyword search in parallel. LLMReranker handles markdown-wrapped JSON. Parent resolution uses deterministic sort-before-dedup. KnowledgeTool embeds query once instead of twice.

---

🆕 New Features

Feature	Description
MultimodalEmbeddingProvider	Cross-modal text + image embedding in the same vector space
MultimodalInput type	Holds text and/or Attachments for multimodal embedding
BlobStore interface	External binary object storage for large image data
provider/openaicompat.Embedding	OpenAI-compatible embedding provider (text + multimodal)
ingest.WithImageEmbedding	Image chunk creation from extracted DOCX/PDF images
ingest.WithBlobStore	External blob storage for image chunks
CosineSimilarity	Exported utility replacing 5 package-local duplicates
WithLLMTimeout	Max duration for individual LLM calls in ingest (default 2 min)
WithRerankerTimeout	Max duration for LLM reranking calls (default 2 min)
WithMaxVecEntries	Bounded SQLite vector index with FIFO eviction
DocumentMetaLister	Lightweight document listing without loading content
SearchChunksBatch	Multi-embedding single-pass vector search
CrossDocWithProgressFunc	Progress callback for cross-document extraction
BidirectionalGraphStore	Single-query bidirectional edge fetch (2→1 round-trips per hop)
WithGraphDocContext	Document-aware graph extraction with structural context
Ingest checkpoint & resume	Per-stage progress persistence with ResumeIngest
IngestBatch / ResumeBatch	Multi-document batch ingestion with per-document tracking
WithExtractRetries	Retry custom extractors with exponential backoff
ResumeCrossDocExtraction	Resumable cross-document edge extraction
Contextual enrichment	LLM-based chunk context prefixing (~35% retrieval improvement)
WithEmbeddingRetry	Retry wrapper for EmbeddingProvider with backoff + jitter
Structured logging	Comprehensive slog logging across agent, network, store, ingest

---

⚠️ Breaking Changes

Change	Migration
store/libsql removed	Use store/sqlite or store/postgres instead
WithCrossDocumentEdges removed	Use WithBatchCrossDocEdges instead
similar_to relation removed	Use references or other valid relation types

---

🔧 Changed

SQLite Performance

• Batched INSERTs — 3001→~33 SQL statements for 1000 chunks
• Min-heap vector search — O(N log K) with pre-computed L2 norms
• Binary embedding storage — ~5x smaller I/O per vector scan
• In-memory vector index — lazy-loaded, eliminates per-query deserialization
• WAL mode + connection pool — MaxOpenConns(4), concurrent read/write
• Bounded vector index — FIFO eviction with disk fallback
• StoreEdges upsert — ON CONFLICT DO UPDATE preserving original edge ID
• StoreDocument batched — multi-value INSERTs in batches of 100
• GetChunksByDocument — skips embedding blob when vec index loaded
• 50K entry warning — recommends Postgres with pgvector at scale

Graph-RAG

• Semantic batch parallelism — worker pool instead of sequential (3x faster)
• O(N·K) batch construction — centroid-based assignment replaces O(N²) greedy
• Temperature 0 — deterministic LLM output for graph extraction
• Improved directionality — better guidance for asymmetric relations
• Batched edge inserts — SQLite 150 rows/batch, Postgres unnest-based bulk

Retrieval

• RRF scores normalized to [0,1] — fixes WithMinRetrievalScore dropping results
• Parallel hybrid search — vector + keyword run concurrently
• LLMReranker handles markdown-wrapped JSON code fences
• Parent resolution — sort-before-dedup, skip redundant DB call
• CosineSimilarity consolidated — single exported function

Ingest

• enrichChunksWithContext safe mutation — no concurrent writes
• batchEmbed saves checkpoint per batch — no lost progress on partial failure
• buildSemanticBatches accepts logger — warns on dropped singleton batches
• buildSequenceEdges index-based sort — avoids 3 MB embedding allocation

Postgres

• hnsw.ef_search per-connection — AfterConnect hook for all pool connections
• StoreEdges — unnest-based bulk insert

---

🐛 Bug Fixes

RAG Pipeline

• Extractor retry respects context cancellation — select with ctx.Done() instead of time.Sleep
• Keyword search errors logged instead of silently swallowed via _
• DocumentChunkLister implemented on store/sqlite and store/postgres
• Parent resolution correctness — deterministic map iteration order
• FTS5 query sanitization — strips metacharacters (C++ no longer crashes FTS5)
• KnowledgeTool double embedding — query embedded once, reused for both searches

Ingest

• DocumentID on checkpoint — eliminates orphan data from repeated resume
• Cross-document extraction uses ListDocumentMeta to avoid loading full content
• Cross-document extraction uses batch search (1 index scan vs N per document)

---

📖 Full Changelog

v0.9.0...v0.10.0

---

go get github.com/nevindra/oasis@v0.10.0