Repository navigation
v3.3.2 — Graceful handling of oversized embedding chunks
Bug Fix
Embedding batch failures no longer cause total data loss.
Problem
When codetect index --force encountered a single chunk exceeding the Ollama model's context window (e.g. nomic-embed-text with 8192 tokens), the error cascaded through three layers:
OllamaClient.EmbedBatchWithContextreturnednil, error— discarding already-computed embeddingsPipeline.embedNewChunkspropagated the error — discarding the entire batchIndexer.processBatchreturned error — skipping the whole file batch and reportingchunks_created=0, chunks_embedded=0
Fix
Graceful degradation — skip individual failing chunks instead of failing entire batches:
OllamaClient.EmbedBatchWithContext: logs a warning and skips texts that fail, returningnilslots. Only errors if ALL texts fail or context is cancelledLiteLLMClient.Embed: falls back to individual retries when the batch API call fails, skipping individual failuresPipeline.embedNewChunks: filters outnilentries from partial results, storing only successful embeddingsPipeline.EmbedChunks: tracks skipped chunks inEmbedResult.Errors
Tests
TestEmbedChunksPartialFailure— 1/5 chunks fails → 4 embedded, 1 errorTestEmbedChunksAllFail— all chunks fail → error returnedTestEmbedChunksMixedBatches— failures spread across batches → successful chunks preserved