Skip to content

v2.1.0 — Embedding Stability: Worker Isolation + File Size Limit

Choose a tag to compare

@CSCSoftware CSCSoftware released this 19 May 14:13
· 37 commits to master since this release

What's New

This release permanently fixes two root causes that crashed the embedding pipeline in production.

🛡️ Worker Process Isolation

Problem: Projects with large generated files (Blazor WASM bundles, large C++ TUs, minified JS) caused ONNX to request 67–93 GB of RAM. Because the model ran inside the MCP server process, the crash killed the server — disconnecting Claude and requiring a manual /mcp reconnect.

Fix: aidex_init({ embeddings: true }) now spawns a dedicated child process for each embedding run (src/embeddings/embed-worker.ts). Communication is via stdin/stdout JSON. When ONNX OOMs, only the worker dies — the MCP server survives and reports a Warning in the result instead of crashing. A 10-minute timeout kills runaway workers automatically.

🔒 25 KB File Size Limit

Problem: Even with worker isolation, files larger than ~50 KB reliably triggered STATUS_ACCESS_VIOLATION (exit code 3228369023) inside the worker. Root cause: the jina-code model's attention layers have O(n²) memory complexity — a 63 KB file with deeply nested methods exceeds the model's fixed-size buffers.

Fix: Methods, types, and doc-sections from files larger than 25 KB are silently skipped during embedding (src/embeddings/pipeline.ts). File sizes are resolved once via statSync and cached per run. Applies to full re-index and incremental updateFile() paths.

✅ Verified

6-worker parallel stress test across all 252 registered projects completed in 33 minutes with 0 crashes and 0 errors — including the previously fatal projects (YouTubeVoiceOver, CiscoWebExTranslator/cpp, UcHome with 767 source files).

Full Changelog

See CHANGELOG.md for the complete details.