Skip to content

v1.6.0: non-blocking indexing with to_thread + 3-way delta

Choose a tag to compare

View all tags
@lidge-jun lidge-jun released this 16 Feb 15:22
· 13 commits to main since this release
v1.6.0
0dbbdc3

What's New

Non-blocking Event Loop (Phase 13)

  • All sync I/O operations (file walk, Milvus ops, chunking, inserts) now run in worker threads via asyncio.to_thread
  • Event loop stays fully responsive during indexing — get_index_status and search_documents respond instantly
  • await asyncio.sleep(0) ensures MCP response flushes before heavy work begins

3-Way Delta Scan

  • New get_index_delta_detailed() classifies files as new, modified, or deleted in a single walk
  • New files skip Milvus delete entirely (no vectors exist yet) — eliminates wasted gRPC round-trips
  • On 1289 new files, this saves 1289 unnecessary delete calls

Completion Metrics

  • Incremental jobs now return delta_counts (new/modified/deleted) and timings_seconds per phase
  • ~40 extra tokens in the result for performance analysis
  • Omitted for force_reindex=true (full rebuild)

README Updates

  • New architecture diagram labels, expanded Background Indexing section
  • Two new Optimization Technique sections (Thread Offloading, 3-Way Delta)
  • Updated flowcharts with NEW/MODIFIED/DELETED labels
Assets 2
Loading