Skip to content

v0.3.3 - Individual File Processing & Deduplication

Choose a tag to compare

View all tags
@maholick maholick released this 16 Sep 10:52
· 18 commits to main since this release
v0.3.3
9910a96

🚀 What's New

This release introduces individual file processing for better search quality and fixes critical issues with duplicate vector creation.

✨ Features

  • Individual File Processing: Files are now processed separately when combine_documents: false, maintaining document boundaries for improved search relevance
  • Deterministic ID Generation: Vector IDs are now generated deterministically based on file path and content hash, preventing duplicates across runs
  • Clean Interrupt Handling: Improved Ctrl+C handling with clean exit messages instead of verbose Python tracebacks

🐛 Bug Fixes

  • Fixed Duplicate Vector Creation: Resolved issue where each run would add exactly 1 duplicate vector to the collection
  • Fixed combine_documents Configuration: The process_repository method now properly respects the combine_documents: false setting
  • Type Safety Improvements: Fixed type annotations throughout the codebase for better IDE support and error detection
  • CI/CD Linting: Removed unused variables that were causing linting failures

📊 Performance Improvements

  • Deterministic processing ensures consistent results across runs
  • Individual file processing provides better context preservation
  • Improved search quality with proper document boundaries

🔧 Technical Changes

  • Modified _generate_chunk_id to use file path + content hash instead of sequential indices
  • Updated process_repository to use _process_repository_internal which checks configuration settings
  • Added _process_files_individually method for file-by-file processing
  • Enhanced EmbeddingCache with proper get/set methods

📝 Configuration

To enable individual file processing (recommended for better search quality):

processing:
  combine_documents: false  # Process files individually
  per_file_chunking: true   # Better context preservation

Commits Since v0.3.2

  • fix: Remove unused variable to pass CI/CD linting
  • docs: Update README with v0.3.3 features and changelog
  • fix: Process files individually when combine_documents is false
  • fix: Add debug logging and improve combine_documents check
  • feat: Add individual file processing for better vector management
  • fix: Resolve duplicate vector creation and improve interrupt handling

Full Changelog: v0.3.2...v0.3.3

Assets 2
Loading