Skip to content

v0.4.0

Choose a tag to compare

View all tags
@maholick maholick released this 17 Dec 04:37
· 6 commits to main since this release
v0.4.0
44e61cc

What's New in v0.4.0

This release focuses on retrieval quality, safe incremental indexing, and performance for large collections and shared multi-repo setups.

🎯 Key Features

Robust Incremental Sync (Dedup-safe)

  • ✅ Safe skipping of unchanged files when processing.track_file_changes: true
    • Uses SHA-256 file_hash and deterministic scoping identifiers
      • repo_id: SHA-256 of repo_url@branch
      • file_id: SHA-256 of repo_id:file_path
      • file_upload_id: SHA-256 of file_id:file_hash
  • ✅ Dedup-safe completeness tracking
    • Writes a lightweight per-file marker only after successful upload
    • Prevents false reprocessing when deduplication reduces stored chunk counts

Why this matters: older chunk-count based “completeness” checks can mis-detect unchanged files as incomplete when deduplication reduces the number of stored chunks, causing unnecessary re-embeds on every run. v0.4.0 avoids that by relying on a post-upload marker.

Multi-repo Shared-Collection Safety

  • ✅ Safe deletes and updates in shared collections
    • Deletes scoped to repo_id + file_id to avoid cross-repo collisions
    • Includes legacy fallback handling for older collections

Qdrant Payload Indexes (Optional)

  • 🆕 Configurable payload index creation for faster filtered queries and grouping
    • Works with both payload layouts:
      • payload.metadata_structure: nested → index paths like metadata.repository
      • payload.metadata_structure: flat → index paths like repository

Token-aware Chunking (Optional)

  • 🆕 token_recursive chunking strategy using tiktoken
    • More stable chunk sizes across models
    • Configurable via chunk_size_tokens, chunk_overlap_tokens, tiktoken_encoding

Retrieval CLI

  • 🆕 Added rag_retrieval.py
    • Queries Qdrant with configurable filters
    • Supports grouping/capping results per file for better context diversity

🔧 Bug Fixes

  • Fixed cases where unchanged files were reprocessed due to dedup-related chunk-count mismatches
  • Improved safety of deletes in shared collections (multi-repo/branch)

📦 Configuration Changes

Enable incremental sync (recommended):

processing:
  track_file_changes: true

Enable payload indexes (recommended for large collections):

qdrant:
  payload_indexes:
    enabled: true
    apply_to_existing_collections: true
    fields:
      - name: repo_id
        type: keyword
      - name: file_id
        type: keyword
      - name: file_upload_id
        type: keyword
      - name: file_hash
        type: keyword
      - name: repository
        type: keyword
      - name: file_path
        type: keyword

Enable token-aware chunking:

processing:
  chunking_strategy: token_recursive
  chunk_size_tokens: 512
  chunk_overlap_tokens: 64
  tiktoken_encoding: cl100k_base

▶️ Usage Examples

Ingest:

python github_to_qdrant.py config.yaml

Query (CLI):

python rag_retrieval.py --config config.yaml --query "how do we configure incremental sync?"

📄 Data / Metadata Notes

  • Multi-repo safety fields: repo_id, file_id, file_upload_id, file_hash
  • File marker mechanism:
    • A per-file marker point is written only after a successful upload for a given file_upload_id and chunking configuration.
    • This makes incremental sync robust across retries and deduplication.

📚 Documentation

  • Updated README.md with new features and expanded changelog
  • Updated config.yaml.example with new configuration options

Full Changelog: v0.3.4...v0.4.0

Assets 2
Loading