Skip to content

v0.3.2 - Performance Optimization & Improved Compatibility

Choose a tag to compare

View all tags
@maholick maholick released this 14 Sep 20:50
· 24 commits to main since this release
v0.3.2
e93e766

🚀 Release v0.3.2

✨ New Features

Optimized Payload Structure

  • Configurable content fields - Choose which fields to include (content, page_content, document, text)
  • Flattened metadata - Improved filtering performance with non-nested structure
  • Preview field - Configurable preview snippets (150-300 chars recommended)
  • Quality scoring - Rank chunks by information density, optimal length, and keyword richness (0-1 scale)
  • Token counting - Track token count for LLM context window management
  • Source type detection - Automatically classify files (pdf, markdown, code, config, etc.)

Embedding Cache System

  • LRU cache with 500 entry limit to avoid redundant API calls
  • 20-30% faster processing for content with duplicates
  • Cache statistics - Track hits, misses, and hit rate
  • Multi-repository support - Cache persists across repository processing

Improved Multi-Repository Processing

  • Cache statistics displayed per repository and cumulative totals
  • Better visibility into processing efficiency
  • Shared cache across all repositories in a batch

🛠️ Improvements

Payload Optimization

  • ~40% size reduction when using minimal mode (2 fields instead of 4)
  • Full compatibility with n8n, LangChain, MCP, and other systems
  • Minimal mode option - Further reduce payload size when needed

CI/CD Optimization

  • 10x faster CI/CD - Reduced from 2-3 minutes to ~30 seconds
  • Only install essential dependencies for linting (pylint, ruff, pyyaml, python-dotenv)
  • Skip heavy dependencies (langchain, qdrant-client, numpy, mistralai)

🐛 Bug Fixes

  • Fixed code formatting to pass ruff checks
  • Removed unused imports
  • Fixed CI/CD workflow dependency issues

📚 Documentation

  • Comprehensive README updates with all v0.3.2 features
  • Added changelog section to README
  • Updated payload structure documentation
  • Added configuration examples for new features
  • Documented embedding cache statistics

📊 Performance Improvements

  • Embedding cache: 20-30% faster for duplicate content
  • Payload size: 40-60% reduction with configurable fields
  • CI/CD pipeline: 10x faster execution
  • Filtering: Improved with flattened metadata structure

🔧 Configuration

New payload configuration section in config.yaml:

payload:
  content_fields:
    - content      # For n8n
    - page_content # For LangChain
    - document     # For MCP
    - text         # Alternative field
  preview_length: 200
  minimal_mode: false  # Set to true for 50% size reduction

📦 Dependencies

  • Optimized CI/CD to use minimal dependencies
  • No changes to production dependencies

🙏 Acknowledgments

Thanks to all users for feedback and suggestions that led to these optimizations!

Full Changelog: v0.3.1...v0.3.2

Assets 2
Loading