Skip to content

v1.5.0: Frontmatter Strip + Metadata Storage

Choose a tag to compare

@lidge-jun lidge-jun released this 15 Feb 18:10
· 16 commits to main since this release

What's New

  • Frontmatter strip + metadata — YAML frontmatter is stripped before embedding to reduce noise; tags and aliases are parsed and stored as Milvus metadata fields
  • New chunking.py module — strip_frontmatter(), _normalize_meta(), inject_header_prefix(), merge_small_chunks()
  • CRLF-safe regex — handles both \n and \r\n line endings
  • Type safety — _normalize_meta() handles None, bare string, and list YAML values
  • Rate limit fix — EMBEDDING_CONCURRENT_BATCHES default reduced from 3 to 2

Full Changelog

v1.4.0...v1.5.0