v1.5.0.dev5 — Search Relevance & Ingestion Speed
Pre-release
Pre-release
·
3 commits
to main
since this release
v1.5.0.dev5 — Search Relevance & Ingestion Speed
Release Date: 2026-08-20
Changes: v1.5.0.dev5 → dev
Summary
This release focuses on making search results more relevant and ingestion of new documents faster and more reliable. It adds a new optional dataset overview index to give searches broader context, improves how documents are split and stored, and fixes several bugs affecting search scoring and metadata handling.
Highlights
- New optional dataset overview index that groups documents by topic and creates short overviews to improve search context.
- Faster and more reliable ingestion for large files and bulk uploads.
- Improved search ranking so returned memories (past documents and notes) are more relevant to your queries.
Breaking Changes
- None: there are no breaking API changes in this release. Existing integrations should continue to work. If you enable the new dataset overview index, search results and scoring may change because the system now uses the new overview context.
New Features
- Optional dataset overview index: A new type of index that automatically groups a dataset (a collection of documents you upload) into topic clusters and generates a short overview for each cluster. What it does: gives the search engine broader context about each dataset so queries return more relevant, coherent answers. Why it matters: you get better, less fragmented search results across large or diverse datasets without extra manual tagging.
- Improved chunking strategy for document ingestion: The ingestion pipeline now splits long documents into chunks more intelligently, preserving sentence and paragraph boundaries where possible. What it does: reduces broken sentences and duplicated context in stored chunks. Why it matters: queries that match content across chunk boundaries are less likely to miss relevant information, improving both recall and answer quality.
- Bulk upload progress and resumable ingestion: When uploading many files or very large files, the system now shows clearer progress and can resume interrupted ingestions. What it does: prevents re-uploading from scratch after network interruptions. Why it matters: saves time and bandwidth during large imports.
Improvements
- Search ranking updates: The search algorithm now weighs dataset overview context higher for broader queries and exact match content higher for specific queries. This balances recall (finding related content) and precision (showing exact matches).
- Metadata preservation: Ingestion now preserves more original metadata fields (timestamps, source filenames, and custom tags) so you can filter and sort results using the same metadata you uploaded.
- Cleaner default dataset naming: New datasets get clearer default names and descriptions based on file contents to make datasets easier to navigate for first-time users.
- Developer SDK: Added convenience helpers for creating and enabling the dataset overview index from application code, plus clearer error messages when configuration is invalid.
Performance
- Up to 2x faster ingestion of large files due to optimized chunking and parallel upload handling. Practically, large PDF and text imports finish noticeably sooner.
- Lower query latency for dataset-scoped searches by caching dataset overview summaries for short intervals, resulting in snappier search results when exploring a single dataset repeatedly.
Security
- Stronger upload validation to reject malformed files earlier in the pipeline, reducing the chance of ingestion-time failures. This improves robustness when handling user-supplied files.
- Tighter token validation on API endpoints to ensure only properly-scoped requests can trigger ingestion or index changes.
Bug Fixes
- Fixed incorrect search scoring where some documents were over-ranked due to duplicated chunk weights.
- Resolved an issue where uploaded files sometimes lost source filename metadata.
- Fixed a race condition that could cause dataset deletion to fail when an ingestion was in progress.
- Corrected edge-case where the dataset overview index could produce empty clusters for very small datasets.
- Fixed an SDK error that surfaced as an obscure stack trace rather than a helpful validation message.
- Addressed a UI bug that showed progress as 0% for very short-but-batched uploads.
Technical Changes
- Refactored the ingestion pipeline to separate chunking, metadata extraction, and storage steps which simplifies future enhancements.
- Updated internal dependencies and added new unit and integration tests around ingestion resiliency and dataset overview generation.
- Improved logging around ingestion and indexing for faster debugging of customer issues.
Compatibility
| Component | Supported / Required |
|---|---|
| Python | >=3.10,<3.15 |
| pydantic | >=2.10.5 |
| litellm | >=1.83.7 |
| fastapi | >=0.116.2,<1.0.0 |
| sqlalchemy | >=2.0.39,<3.0.0 |
| lancedb | >=0.24.3,<1.0.0 |
| ladybug | ==0.19.0 |
— The Cognee Team · 2026-08-20