Conversation
- Add y.md file pattern to .gitignore to prevent accidental commits of test/workspace related markdown files
- Add vectorless-primitives crate to workspace and Python bindings - Implement DocumentNavigator with full navigation capabilities (ls, cd, cd_by_title, cd_up, cd_root, pwd, cat, grep, find, head, wc) - Add metadata access methods (summary, section_count, doc_id, doc_name) - Include reasoning index functionality (keyword_entries, topic_summary, related_sections) - Provide evidence collection and management (collect_evidence, evidence, clear_evidence) - Add tree inspection methods (root_id, current_id, children_of, parent_of, depth_of, node_title, all_node_ids) - Introduce helper types (PyNodeInfo, PyMatchResult, PyFindResult, PyWordCount, PyCollectedEvidence, PyTopicEntry, PySectionSummary) - Enable async operations with proper error handling using VectorlessError - Update Cargo.toml dependencies to include sync feature for tokio - Register new Python classes in module initialization
- Add internal types module with TraceStep, WorkerEvidence, and WorkerResult - Implement orchestrator agent module for coordinating multi-document retrieval - Add evaluation module for assessing evidence sufficiency via LLM - Create orchestrator class that follows 3-phase process: analyze, supervise, return - Include DocCard for lightweight document metadata handling - Add supervisor loop with dispatch → evaluate → replan cycle - Implement confidence computation based on evaluation outcomes - Add cross-document keyword search functionality - Support parallel worker dispatch and result collection
- Add query module with understanding and planning capabilities including QueryIntent, Complexity enums and QueryPlan dataclass - Implement LLM-driven query analysis with keyword extraction and intent classification - Create rerank module with evidence deduplication using Jaccard similarity and intent-aware answer formatting - Add proper __all__ exports for new modules and components - Include comprehensive type definitions and documentation
…gy layer BREAKING CHANGE: The main API class has been renamed from `Session` to `Engine`. The Python wrapper now uses a dual-layer architecture where compilation (ingest) runs in Rust and querying (retrieval) runs in Python strategy layer with query understanding, orchestrator, and reranking capabilities. - Rename Session class to Engine and move from session.py to engine.py - Update all imports and references from Session to Engine across codebase - Replace ingest method with index method in public API - Add Python strategy layer for query processing with understanding → orchestrator → workers → rerank pipeline - Update CLI commands to use Engine instead of Session - Remove sync_session.py module - Update compatibility layers for langchain and llamaindex to use Engine - Modify streaming to work with Python strategy pipeline instead of Rust streaming
- Update module docstring to clearly distinguish between public and internal APIs - Add clarification that vectorless._core module is not part of public API - Document that public Engine interface is at vectorless.engine.Engine - Explain that internal Engine refers to raw Rust engine for compile/document management
- Replace IndexContext usage with direct calls to Rust engine's ingest method - Update single file ingestion to use rust.ingest() instead of index context - Add proper handling for content and bytes_data through temporary files - Update directory scanning to filter supported file extensions (.md, .pdf, .markdown) - Remove unused IndexContext and IndexOptions imports feat(engine): update indexing methods and result wrappers - Modify index_batch to return single IndexResultWrapper instead of list - Add from_doc_info and from_doc_infos class methods to IndexResultWrapper - Update list_documents and remove_document to use new Rust method names - Change remove_document to call rust.forget and return True consistently docs(cli): update REPL command documentation - Correct parameter documentation from 'session: Session instance' to 'session: Engine instance' - Update CLI stats output from 'Session statistics' to 'Engine statistics' refactor(engine): simplify query document resolution logic - Remove commented out emit events in query method - Move document listing before target resolution in python strategy - Use list_documents instead of list method for consistency - Remove redundant document info fetching in orchestrator flow
- Export new types including CollectedEvidence, DocumentGraphNode, FindResult, LlmMetricsReport, MatchResult, NodeInfo, RetrievalMetricsReport, SectionSummary, TopicEntry, and WordCount - Update __all__ exports to include newly available types - Remove deprecated DocumentGraphEdge reference - Organize exported types into logical groups (document, graph, metrics)
- Replace OpenAI SDK with litellm for multi-provider support including OpenAI, Anthropic, DeepSeek and 100+ other providers - Add instructor library for structured output validation with Pydantic - Implement automatic retry mechanism with feedback on validation failure - Add per-request timeout configuration - Introduce in-memory response caching for improved performance - Support both regular completion and structured JSON output - Add comprehensive error handling for rate limits and timeouts - Update dependencies in pyproject.toml with litellm>=1.50 and instructor>=1.0
- Add event_callback parameter to Orchestrator for real-time progress notifications during worker execution - Emit events for worker_started, worker_error, and worker_done states - Introduce event_queue support in Engine for streaming consumers - Add comprehensive event emission throughout the pipeline: understanding_started, understanding_done, orchestrator_started, orchestrator_done, rerank_started, rerank_done - Implement DocumentNotFoundError and EmptyWorkspaceError exceptions - Replace polling mechanism with asyncio.Queue in StreamingQueryResult - Update streaming module to use producer-consumer pattern with sentinels - Add proper type hints including Callable for event callbacks
- Remove vectorless-agent dependency that was no longer needed - Remove vectorless-retrieval dependency that was no longer used These dependencies were not being utilized in the vectorless-engine crate and have been removed to clean up the dependency list.
- Comment out vectorless-query, vectorless-agent, and vectorless-retrieval crates from Cargo.toml as they are being moved to Python implementation - Keep the crates in the repository but exclude them from compilation - Update build configuration to reflect new architecture decision
- Remove RetrieverClient from EngineBuilder as retrieval is now handled by Python strategy layer - Remove retriever field from Engine struct and update with_components constructor - Replace ask() method implementation with error that directs users to Python SDK - Remove output_to_answer conversion function - Update retriever.rs to stub module that returns errors for query attempts - Redefine query types locally since strategy layer moved to Python
Remove unused RetrieverClient import from engine module to clean up code and reduce unnecessary dependencies.
Add a new quality filtering module that evaluates evidence relevance using LLM scoring. The filter assesses whether each evidence item actually answers the question and removes low-quality or irrelevant evidence before synthesis. Skip LLM evaluation for small evidence sets (≤3) to maintain performance, while batch-evaluating relevance for larger sets. The filter ensures minimum evidence count is maintained and logs filtering statistics. Also export the new filter_by_quality function in the module's __all__ list.
BREAKING CHANGE: The agent module has been renamed to ask and moved to a new location. All imports need to be updated from vectorless.agent to vectorless.ask. The evaluation module now provides structured evaluation with coverage, quality scores, and specific missing aspects instead of simple sufficiency checking. feat(ask): add comprehensive evidence evaluation with coverage metrics The new evaluation system returns structured results including: - Coverage score (0.0-1.0) indicating how much of the query is addressed - Quality score (0.0-1.0) measuring evidence relevance - Specific missing aspects requiring additional research - Relevant evidence IDs for focused processing refactor(imports): update all module references from agent to ask Update imports across the codebase to reflect the module rename from agent to ask, including engine.py, rerank modules, and configuration files.
- Add navigation history stack to track previous positions for back() functionality - Implement back() method to navigate to previous positions in history - Add comprehensive toolset including: - toc(): retrieve full table of contents as flat list - stats(): get detailed statistics about nodes - grep_node(): search within specific node content without cursor movement - similar(): find semantically similar nodes using reasoning index - section_overview(): get pre-computed section overviews - Add new data types TocEntry, NodeStats, and SimilarResult for enhanced functionality - Expose all new methods and types to Python bindings with proper async support
- Add 'back' command to return to previously visited position - Add 'toc [depth]' command to show table of contents with optional depth filter - Add 'stats <name>' command to display node statistics (size, depth, children) - Add 'grep_node <node> <pattern>' for regex search within specific node - Add 'similar <name>' to find nodes by keyword overlap - Add 'overview <name>' to show pre-computed section summaries - Add 'compare <a> <b>' to analyze two nodes using LLM - Add 'trace <name>' to follow reasoning chains with LLM assistance - Add 'summarize <name>' to generate dynamic LLM summaries - Implement new tools module with compare, trace, and summarize functions - Update command parsing and execution logic to handle new commands - Enhance worker state management for advanced navigation features
…etter organization - Move inspection tools (toc, stats, siblings, ancestors, doc_card, concepts, find_section) to navigator_inspection.rs - Move search tools (grep, find, grep_node, similar, section_overview, keyword_entries, topic_summary, related_sections) to navigator_search.rs - Add file structure documentation in main navigator.rs - Include new files via include! macro in navigator.rs feat(document): add new inspection and search methods to Python bindings - Add siblings(), ancestors(), doc_card(), concepts(), and find_section() methods - Implement PySectionCard, PyDocCard, and PyConceptInfo wrapper classes - Register new types in module initialization docs(worker): update command documentation with new inspection tools - Add siblings, ancestors, doc_card, concepts, and find_section commands to available commands list - Update worker dispatch to handle new command types - Implement execution logic for new inspection commands refactor(core): move internal modules to separate directory - Rename _async_utils.py to _internal/_async_utils.py - Rename _core.py to _internal/_core.py - Update import paths in langchain and llamaindex compatibility layers - Add __init__.py to internal module feat(worker): implement new document inspection commands - Add parsing support for siblings, ancestors, doc_card, concepts, and find_section commands - Implement execution handlers for new inspection methods - Update worker feedback messages for new commands
BREAKING CHANGE: Removes the following modules that were previously available: - tests/conftest.py: Shared test fixtures - tests/test_config.py: Configuration models and loading tests - tests/test_events.py: Event system tests - tests/test_session.py: Session high-level API tests - tests/test_types.py: Typed result wrappers tests - vectorless/_compat/: Framework integrations for LangChain and LlamaIndex including langchain.py and llamaindex.py integration modules This change removes unused test code and optional framework integration modules that were not being actively maintained.
… implementation - Remove Crates.io badges from README as they are no longer maintained - Add comprehensive type definitions in types.py that mirror the Rust vectorless-agent implementation - Replace old WorkerEvidence and WorkerResult with new Evidence-based types - Introduce new core types: Evidence, WorkerOutput, Output, Metrics, and OrchestratorState - Update imports in __init__.py to expose new types and reorganize module structure - Add detailed docstrings explaining the type hierarchy and alignment with Rust counterparts
Add comprehensive evidence sufficiency evaluation system with two distinct modes: - Cross-doc evaluation (Orchestrator level): simple SUFFICIENT/INSUFFICIENT text parsing that mirrors Rust implementation from vectorless-agent - Structured evaluation (Worker level): detailed JSON output with coverage, quality scores and missing aspects Update Evidence type usage to align with shared types.py definition and modify relevant_ids to use node_titles instead of node_ids for consistency. Include robust parsing functions with fallback mechanisms and maintain feature parity with existing Rust orchestrator implementation.
- Replace WorkerEvidence with Evidence type throughout quality and synthesize modules - Update deduplication key from (source_path, title) to (doc_name, source_path) - Change node title field from 'title' to 'node_title' in evidence formatting - Add doc_name attribution in evidence formatting output - Update QueryIntent mappings: replace COMPARATIVE/PROCEDURAL/EXPLORATORY with NAVIGATIONAL/SUMMARY intents - Add mirroring comments to indicate alignment with Rust implementation - Include llm_calls field in RerankOutput with default value of 0
- Add WorkspaceClient for async CRUD operations on documents with thread-safe design using Arc internally - Implement async workspace with LRU caching, metadata indexing, and file backend support - Add document graph persistence and cross-document relationship tracking - Introduce unified dispatch function for query processing pipeline - Simplify README with concise project description and installation instructions - Clean up gitignore by removing test workspace entries
BREAKING CHANGE: The index() method has been renamed to compile() across the entire codebase including CLI commands, documentation, and examples. - Rename engine.index() to engine.compile() - Rename index_batch() to compile_batch() - Update CLI 'add' command documentation to reflect compilation - Update README.md examples and documentation - Update docstrings and comments to use "compile" terminology
- Update documentation URLs in README.md to point to correct paths - Replace "index" terminology with "compile" throughout documentation - Rename method names from "index/query/remove/list" to "compile/ask/remove_document/list_documents" - Change Engine.query() to Engine.ask() in architecture documentation - Update example code snippets to reflect new API naming conventions - Replace "Batch Indexing" with "Batch Compiling" in examples - Update "Multi-Document Retrieval" to reflect compilation terminology - Change "Quick Query" to demonstrate compile-and-ask workflow - Remove outdated feature documentation files that are no longer relevant - Restructure getting started guide with clearer introduction - Add installation documentation page - Update score terminology to confidence in multi-document examples
- Add CSS classes for code syntax highlighting (.kw, .fn, .str, .cmt) - Replace principles card with code preview card showing quick start example - Update homepage styling with new layout, buttons, and responsive design - Add code syntax highlighting to index module CSS - Replace feature pills with tagline and update hero content structure - Implement code card component with syntax highlighted example refactor(docs): replace GitHub stats with GitHub star button in navbar - Remove GitHubStats component from homepage - Add GitHubStar component to navbar - Center navbar items and adjust spacing - Update navbar styling for better alignment and positioning
- Format long function signatures with proper line breaks and indentation - Simplify nested code blocks and remove unnecessary wrapping - Update placeholder message with proper line breaks style: adjust code formatting and improve readability - Reformat multi-line expressions for better readability - Remove unnecessary parentheses and line breaks - Clean up assertion statements and chained method calls feat: update API response structure and variable names - Change `engine.ask()` response attribute access from `.single().content` to `.answer` - Replace individual metric accumulation with consolidated approach - Update CLI commands to use new response structure refactor: streamline worker state management and data structures - Consolidate worker state into shared WorkerState class - Remove redundant local state tracking in favor of centralized metrics - Simplify evidence collection and metric accumulation logic refactor: update CLI interface and remove deprecated options - Remove unused workspace scope option from CLI commands - Simplify query command argument handling - Update CLI output formatting to match new response structure refactor: modernize data structures and remove legacy conversion code - Replace old result types with new unified Output structure - Remove obsolete conversion helpers between different result formats - Update Jupyter display components to use new result structure
- Change `answer.content` to `answer.answer` for consistency with the actual attribute name - Update both the conditional check and the content splitting logic
Refect compile ask
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Checklist
cargo build)cargo test --lib --all-features)cargo clippy --all-features)Notes