Feat/deterministic aggregation#28
Merged
Merged
Conversation
…ate management, various processing nodes, RBAC, and CLI execution.
…node responses - Deleted outdated audit documentation files: architecture_remediation.md, remediation_plan_observability.md, and remediation_plan.md. - Updated llm.yaml configuration to remove unused model settings. - Enhanced response schemas for various pipeline nodes (e.g., AnswerSynthesizerNode, ExecutorNode, and others) to include error handling and reasoning fields. - Introduced new nodes for answer synthesis and schema retrieval in the pipeline. - Improved overall structure and clarity of the codebase by consolidating and organizing node responses.
- Introduced a new pytest.ini file to specify test paths for core, adapters, and SQLAlchemy tests. - Added documentation for executor artifacts, detailing configuration options and usage. - Created test configuration files for SQLAlchemy integration and unit tests, improving test organization. - Removed outdated compliance tests for MSSQL, MySQL, Postgres, and SQLite adapters to streamline the test suite. - Implemented new test cases for artifact handling and execution layers, ensuring robust coverage of the new features.
- Added new configuration for indexing enrichment in llm.demo.yaml and llm.yaml. - Implemented SchemaEnrichment and related classes in enrichment_service.py to facilitate schema metadata enrichment. - Updated SchemaChunkBuilder to include column names in the generated schema chunks. - Enhanced the IndexingOrchestrator to utilize the new enrichment functionality during schema snapshot registration. - Introduced new methods for retrieving column candidates in vector_store.py to improve schema retrieval. - Added unit tests for the enrichment service and models to ensure functionality and correctness.
- Added new configuration options for schema store backend and path in configuration.md. - Updated VectorStore initialization to include collection name and improved error handling for missing settings. - Refactored NL2SQLContext to enforce required settings for vector store and schema store. - Introduced SQLite schema store implementation and in-memory store for better schema management. - Enhanced error handling in the AggregationService and other pipeline nodes to improve robustness. - Added integration tests for schema retrieval and validation processes.
…dling - Introduced cancellation functionality using threading events to allow users to cancel long-running operations. - Updated various pipeline nodes (e.g., ExecutorNode, SQLExecutorService) to handle cancellation gracefully and return appropriate error messages. - Enhanced error handling in the pipeline to include cancellation errors, improving user experience during execution. - Added new configuration options for SQL agent retries and delays to manage execution flow more effectively. - Refactored graph building and routing logic to integrate cancellation checks, ensuring responsiveness during execution. - Added pytest markers for end-to-end tests to facilitate better test organization and execution.
…eline - Enhanced artifact handling by introducing a unified method for creating artifact references across different storage backends (S3, ADLS, Local). - Refactored the execution contracts to replace the deprecated ExecutorBaseModel with ExecutorResponse, improving clarity and consistency. - Updated the ExecutorNode to include tenant_id in requests, facilitating multi-tenant support. - Improved error handling and logging across various pipeline nodes, ensuring better traceability and debugging capabilities. - Removed unused schema management methods and streamlined the datasource resolution process for improved performance.
- Removed deprecated Column and Table classes from the schema module, introducing ColumnRef for better reference management. - Updated the SchemaRetrieverNode to build tables from schema snapshots, incorporating relationships and metadata. - Enhanced the LogicalValidatorNode to enforce join relationships and validate against column statistics. - Improved error handling and logging in various pipeline nodes for better traceability. - Refactored imports across the codebase to utilize the new schema structure, ensuring consistency and clarity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request contains a mix of documentation updates, configuration changes, and terminology clarifications. The most significant changes improve the clarity and accuracy of adapter development documentation, update the LLM agent configuration, and introduce new documentation for executor artifacts and schema store configuration. Below are the most important changes grouped by theme.
Adapter Protocol & Documentation Updates:
DatasourceAdapterProtocolinstead of the previous "SDK" orDatasourceAdapter, with updated examples and compliance instructions indocs/adapters/development.mdanddocs/adapters/sdk.md[1] [2] [3] [4].DatasourceAdapterProtocoland related contracts [1] [2].LLM Agent Configuration:
intent_validatortoindexing_enrichmentand updated its model togpt-5.2in bothconfigs/llm.yamlandconfigs/llm.demo.yaml[1] [2].Executor Artifacts & Storage Configuration:
docs/core/artifacts.md.SCHEMA_STORE_BACKEND,SCHEMA_STORE_PATH) indocs/ops/configuration.md.Terminology & Service Renaming:
OrchestratorVectorStoretoVectorStorein documentation and code references for consistency [1] [2] [3].Cleanup & Audit Documentation:
Let me know if you want to discuss any of these changes in more detail!