Merged
Conversation
Replaces std::sync::{Mutex, RwLock} with parking_lot equivalents across kalamdb-api, kalamdb-core, and kalamdb-system for improved performance and ergonomics. Updates Cargo.toml dependencies and refactors lock usage to remove .unwrap() calls. Also adds a wait_for_sql_output_contains helper and improves CLI subscription tests for reliability.
Removed object_store, tonic, num_cpus, and reqwest dependencies from kalamdb-core, moving storage logic to kalamdb-filestore. Added ShardId type to kalamdb-commons and refactored manifest and schema registry code to use new filestore modules. Split and reorganized kalamdb-filestore into core, manifest, parquet, paths, and registry submodules, removing legacy files and updating references throughout backend. Updated tests and documentation to reflect new structure and improved type safety.
Centralizes table registration and provider management in SchemaRegistry, removing redundant logic from DdlExecutor and related helpers. Refactors ManifestService to use ManifestTableProvider and removes moka hot cache in favor of direct RocksDB-backed caching. Updates method signatures and usage across the codebase to reflect these changes, and improves error handling and test coverage for schema and manifest operations.
Replaces custom key encoding logic with the storekey crate to ensure lexicographic ordering of storage keys, fixing ordering bugs for variable-length string keys. Updates StorageKey trait and all usages to use storekey's encode_key and decode_key helpers. Removes legacy key_encoding, sharding, and traits modules from kalamdb-store. Updates tests and documentation to reflect the new encoding approach.
…into feature/cluster
Adds flow control and buffering for live query subscriptions to ensure notifications are not delivered before the initial data load completes. Updates the API and core logic to flush buffered notifications after the initial load, preventing race conditions and duplicate delivery. Also includes fixes for segment deduplication in Manifest, expands Service role table alter permissions, and exposes Raft peer wait configuration in cluster config.
…into feature/cluster
Manifest cache operations now handle corrupted entries robustly, using direct backend scans and improved error handling. Added support for DROP STORAGE IF EXISTS in SQL parser and executor, including tests. SchemaRegistry cache hit/miss tracking and metrics reporting are implemented. CLI and test code updated for new manifest and system table behaviors. Miscellaneous improvements to serialization and error handling throughout manifest and storage modules.
Introduces ManifestId as a type-safe wrapper for manifest cache identifiers, replacing ManifestCacheKey and moving it to its own module. Refactors manifest and column stats serialization to use StoredScalarValue for min/max, enabling zero-copy bincode for RocksDB and proper JSON output. Updates all related code to use ManifestId and SeqId, removes custom serde logic, and improves type safety and query planning. Also reorganizes flush job modules under manifest, and removes unused storage and PK modules.
Eliminates the flush notification feature and related code from live query and flush job logic. Refactors index and table store code to use the Partition type instead of raw strings for partition names, improving type safety and consistency. Cleans up redundant logic and builder patterns for LiveQueryManager, and updates related tests and helpers.
Consolidated duplicated ScalarValue conversion functions into a new kalamdb-commons/src/conversions module, including scalar-to-bytes, numeric, string, and Arrow/JSON conversions. Updated all call sites to use the centralized functions, removed redundant code from table providers, and added comprehensive unit tests. This improves maintainability and ensures a single source of truth for data type conversions across the codebase.
Migrates key encoding for IDs and indexes to use storekey tuple encoding for improved prefix scanning and collision avoidance. Updates all relevant ID types, index definitions, and provider logic to use encode_key/encode_prefix, replacing legacy delimiter-based formats. Adds new prefix methods for efficient range scans and updates tests to reflect the new encoding scheme.
Replaces raw byte key scans with typed key/value scans across entity and indexed stores, and updates all system providers and tests to use the new typed scan methods. Removes the old stats provider from schema_registry and migrates stats view logic to the views module. This improves type safety, reduces key parsing errors, and simplifies code in table providers and flush jobs.
Introduces common traits (SystemTableScan, SimpleSystemTableScan) in a new base.rs module to unify and deduplicate scan logic across system table providers. Refactors all system table providers to use these traits, reducing boilerplate and improving maintainability. Also optimizes stream table user scan with a streaming, early-termination approach for better performance with LIMIT queries.
Adds refresh token support to authentication handlers, including refresh token issuance and expiration tracking. Refactors job creation and execution logic to support leader-only jobs and instant job dispatch via an awake channel, improving job processing responsiveness and correctness in distributed environments. Removes obsolete test output files.
Introduces detailed info-level debug logging to job execution and Parquet scan paths for improved traceability. Adds skill definition files for DataFusion, MVCC, Raft, and Rust under .github/skills to guide automated agents. Enhances test_update_all_types.rs with more robust flush job handling and diagnostics. Also includes a server_log.txt for reference.
Refactored manifest user ID retrieval to use storekey-encoded prefixes and decode ManifestId from storage keys for more robust RocksDB scans. Enhanced debug logging in manifest service and parquet scan utilities, and standardized log levels from info to debug for better traceability during development.
Introduces next_storage_key_bytes utility for safe key advancement and replaces manual key manipulation with this function across the codebase. Refactors ManifestService to use new EntityStore batch and scan methods for cache operations, improving type safety and efficiency. Adds EntityStore methods for counting, compaction, batch deletion, and partition management. Updates related imports and documentation. Adds architecture documentation for the job system.
…into feature/cluster
Introduces `scan_keys_with_raw_prefix` to EntityStore for efficient key-only scans using a raw byte prefix, avoiding unnecessary value deserialization. Adds a hidden `backend` accessor for internal use in SystemTableStore. Refactors CLI test utilities to robustly check host:port reachability with fallback logic for localhost and IPv6 variants.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.