Merged
Conversation
Unifies storage paths under a single data_path in config, adds file_helpers utilities, and updates all references to use new path helpers. Refactors job and live query creation, update, and deletion to use Raft MetaCommands for cluster-wide replication, including new methods in ProviderMetaApplier and SubscriptionService. Moves system views out of kalamdb-system into kalamdb-core/views, removes legacy cluster/stats providers, and updates tests and documentation accordingly. Updates dependencies and build scripts to support new features.
Moved and reorganized backend and CLI integration tests into new testserver and categorized directories. Updated Cargo.toml test paths to match new locations. Removed legacy integration tests and added new HTTP-based test modules. Adjusted configuration examples and documentation to use updated cluster_id defaults. Refactored test_dml_parameters to use shared session context from the test server.
Implemented support for SQL parameter placeholders (e.g., $1, $2) in DELETE and INSERT handlers, including conversion and validation. Prevented primary key updates in UPDATE handler. Improved test coverage and reliability by using internal user IDs for storage assertions, added helper functions, increased timeouts, and clarified error handling. Added example server.toml configuration file.
Moved common test helpers to a shared location, reorganized and renamed test files for clarity, and split live query WebSocket tests into separate files for better isolation. Added new test entries in Cargo.toml, improved JWT handling in test infrastructure, and made minor cross-platform test fixes. Deprecated or removed legacy and combined tests, and updated documentation to reflect the new test organization.
Introduces secondary index scan support for system.jobs and system.live_queries providers, enabling more efficient queries using DataFusion filters. Adds validation for StorageId and UserName to prevent path traversal and injection attacks. Refactors JWT claims to use strongly typed Role and UserName. TableId now serializes as 'namespace.table' for JSON compatibility. Also includes new integration tests for index usage and scenario coverage, and uses hostname for single-node cluster ID.
Introduces `bootstrap_isolated` and `AppContext::create_isolated` to ensure each test gets a fresh AppContext, preventing state leakage between tests. Adds clean Raft group shutdown to avoid 'Fatal(Stopped)' errors in tests. Updates test helpers and scenarios to use isolated bootstrapping and user creation, increases test rate limits, and improves TableId deserialization. Also adds a minimal user table debug test and enhances EXPLAIN test reliability.
Refines USER table row-level security (RLS) logic to allow access for any authenticated user, with filtering applied at query time. Adds user_id caching to the test HTTP server for correct JWT generation and RLS enforcement, and updates helpers and test scenarios to require explicit user_id lookups and caching. Test scenarios now use per-user clients and data seeding to ensure correctness under RLS, and several tests are marked as ignored where SHARED table subscriptions are not supported. Also improves error handling, test reliability, and code clarity in test infrastructure and scenarios.
Adds debug logging for default column handling in insert and table definition logic. Updates CREATE TABLE parser to rewrite AUTO_INCREMENT to DEFAULT SNOWFLAKE_ID() for consistent default assignment. Enhances error handling in QueryExecutor to ensure authentication/authorization errors return proper error codes, preventing CLI false positives. Updates CLI test to check table_id in namespace.table format. Updates raft replication spec task to recommend using TEXT instead of VARCHAR in tests.
Replaced raw byte serialization with type-safe model objects for DML operations in applier traits and executors. Raft meta applier methods now return descriptive messages instead of unit, improving error handling and feedback. Updated job and user operations to use domain types and propagate result messages. Commented out verbose subscription logging. Added restore_raft_state_machines to AppContext for snapshot recovery. Updated dependencies to include kalamdb-raft.
Refactored schema registry, manifest service, and related components to require explicit AppContext references, removing global AppContext::get() usage in production code. Updated CachedTableData, PathResolver, and persistence logic to accept AppContext as a parameter. Added new ManifestService::new_with_registries for dependency injection. Adjusted tests and flush jobs to pass AppContext. This improves testability, clarity, and future extensibility of storage and schema resolution.
Reorganize and modularize integration tests by moving and renaming test files, introducing submodules for auth, production, schema, sql, storage, and system tests. Update Raft command and applier test code to use new row and filter parameter types, reflecting recent API changes. Add isoc23_shim.c for glibc compatibility, and clean up Cargo.toml test entries to use new test drivers. Minor improvements to test server initialization and randomization logic.
Introduced the ntest crate to enforce per-test timeouts for HTTP integration tests, improving test reliability and preventing hangs. Added a run_tests_serial.sh script to run tests serially to avoid state conflicts. Adjusted flush job wait deadlines for faster feedback.
Introduces handlers for CLUSTER FLUSH, CLUSTER CLEAR, and CLUSTER LIST SQL commands, registering them in the handler registry. Refactors live query and Raft APIs to use ConnectionId for type safety. Improves error messages for subscription permissions, normalizes CREATE TABLE success messages, and enhances test determinism in stream eviction. Updates tests and internal APIs to match new command and row structures.
Refactors user table provider to allow privileged roles to scan all users, adds stricter JWT claim handling in tests, and ensures test isolation by using unique identifiers based on timestamps. Also improves error handling in test user creation, restricts FLUSH TABLE for stream tables, and enhances system jobs index usage tests for reliability.
Introduce comprehensive documentation files for cluster node lifecycle management and testing, including delivery summary, implementation completion, checklist, visual guide, and a documentation index. These documents provide overviews, API references, architecture details, test scenarios, and verification steps for the new cluster test infrastructure.
…into feature/cluster
Consolidates system table and view definitions into a single enum, adds support for virtual system views (e.g., cluster, stats, cluster_groups), and updates registration logic to handle both persisted tables and views. Removes legacy constants and re-exports, updates schema registry and persistence APIs to use a unified get_table_if_exists method, and ensures all system objects are registered with DataFusion. Also adds new documentation and tests for system table/view handling.
Refactors the cluster_groups virtual view to provide direct OpenRaft metrics, reorders and renames columns, and adds new fields such as vote, last_log_index, purged, committed, state, current_leader, and millis_since_quorum_ack. Updates system table schemas to use KalamDataType::Json for JSON columns in manifest, storages, and users tables. Changes a log statement in raft_store.rs from info to debug. Comments out a test in system.rs. Updates server.toml to enable and configure the [cluster] section.
Implements stub handlers for CLUSTER JOIN and CLUSTER LEAVE commands, returning warnings that these features are not yet implemented. Enhances CLUSTER FLUSH and CLUSTER CLEAR to perform actual snapshot operations and provide detailed output, and upgrades CLUSTER LIST to display a formatted, detailed cluster overview. Refactors job logging to use log::Level, restricts cluster operations to admin users (except CLUSTER LIST), and exposes snapshot info in the Raft manager. Updates SQL classifier and parser to support new cluster commands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.