Merged
Conversation
Migrates manifest, plan, and query caches from DashMap to moka::sync::Cache for improved performance and automatic eviction (TinyLFU, TTI, TTL). Removes custom LRU/TTL logic and leverages moka's built-in expiration and capacity management. Updates affected tests and configuration to use moka. Adds per-key TTL support for query cache and refactors cache APIs for consistency.
Removes unused last_accessed getter logic from manifest cache and provider, simplifying the API and schema. Refactors PlanCache to use a structured key (namespace, role, SQL) for improved cache efficiency and removes user_id from the cache key. Updates workspace Cargo.toml files to use workspace-inherited metadata fields for version, edition, rust-version, authors, license, and repository. Updates query cache stats and test assertions for consistency.
Migrates CLI credential management to store only JWT tokens (not plaintext passwords) in the credentials file, updating all related logic, help, and tests. Adds support for logging in with username/password to obtain and persist JWT tokens, and for using --save-credentials to store tokens after login. Removes legacy connect meta-command, updates session and credential display, and improves security by never storing passwords. Also optimizes batch PK existence checks in cold storage for shared tables.
Removed ManifestCacheService and integrated its hot cache and RocksDB persistence logic into ManifestService. Updated all references and helpers to use ManifestService for manifest caching, invalidation, and eviction. Cleanup, flush, and eviction jobs now operate on the unified ManifestService, simplifying manifest management and reducing code duplication.
Refactor integration tests to use unique storage IDs for isolation, add utility to wait for async cleanup job completion after table drops, and relax performance assertions for versioned queries. Also, improve directory cleanup logic in object store operations for more robust file removal.
Implements tiered eviction in ManifestService to prioritize shared table manifests over user tables using a configurable weight factor. Removes unused dependencies (actix, env_logger, crossterm, tabled, console) and updates Rust version to 1.92 in documentation and configuration. Refactors password complexity validation to use kalamdb_auth::password::validate_password_with_policy. Updates dependencies in Cargo.toml and Cargo.lock, and adds serial_test for improved test isolation.
Added sccache caching to all major CI/CD build jobs in release.yml for faster compilation. Updated .cargo/config.toml to use the sparse registry protocol for improved dependency resolution. Removed env_logger initialization from a backend test. Added documentation files summarizing and guiding build speed improvements and optimizations.
Introduces a new system.datatypes virtual view in the backend to provide Arrow→KalamDB/SQL type mappings, enabling the UI to display user-friendly type names. Updates the TypeScript SDK, API types, and UI components to use schema-based column definitions and type-aware formatting. Adds a React hook (useDataTypes) for fetching and caching type mappings, centralizes table and timestamp formatting config, and improves audit logging by redacting passwords. Also updates documentation to clarify build performance and SDK response formats.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements to the build system and dependency management, with a focus on modernizing the Rust toolchain, optimizing build performance, and cleaning up dependencies. The most significant changes include upgrading the Rust version to 1.92, integrating
sccachefor faster CI builds, and refining the set of dependencies and features used across the workspace.Build system and CI improvements:
sccacheinto all relevant GitHub Actions build jobs to speed up CI by caching compilation artifacts; added related environment variables and stats reporting. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]sparsefor faster dependency resolution and clarified parallel compilation defaults in.cargo/config.toml.Dependency and feature management:
Cargo.toml:tokiofeatures for more precise async runtime usage.arrowfeatures for broader data format support and setdefault-features = false.actix,tabled,crossterm, andconsole. [1] [2] [3]serial_test,bytes,moka, andntestversions/features. [1] [2]workspace = truefor more crates and dependencies. [1] [2] [3] [4] [5]Development profile enhancements:
[profile.dev]settings for faster development builds: reduced debug info, maximized codegen parallelism, and enabled faster debug info handling on macOS..cargo/config.tomlfor clarity.API and code improvements:
new) to theErrorResponsestruct inkalamdb-apifor easier error handling.