Merged
Conversation
Refactor table provider initialization to use a shared TableServices and pass Arc<TableDefinition>/Arrow schema into TableProviderCore; update schema registry and many tests/providers to the new TableProviderCore/User/Shared/Stream provider APIs. Add parquet reader performance features (column projection, bloom-filter row-group pruning, row-group selection) with new helper functions and expose them from filestore. Map TableError::ConstraintViolation to KalamDbError::InvalidOperation. Add CTE integration tests and remove the legacy test_helpers module; update tests to match the new APIs.
Improve CLI test infrastructure and silence flaky/slow tests. Key changes: - Notes.md: add items about delivery_policy and stream delivery option. - cli/tests/common/mod.rs: major refactor of test_context and server detection (introduce KALAMDB_SERVER_TYPE handling, explicit Running/Fresh/Cluster modes, earlier .env loading, safer Unix lock handling, skip cluster probing for single-node, more explicit error messages). Simplify force_auto_test_server_url and async variant. Improve server readiness and is_server_running logic. - Remove helper run scripts: delete backend/run_tests_serial.sh and cli/run-tests.ps1. - Comment out slow/integration tests in kalamdb-core (live_multi_subscription.rs, test_config_access.rs) to avoid starting Raft in CI. - Tests & utils tweaks: add SchemaRegistry usage in some core tests, adapt TableProviderCore construction in stream eviction test, remove an obsolete pk unit test, remove unused Duration imports across several test files, and fix timestamp parsing by adding .and_utc() in test_update_all_types.rs. These changes aim to make CLI tests more robust and fail-fast when misconfigured, reduce flaky behavior in CI, and clean up minor test issues.
Instrument core components with tracing spans for better observability: add debug/trace spans in SQL handler and RocksDB backend (get/put/delete/batch/scan) with contextual fields (partition, value_len, op_count, etc.). Remove the unused `classify_span` variable and drop `fern` from the backend workspace dependencies. Add a Jaeger all-in-one service to docker-compose (OTLP ports exposed) and comment out the previous WebDAV example. Update Notes.md with a checklist of intended span coverage and a note to add Jaeger.
Introduce OpenTelemetry (opentelemetry, opentelemetry-otlp, tracing-opentelemetry) and upgrade tracing crates; wire OTLP settings into config (types, defaults, and env overrides). Add tracing spans/instrumentation across runtime components: authentication (header/credentials/jwt flows record role/user and request kind), jobs runner (job execution spans), topic publisher (publish spans and published_count), and minor connection manager tracing scaffolding. Update tests to use isolated AppContext with RocksDB backend and the new ExecutionContext API. Small docs updates: AGENTS.md tracing field convention and Notes.md additions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces tracing and observability improvements across the KalamDB codebase, focusing on authentication, logging conventions, and configuration. The main changes add OpenTelemetry and Jaeger support, enhance span logging for authentication flows, and provide environment-based overrides for tracing settings.
Tracing & Observability Integration:
Cargo.tomlfiles for the main project and relevant crates, enabling distributed tracing and observability features. [1] [2] [3] [4]authenticate,authenticate_header,authenticate_username_password,authenticate_bearer) withinbackend/crates/kalamdb-auth/src/services/unified.rs, capturing request kind, user, role, and authentication outcomes. [1] [2] [3] [4]Configuration Enhancements:
backend/crates/kalamdb-configs/src/config/defaults.rsandbackend/crates/kalamdb-configs/src/config/override.rs, allowing customization of tracing endpoints, protocols, service names, and timeouts. [1] [2] [3]Documentation & Logging Conventions:
AGENTS.mdto standardize span/event logging to usetable_id(namespace.table) instead of separate fields.Notes.mdfor new span points (e.g., auth check, provider scan, jobs execution, storage ops, parquet ops, push/publish events) and Jaeger integration. [1] [2]Feature Planning:
Code Quality & Refactoring:
These changes collectively enhance the system's observability, provide better diagnostics for authentication and other critical flows, and lay groundwork for distributed tracing with OpenTelemetry and Jaeger.