Conversation
…lity - Introduced models for database inspection options, reports, and integrity issues. - Implemented the WalInspector class to inspect WAL files, including validation of headers and frames. - Added tests for various inspection scenarios, including missing WAL files and integrity checks. - Updated project references to include the new diagnostics module.
…tions - Updated SystemCatalogTests to pass cancellation token to database operations. - Updated WalTests to pass cancellation token to database operations. - Ensured all async calls in tests are properly handling cancellation tokens for better control over test execution.
…ema, and TriggerSchema - Added SchemaSerializer for serializing and deserializing TableSchema and IndexSchema to/from byte arrays. - Introduced Varint class for variable-length integer encoding. - Created DefaultStorageEngineFactory to manage storage engine components. - Defined IStorageEngineFactory interface for storage engine creation. - Implemented StorageEngineContext to hold references to storage engine components. - Added StorageEngineOptions for configuring storage engine behavior. - Introduced IClock interface and SystemClock implementation for time-based policies. - Developed TransactionCoordinator for managing transaction state and write-lock lifecycle. - Created IWriteAheadLog interface and WriteAheadLog class for write-ahead logging. - Implemented WalIndex for tracking committed frames in the WAL. - Added tests for serialization and varint functionality.
… interceptor support - Added `IPageChecksumProvider` interface and `AdditiveChecksumProvider` implementation for customizable checksum calculations in WAL. - Introduced `IPageOperationInterceptor` interface and `CompositePageOperationInterceptor` for handling lifecycle events during page operations. - Updated `Pager` and `PageBufferManager` to utilize interceptors for read/write operations, commits, checkpoints, and recovery. - Enhanced `StorageEngineOptions` to include a checksum provider and interceptors, with a builder pattern for configuration. - Created tests to validate the functionality of custom checksum providers and the behavior of interceptors during various database operations. - Refactored `WriteAheadLog` to use the new checksum provider for frame and header verification.
…ed async operations
Refactor WAL commit and read logic for fewer allocations and faster commit path. Improve AdditiveChecksumProvider with SIMD for checksum calculation. Use Buffer.BlockCopy for slotted page pointer shifts. Simplify page cache fetch logic. Add advanced benchmark diagnoser and support for --filter option. Remove redundant varint header length code in BTree.
- Introduced deferred checkpoint requests in CheckpointCoordinator to handle scenarios where checkpoints are requested while readers are active. - Updated ReleaseReaderSnapshot to prevent negative active reader counts. - Modified ShouldCheckpoint logic to set deferred checkpoint requests based on active readers and committed frame counts. - Implemented TryConsumeDeferredCheckpointRequest and ClearDeferredCheckpointRequest methods for managing deferred checkpoints. - Added MaxWalBytesWhenReadersActive option in PagerOptions to limit WAL growth when readers are active, preventing excessive WAL size. - Enhanced WriteAheadLog to handle exceptions more gracefully during WAL operations, ensuring robust error handling. - Updated tests to validate the behavior of deferred checkpoints and WAL backpressure limits under various scenarios. - Removed IBTreeIndexStore interface as it is no longer needed, simplifying the index store implementations.
…SQL editing capabilities
…rmance for indexed queries
…cans in query planner
…mary key lookups and enhancing statement caching
…ual predicates and optimizing query planner logic
…ove performance and reduce unnecessary cloning
…eate detailed documentation for storage improvements
…idual predicate handling and removing unnecessary row ownership checks in QueryResult
…operators and query planner
…re all columns are preserved
…ow ID handling and decode optimizations
…Result handling for materialized rows
…er reuse and estimated row count
…w count and capacity hint support
…ount support and improve row buffer handling
…DefaultRecordSerializer
…r and WriteAheadLog
…ance in WriteAheadLog
…chmarks for performance evaluation
…new methods for frame handling
…atch frame publishing
…ation optimization
…and checksum handling
…ze checkpoint handling in WriteAheadLog
…d prevent null reference exceptions
…gic from WriteAheadLog
…oved cell handling
- Introduced `perf-thresholds.json` to define performance thresholds for various benchmarks. - Created `perf-guardrails-last.md` to summarize benchmark results against the defined thresholds. - Implemented `Capture-Baseline.ps1` to capture baseline performance metrics for specified benchmarks. - Added `Compare-Baseline.ps1` script to compare current benchmark results against the baseline and generate a report. - Developed `Run-Perf-Guardrails.ps1` to orchestrate the benchmarking process and report generation. - Enhanced benchmark filtering to include additional benchmarks in the performance guardrails.
… page cache, and storage device functionality
… update project properties for packaging
… RELEASE_NOTES.md
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad2dda682c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Major storage engine refactoring, query execution performance optimizations, new CSharpDB.Storage.Diagnostics package, CI/CD infrastructure, and per-package NuGet README documentation. 169 files changed (~19.8K lines added).
Type of Change
Related Issues
N/A
Testing
dotnet build CSharpDB.slnxNew test suites: CheckpointPolicyTests, IndexCursorTests, PageCacheTests, StorageDeviceAndChecksumTests, StorageDiagnosticsTests, StorageEngineExtensibilityTests. New benchmarks: WalCoreBenchmarks, SqlTextStabilityBenchmarks, SystemCatalogBenchmarks, TriggerDispatchBenchmarks, ScanBenchmarks.
Checklist
Notes for Reviewers