perf: eliminate queue/thread overhead in Python; remove redundant storage Mutex and use parking_lot in Rust#278
Merged
harumaki4649 merged 3 commits intomainfrom Apr 4, 2026
Conversation
Agent-Logs-Url: https://github.com/disnana/DictSQLite/sessions/06466c70-a433-4308-ad8c-9a8c9b23b192 Co-authored-by: harumaki4649 <83683593+harumaki4649@users.noreply.github.com>
…_lot in Rust, remove storage mutex Agent-Logs-Url: https://github.com/disnana/DictSQLite/sessions/06466c70-a433-4308-ad8c-9a8c9b23b192 Co-authored-by: harumaki4649 <83683593+harumaki4649@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Optimize Rust library wrapper to improve performance
perf: eliminate queue/thread overhead in Python; remove redundant storage Mutex and use parking_lot in Rust
Apr 4, 2026
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Rust-backed
dictsqlite_v2and the pure-Pythondictsqliteboth had structural bottlenecks preventing them from reaching their potential throughput.Python (
dictsqlite/main.py)queue.Queue+ daemon worker thread withthreading.RLock— every read previously required allocating aqueue.Queue, two thread context-switches, and a blockingget(); now it's a single uncontested lock acquisition.synchronous=NORMAL, 64 MBcache_size,temp_store=MEMORY, and 128 MBmmap_size— no longer requires the caller to opt in.operation_queuekept as a_SyncQueueno-op stub for backward compatibility (join()is now a no-op since all ops are synchronous)._process_queue_conflict_resolverthat recursively called_process_queue().Rust (
dictsqlite_v2)Remove redundant
MutexaroundStorageEngine(lib.rs)StorageEngineis internally thread-safe via itsr2d2connection pool andDashMap. Wrapping it in an additionalMutexwas pure overhead on every read and write:Fix unconditional LRU mutex acquisition in
get()/set()(lib.rs)The
WriteThroughmode causedaccess_tracker(aMutex<LruCache>) to be locked on every hot-tier read, even when the cache was nowhere near its eviction threshold. The condition now only tracks when approaching capacity, consistent with the behaviour of other persist modes.Switch
std::sync::Mutex→parking_lot::Mutex(lib.rs,async_ops.rs)parking_lotwas already a declared dependency butstd::sync::Mutexwas being used instead.parking_lot::Mutexhas lower overhead and removes the need forunwrap()on lock acquisition.