Repository navigation
v1.9.2
#45
Replies: 1 comment
NeoSQLite Benchmark ResultsDate: 202603150230 Summary
Category Results
NotesPartial Comparisons (one side skipped):
Detailed Timing (Iteration-level)CRUD Operations
Query Operators
$expr Operator
Update Operators
Aggregation Stages
Aggregation (Additional)
Aggregation Stages (Extended)
Aggregation (Extended)
Aggregation Cursor
Index Operations
Search Index
Reindex
Bulk Operations
Bulk Executors
Find & Modify
Distinct
Nested Queries
$elemMatch
Cursor Operations
Cursor Methods
Session & Transactions
Binary Data
ObjectId
$type Operator
Date Operations
Math Operators
String Operators
Array Operators
Object Operators
$expr (Additional)
$expr (Extended)
$expr (Complete)
$expr (Success Cases)
Raw Batches
Change Streams
Text Search
GridFS
Window Functions
Window Math
$graphLookup
$fill Stage
JSON Schema
Bucket Aggregation
$mod Operator
Update (Additional)
Update Modifiers
Bitwise Operators
$pullAll Operator
Collection Methods
Collection (Additional)
Database Methods
Type Operators
Expression Operators
Object Operators (Extended)
Array Operators (Extended)
Binary Operators
Options Classes
|
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
NeoSQLite v1.9.2 Release Notes
TL;DR
🚀 Performance Release — SQL-Tier Aggregation Expansion & Update Operator Optimizations
Overview
NeoSQLite v1.9.2 is a performance-focused release that significantly expands SQL-tier aggregation pipeline optimization and implements SQL-level optimizations for complex update operators. This version reduces reliance on Python fallback, delivering faster query execution and better resource utilization while maintaining full PyMongo API compatibility.
Key Achievement: With this release, NeoSQLite achieves unprecedented SQL-tier coverage for aggregation pipelines and update operations, moving critical performance paths from Python to native SQLite execution for maximum efficiency.
Major Feature: Expanded SQL-Tier Aggregation Optimization
Comprehensive support for additional aggregation pipeline stages translated to native SQL:
New SQL-Translated Stages
New SQL Operators
$objectToArray:$redact Stage Enhancements
PRUNE/KEEP Pattern Support:
KEEP/PRUNEandPRUNE/KEEPconditional patternsthen: $$PRUNE(inverts to WHERE NOT condition)Example:
Performance: SQL-Level Update Operator Optimization
Optimized update operations by implementing Tier-2 SQL paths for complex array operators:
Optimized Update Operators
$pop:json_remove()and array indexing$pushwith$each:json_patch()$slicemodifier for array size limits$addToSet:$eachbatchesSQLite Version Detection
Automatic Feature Detection:
_supports_relative_json_indexing()detects SQLite 3.42.0+ capabilitiesFeature Cache:
_SQLITE_FEATUREScache for efficient capability checkingPerformance Impact
Benchmark Improvements: High-Precision Timing
Comprehensive refactoring of API comparison benchmark suite for accurate performance measurement:
Benchmark Enhancements
Precise Timing Placement:
Accumulation Mode:
Command Standardization:
Extended Coverage:
$contains)Benchmark Infrastructure Updates
Reporter Improvements:
Runner Optimizations:
watch()from benchmarking (change streams not comparable)Shell Script:
scripts/run-api-benchmark.shfor automated executionImplementation Details
Core Improvements
SQL Tier Aggregator (
neosqlite/collection/sql_tier_aggregator.py):Update Operations (
neosqlite/collection/query_helper/update_operations.py):_can_use_sql_updates()extended for $pop, $push, $addToSet_perform_enhanced_sql_update()with optimized match blocks_build_update_clause()and_build_sql_update_clause()enhancedQuery Engine (
neosqlite/collection/query_engine/__init__.py):Connection Handling (
neosqlite/connection.py):New Modules
Query Engine (
neosqlite/collection/query_engine/):Modified Files Summary
Test Suite Enhancements
New Test Coverage:
tests/test_expr/test_sql_tier_optimization.py- SQL-tier aggregation teststests/test_query_engine_suite.py- Query engine comprehensive testsTest Results:
Test Results
Unit Tests
All existing tests pass without modification, confirming 100% backward compatibility:
SQL-Tier Optimization Tests
New test coverage for SQL translation features:
API Comparison Tests
Benefits for Users
For Application Developers
Faster Aggregations: 10+ new stages now execute in SQL tier for maximum performance.
Optimized Updates: $pop, $push, $addToSet operations significantly faster.
Better $redact Support: Both PRUNE/KEEP and KEEP/PRUNE patterns optimized.
No Breaking Changes: All existing code continues to work without modification.
For Data Engineers
Complex Pipelines: More aggregation stages benefit from SQL-tier optimization.
Bucket Analysis: $bucket and $bucketAuto for efficient data binning.
Joins: $lookup and $unionWith for cross-collection queries.
Document Transformation: $replaceRoot and $replaceWith for reshaping results.
For DevOps
Consistent Performance: Reduced Python fallback means more predictable execution.
Better Resource Utilization: SQL-tier operations use less memory and CPU.
SQLite Version Detection: Automatic feature enablement based on SQLite version.
For Contributors
Clear Optimization Patterns: Well-documented SQL translation strategies.
Modular Architecture: Easy to add new SQL-tier optimizations.
Benchmark Infrastructure: Data-driven performance validation.
Migration Guide
Upgrading from v1.9.1
Steps:
pip install neosqlite==1.9.2Effort: Minimal — typically 5-10 minutes for most users.
Leveraging New SQL-Tier Optimizations
Automatic Optimization (no code changes):
Update Operations (automatically optimized):
Using Benchmark Mode
Run Updated Benchmarks:
Automated Benchmarking:
Analyze Results:
documents/benchmarks/Benchmark-NeoSQLite-1.9.2-YYYYMMDDHHMM.mddocuments/benchmarks/Benchmark-NeoSQLite-1.9.2-YYYYMMDDHHMM.csvKnown Limitations
SQL-Tier Aggregation Limitations
Python Fallback Required When:
Performance Impact:
Update Operator Limitations
SQLite Version Dependencies:
Complex Modifiers:
$redact Stage Limitations
DESCEND Pattern:
Complex Conditions:
Acknowledgments
This release represents a significant investment in SQL-tier optimization and performance engineering. The expansion of SQL-translated aggregation stages and update operators demonstrates NeoSQLite's commitment to maximizing SQLite's native capabilities while maintaining full PyMongo API compatibility.
The comprehensive benchmark refactoring establishes a foundation for data-driven performance improvements, ensuring that optimization efforts are guided by empirical measurements rather than assumptions.
Special recognition for the major refactoring effort in
sql_tier_aggregator.py, reducing technical debt while expanding functionality — a testament to sustainable software engineering practices.NeoSQLite v1.9.2 — SQL-Tier Performance, PyMongo Compatibility.
This discussion was created from the release v1.9.2.
All reactions