Feat/hardening v2.2#163
Merged
Merged
Conversation
All Engine write methods now take &self instead of &mut self, so let mut engine is no longer needed in tests, benches, and examples.
- fix(wal): replace try_clone() with read-only open handle (EBADF fix) - fix(wal): tolerant recovery with resync + MIN_LENGTH=35 + version byte validation - fix(wal): position tracking to prevent OOM on false-positive frame lengths - feat(metrics): add EngineMetrics with atomic counters and Prometheus format - feat(api): add GET /metrics endpoint exposing engine metrics - feat(backup): add create_snapshot, list_snapshots, restore_snapshot - chore(wal): remove unused Seek/SeekFrom imports Closes #146 Closes #149 Closes #150
- ApiToken::new() now returns Result<Self, AuthError> - ApiToken::is_expired() now returns Result<bool, AuthError> - SystemTime errors are propagated as AuthError::Internal instead of panicking - All callers updated (TokenManager + tests) Closes #136
- feat(engine): close() now syncs WAL before shutdown for durability - fix(engine): close() joins compaction thread before WAL sync (prevents deadlock with compaction holding core lock) - docs(engine): explain why memtables are NOT flushed on close (no manifest persistence would cause data loss on restart) - chore(agents): remove step limits from all 14 agent configs (maxSteps/steps set to 9999) Closes #137
… panic - MergeIterator::seek() now seeks each sub-iterator to the target key and rebuilds the binary heap with valid iterators - Adds 2 new tests: test_merge_iterator_seek and test_merge_iterator_seek_with_duplicates - Replaces unimplemented!() with full implementation Closes #138
- Removed root-level orphans: engine.rs, iterator.rs, memtable.rs, table.rs - Removed src/ dead modules: engine.rs, error.rs, merge_iterator.rs, record.rs, value.rs - None were referenced by any compiled module (src/lib.rs only has api/core/features/infra/storage) Closes #139
- Removed private engine::MemTable (BTreeMap<Vec<u8>, Vec<u8>>) from engine/mod.rs - Engine now uses core::memtable::MemTable (BTreeMap<Vec<u8>, LogRecord>) - Removed InternalMemTableIterator, replaced with MemTableIterator from storage/iterator.rs - Added MemTable::put() and delete() convenience methods - Added MemTable::new_unlimited() for zero-max-size instances - get_cf() now returns None for tombstoned records - flush_memtable_impl converts LogRecord values to raw Vec<u8> for Table Closes #140
- Added clap 4.5 dependency with derive feature - Replaced no-op skeleton with full command set: get, set, delete, scan, keys, count, stats, flush, compact - Commands support --db path and --cf column family flags - scan and keys support --prefix, --lower/--upper, --limit - Engine opened with GlobalBlockCache via CliEngine type alias Closes #141
- GET /keys/{key} — get single key
- PUT /keys/{key} — upsert key with JSON body
- DELETE /keys/{key} — delete key
- GET /keys?prefix=&limit= — list/filter keys (enhanced)
- GET /stats — engine statistics
- POST /admin/flush — force memtable flush
- POST /admin/compact — force compaction
Closes #142
…sst_iterator.rs) - Removed src/core/cache.rs (empty Cache struct, unused) - Removed src/core/version.rs (PhantomData wrapper, unused after version_set.current_version() removal) - Removed src/core/engine/manifest.rs (empty Manifest struct) - Removed src/storage/sst_iterator.rs (stub with engine reference only) - Cleaned up module declarations in core/mod.rs, engine/mod.rs, storage/mod.rs Closes #143
- Split compact_cf_core into explicit Plan/I-O/Apply phases - Plan phase clones table metadata under the lock (fast) - I/O phase prepared to run without lock (future: release mutex) - Apply phase atomically replaces compacted tables - Documented the three-phase approach for future lock-release Closes #145
…lookups - Table::build() now creates a Bloom filter with 1% false-positive rate - Existing VersionSet::get() already checks table.bloom_filter before searching the BTreeMap — negative lookups now skip the map entirely - Closes benchmark gap: bloom_filter negatives go from O(log n) BTreeMap search to O(k) Bloom filter check (k = number of hash functions) Closes #158
- Added WAL_SYNC_INTERVAL=4: every 4th write_record triggers an fsync instead of every single write (was ~1100 ops/s bottleneck) - write_batch continues to fsync once for the entire batch - sync() resets batch counter for clean shutdown durability - batch counter is Mutex-protected for thread safety Closes #156
- EngineCore now stores per-CF WALs (HashMap<String, WriteAheadLog>)
- WAL files: wal.log (default), wal-{cf}.log (other CFs)
- Init discovers all wal-*.log files and recovers each independently
- flush_memtable_impl uses clear() instead of retain() — O(1) per flush
instead of O(N) rewrite of entire WAL
- close() syncs all WALs
- stats sum sizes across all WALs
- Snapshot saves/restores all per-CF WAL files
- Restore handles wal-*.log patterns
Closes #157
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements 22 issues across hardening, performance, features, and cleanup for the ApexStore LSM-tree engine.
Issues Implementadas
🔴 Critical / High — Bugs & Hardening
panic!()emApiToken— crash HTTPEngine::drop()não finaliza WALMergeIterator::seek()unimplemented!()— runtime panic⚡ Performance
compact_cf— three-phase approachwrite_batch()— agrupar fsyncsretain()no flushTable::build()— cachear filters✨ Features
get,set,delete,scan,keys,stats,flush,compactGET/PUT/DELETE /keys/{key},GET /stats,POST /admin/flush,POST /admin/compactEngineMetrics+ endpointGET /metrics(Prometheus)create_snapshot,list_snapshots,restore_snapshot♻️ Refactors & Cleanup
engine.rs,iterator.rs,memtable.rs,table.rs+src/engine.rs,error.rs,merge_iterator.rs,record.rs,value.rs)MemTableduplicado (engine privado →core::memtable)cache.rs,version.rs,manifest.rs,sst_iterator.rs)&mut self→&selfnos writes (já existente)search_in_blockbinary search (já existente)EngineCore(já existente)parking_lot(já existente)CI & Quality
cargo test --all-features --workspace— 119 passedcargo clippy --all-targets --all-features -- -D warnings— cleancargo fmt --all -- --check— cleancargo build --release— cleanCommits (13)