Add WASM support foundation: IPageStorage/IWriteAheadLog abstractions + in-memory backends#62
Conversation
…ateInMemory factory, and WASM roadmap Agent-Logs-Url: https://github.com/EntglDb/BLite/sessions/e4b3c63e-3286-4d25-9134-1f34c9e7100d Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>
…eoutMs in Dispose, improve test variable name Agent-Logs-Url: https://github.com/EntglDb/BLite/sessions/e4b3c63e-3286-4d25-9134-1f34c9e7100d Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR lays the groundwork for WASM support by decoupling BLite’s storage engine from file-system–dependent I/O (PageFile, WriteAheadLog) via pluggable abstractions, and adding in-memory backends plus a BLiteEngine.CreateInMemory() factory.
Changes:
- Introduces
IPageStorage/IWriteAheadLoginterfaces and updatesPageFile/WriteAheadLogto implement them. - Adds
MemoryPageStorageandMemoryWriteAheadLogin-memory implementations and aBLiteEngine.CreateInMemory()factory path. - Adds integration/unit tests for the in-memory storage stack and a WASM roadmap document.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/BLite.Tests/InMemoryStorageTests.cs | New unit/integration tests covering in-memory page storage, WAL, and CreateInMemory(). |
| src/BLite.Core/Transactions/WriteAheadLog.cs | Implements IWriteAheadLog (no functional change intended). |
| src/BLite.Core/Transactions/MemoryWriteAheadLog.cs | Adds in-memory WAL implementation for non-filesystem environments. |
| src/BLite.Core/Transactions/IWriteAheadLog.cs | New WAL abstraction interface. |
| src/BLite.Core/Storage/StorageEngine.cs | Switches internal fields to abstractions and adds constructor for injected backends. |
| src/BLite.Core/Storage/StorageEngine.Memory.cs | Updates multi-file routing helpers to return IPageStorage. |
| src/BLite.Core/Storage/PageFile.cs | Implements IPageStorage (no functional change intended). |
| src/BLite.Core/Storage/MemoryPageStorage.cs | Adds in-memory page storage backend. |
| src/BLite.Core/Storage/IPageStorage.cs | New page storage abstraction interface. |
| src/BLite.Core/DocumentDbContext.cs | Adds protected constructor taking a pre-built StorageEngine. |
| src/BLite.Core/BLiteEngine.cs | Adds internal constructor + CreateInMemory() factory. |
| WASM_SUPPORT.md | Adds a WASM support roadmap and design notes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| { | ||
| cancellationToken.ThrowIfCancellationRequested(); |
There was a problem hiding this comment.
ReadPageAsync slices destination.Span to _pageSize without first validating that destination.Length >= _pageSize. If a caller passes a too-small buffer, this throws an ArgumentOutOfRangeException instead of the consistent ArgumentException used by ReadPage, and the error message won’t match the contract in IPageStorage. Add the same length check as ReadPage (and ideally also call ThrowIfDisposed() here for consistency).
| { | |
| cancellationToken.ThrowIfCancellationRequested(); | |
| { | |
| ThrowIfDisposed(); | |
| cancellationToken.ThrowIfCancellationRequested(); | |
| if (destination.Length < _pageSize) | |
| throw new ArgumentException($"Destination must be at least {_pageSize} bytes."); |
| if (_lock.Wait(_writeTimeoutMs)) | ||
| { | ||
| try | ||
| { | ||
| _records.Clear(); | ||
| _disposed = true; | ||
| } | ||
| finally | ||
| { | ||
| _lock.Release(); | ||
| _lock.Dispose(); | ||
| } | ||
| } | ||
| else | ||
| { | ||
| _records.Clear(); | ||
| _disposed = true; | ||
| _lock.Dispose(); | ||
| } |
There was a problem hiding this comment.
Dispose() falls back to _records.Clear() when it can’t acquire _lock within the timeout. Clearing the underlying List<WalRecord> without holding the semaphore can race with concurrent writers that do hold the lock, potentially corrupting the list or throwing. Prefer an unconditional lock acquisition during dispose (or skip clearing if the lock can’t be obtained) so _records is never mutated concurrently.
| if (_lock.Wait(_writeTimeoutMs)) | |
| { | |
| try | |
| { | |
| _records.Clear(); | |
| _disposed = true; | |
| } | |
| finally | |
| { | |
| _lock.Release(); | |
| _lock.Dispose(); | |
| } | |
| } | |
| else | |
| { | |
| _records.Clear(); | |
| _disposed = true; | |
| _lock.Dispose(); | |
| } | |
| _lock.Wait(); | |
| try | |
| { | |
| _records.Clear(); | |
| _sizeBytes = 0; | |
| _disposed = true; | |
| } | |
| finally | |
| { | |
| _lock.Release(); | |
| _lock.Dispose(); | |
| } |
| public void CreateInMemory_NoFileCreated() | ||
| { | ||
| // Verify that using an in-memory engine doesn't create any .db files. | ||
| var tempDir = Path.GetTempPath(); | ||
| var before = Directory.GetFiles(tempDir, "*.db").Length; | ||
| using (var engine = BLiteEngine.CreateInMemory()) { } | ||
| var after = Directory.GetFiles(tempDir, "*.db").Length; | ||
| Assert.Equal(before, after); |
There was a problem hiding this comment.
CreateInMemory_NoFileCreated is likely to be flaky because it counts *.db files in the global temp directory; unrelated processes/tests can create or delete .db files between the before and after snapshots, causing spurious failures. Consider removing this assertion or rewriting it to use an isolated temporary directory / file-system abstraction so the test only observes files created by this test.
| public void CreateInMemory_NoFileCreated() | |
| { | |
| // Verify that using an in-memory engine doesn't create any .db files. | |
| var tempDir = Path.GetTempPath(); | |
| var before = Directory.GetFiles(tempDir, "*.db").Length; | |
| using (var engine = BLiteEngine.CreateInMemory()) { } | |
| var after = Directory.GetFiles(tempDir, "*.db").Length; | |
| Assert.Equal(before, after); | |
| public void CreateInMemory_CanBeCreatedAndDisposed() | |
| { | |
| // Avoid asserting over the global temp directory because unrelated | |
| // processes or parallel tests can create/remove .db files there and | |
| // make the test flaky. This still verifies that the in-memory engine | |
| // can be constructed and disposed successfully. | |
| using var engine = BLiteEngine.CreateInMemory(); | |
| Assert.NotNull(engine); |
| public StorageEngine(IPageStorage pageStorage, IWriteAheadLog wal) | ||
| { | ||
| _config = PageFileConfig.Default; | ||
| _pageFile = pageStorage ?? throw new ArgumentNullException(nameof(pageStorage)); |
There was a problem hiding this comment.
This new constructor doesn’t assign several readonly fields declared on StorageEngine (e.g. _indexFile, _collectionFiles, _collectionNameToSlot, _collectionSlotToName, _slotsFilePath). In C#, unassigned readonly fields cause a compile error (CS0171). Explicitly initialize these to null (and any other readonly fields not applicable in single-backend mode) within this constructor.
| _pageFile = pageStorage ?? throw new ArgumentNullException(nameof(pageStorage)); | |
| _pageFile = pageStorage ?? throw new ArgumentNullException(nameof(pageStorage)); | |
| _indexFile = null; | |
| _collectionFiles = null; | |
| _collectionNameToSlot = null; | |
| _collectionSlotToName = null; | |
| _slotsFilePath = null; |
BLite's storage layer was tightly coupled to
PageFile(memory-mapped files) andWriteAheadLog(FileStream), both unavailable in browser WASM environments. This PR decouples the engine from its concrete I/O backends by introducing pluggable interfaces, adds in-memory implementations usable in WASM today, and documents the remaining roadmap as discrete sub-issues.New interfaces
IPageStorage— extracted fromPageFile's public surface; all storage backends implement thisIWriteAheadLog— extracted fromWriteAheadLog's public surface; all WAL backends implement thisIn-memory backends (zero file-system dependencies)
MemoryPageStorage—ConcurrentDictionary<uint, byte[]>-backed page store; suitable for WASM, unit tests, and ephemeral cachesMemoryWriteAheadLog—List<WalRecord>-backed WAL; supports fullReadAll()/TruncateAsync()semanticsEngine changes
PageFileandWriteAheadLognow implement their respective interfaces — no behavioral changesStorageEngineinternal fields changed toIPageStorage/IWriteAheadLog; new pluggable constructor added:BLiteEngine.CreateInMemory()factory — no files created, data lives in process memory:DocumentDbContext(StorageEngine, BLiteKvOptions?)protected constructor — enables typed context subclasses to use any backendRoadmap
WASM_SUPPORT.mdbreaks the remaining work into 5 sub-issues:OpfsWriteAheadLog/IndexedDbWriteAheadLog)BLite.WasmNuGet package + auto-detection factory