Skip to content

feat(Storage): async partition scan with 'ready' event (Option B)#303

Draft
Copilot wants to merge 3 commits intomainfrom
copilot/sub-pr-301
Draft

feat(Storage): async partition scan with 'ready' event (Option B)#303
Copilot wants to merge 3 commits intomainfrom
copilot/sub-pr-301

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 1, 2026

Storage initialization was synchronous (scanForFilesSync in constructor), unlike EventStore's async 'ready' pattern. This implements Option B: partition scanning is now async, and Storage emits 'ready' once both the scan and open() have completed.

Core changes

  • ReadableStorage: scanPartitions() uses async scanForFiles(); _emitReadyIfConditionsMet() fires 'ready' when both _scanDone and _opened are true; close() resets _opened so 'ready' re-fires on the next open()
  • WritableStorage: removes synchronous this.emit('ready') from open(); defers checkTornWrites() for LOCK_RECLAIM (via _needsRepair flag) to just before 'ready' fires, via an override of _emitReadyIfConditionsMet()
  • EventStore.initialize(): registers storage.on('ready', ...) before calling scanStreams(), so checkUnfinishedCommits() only runs after all partitions are loaded
  • util.scanForFiles(): handles ENOENT gracefully on non-root subdirectories to survive transient lock-directory deletion during scanning

New caller contract

// Before — immediately usable after open()
const storage = new Storage({ dataDirectory });
storage.open();
storage.read(1); // OK only if data was written in this session

// After — wait for 'ready' before accessing pre-existing partitions
const storage = new Storage({ dataDirectory });
storage.on('ready', () => {
    storage.read(1); // safe: all existing partitions loaded
});
storage.open();

New writes and in-session reads work immediately (new partitions are created on demand). Only reading data from a previous session requires waiting for 'ready'. Re-opening the same instance re-emits 'ready' synchronously (scan already done).

Test impact

Six Storage.spec.js tests that created a new Storage on existing data were updated to the 'ready' pattern. The LOCK_RECLAIM test's synchronous readyFired assertion was removed since repair is now deferred to just before async 'ready'.

Copilot AI changed the title [WIP] [WIP] Implement option B for hierarchical stream names based on review feedback feat(Storage): async partition scan with 'ready' event (Option B) May 1, 2026
Copilot AI requested a review from albe May 1, 2026 18:13
Copilot finished work on behalf of albe May 1, 2026 18:13
Base automatically changed from copilot/add-folder-partitioning-scheme to main May 1, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants