feat(buffer): implement Buffer with pluggable storage#15
Merged
Conversation
- Create BufferStorage protocol for pluggable backends - Implement InMemoryBufferStorage (default, fast, volatile) - Create Buffer class with per-partition buffering - Implement enqueue() with size-based flush (default: 100 events) - Add retry-flush-before-failing strategy - If buffer at max_size, try flushing first - Only raise BufferFullError if flush fails - Add BufferFullError for defensive max_size limit (default: 1000) - Add asyncio.Lock to prevent race conditions - Add comprehensive tests for enqueue, flush, isolation, and error cases - Coverage: 97% on buffer.py, 95% on buffer_storage.py Design decisions: - Pluggable storage enables disk, Redis, or custom backends - Retry-flush follows production CDP patterns (fail fast with one retry) - Protocol-based design matches EventStore and Sequencer patterns
- Implement start_flusher() to start background task - Implement _run_flusher() for periodic flushing every timeout seconds - Implement flush_all() to flush all partitions - Implement stop_flusher() with graceful shutdown - Signals shutdown - Waits for flusher task (with timeout) - Final flush to ensure no data loss - Add comprehensive tests for time-based flush, shutdown, and concurrency - Coverage: 96% on buffer.py, 100% on buffer_storage.py All 11 tests passing
All acceptance criteria met: - Per-partition buffers with pluggable storage - Size-based flush (default: 100 events) - Time-based flush (default: 5 seconds) - Async background flusher task - Graceful shutdown flushes all buffers - Comprehensive unit tests (11 tests, 96% coverage)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What This PR Does
Implements Issue #5 (Buffer) with per-partition buffering, size-based and time-based flushing, and pluggable storage backends.
Key Features
Commits
feat(buffer): add Buffer with pluggable storage and retry-flush
feat(buffer): add time-based flush and graceful shutdown
docs: mark Task 8 complete in tasks.md
Test Coverage
Design Decisions
See
notes/projects/eventkit-impl/005-buffer-batching.mdfor detailed analysis:Closes #5