Skip to content

async sub-block backfill for forked writes#22

Merged
jaredLunde merged 12 commits into
mainfrom
jared/even-better-backfill
Mar 8, 2026
Merged

async sub-block backfill for forked writes#22
jaredLunde merged 12 commits into
mainfrom
jared/even-better-backfill

Conversation

@jaredLunde
Copy link
Copy Markdown
Contributor

  • When a forked VM writes a sub-block (e.g., 4KB) to a 128KB block that only exists in S3, the write now lands on local SSD immediately instead of synchronously fetching the full block from S3
  • Tracks unfilled sub-regions via a 32-bit bitmap in partial_blocks: DashMap<usize, AtomicU32>, with background tasks backfilling from S3
  • Extends WAL with 24-byte partial entries (high-bit flag + bitmap) so partial state survives crashes
  • Flush skips partial blocks (no garbage upload); on-demand reads merge SSD + S3 data
  • Post-recovery backfill spawning ensures partial blocks don't stay dirty indefinitely
  • Cap of 1024 simultaneous partial blocks; beyond that, falls back to synchronous fetch
  • Adds a ton more data integrity tests

@jaredLunde jaredLunde merged commit f88da2b into main Mar 8, 2026
20 checks passed
@jaredLunde jaredLunde deleted the jared/even-better-backfill branch March 8, 2026 05:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant