feat(fast_io): add IocpDiskBatch matching IoUringDiskBatch surface (#1898)#3609
Merged
Conversation
…1898) Adds a Windows IOCP batched-write API that mirrors the IoUringDiskBatch public surface, so the disk-commit thread can batch overlapped WriteFile submissions through a completion port and drain them with GetQueuedCompletionStatusEx. The new IocpDiskBatch: - begin_file / write_data / flush / commit_file / bytes_written - Reopens the file with FILE_FLAG_OVERLAPPED via ReOpenFile and associates it with a private completion port. - Tracks in-flight overlapped operations as Pin<Box<OverlappedOp>> so pointers passed to WriteFile remain stable until completion, and matches each completion to its op by OVERLAPPED address. - Best-effort flush on Drop to avoid leaking pending I/O. - Stub on non-Windows / when the iocp feature is off so cross-platform builds keep compiling. Wires through disk-commit: - DiskCommitConfig gains iocp_policy: fast_io::IocpPolicy (default Auto) - Writer enum gains an Iocp variant (Windows + iocp feature only) - process_file / process_whole_file accept an iocp_batch parameter and select it in make_writer when sparse mode is off and append_offset is zero. - The disk thread constructs a single IocpDiskBatch up-front (only when io_uring is unavailable so the two backends stay mutually exclusive) and threads it into every per-file call. Tests cover policy plumbing, Windows batched submission of N writes, completion ordering independent of submission order, error propagation from a failed reopen, large writes that exceed the buffer, fsync via FlushFileBuffers, drop semantics, and many rotations to verify no leaked overlapped handles.
The transfer crate uses #[cfg(feature = "iocp")] in disk_commit/{process,writer}.rs
without declaring `iocp` as a feature in its own Cargo.toml. Under
RUSTFLAGS=-D warnings this triggers `unexpected_cfgs` as a hard clippy error,
breaking CI on the feat/iocp-disk-batch branch.
Add `iocp = ["fast_io/iocp"]` to the transfer crate features (mirroring the
io_uring pattern), and forward `transfer/iocp` from the workspace top-level
iocp feature so a single `--features iocp` activates both crates.
# Conflicts: # crates/fast_io/src/iocp/mod.rs # crates/fast_io/src/lib.rs
Owner
Author
|
@dependabot rebase |
oferchen
added a commit
that referenced
this pull request
May 5, 2026
…1898) (#3609) * feat(fast_io): add IocpDiskBatch matching IoUringDiskBatch surface (#1898) Adds a Windows IOCP batched-write API that mirrors the IoUringDiskBatch public surface, so the disk-commit thread can batch overlapped WriteFile submissions through a completion port and drain them with GetQueuedCompletionStatusEx. The new IocpDiskBatch: - begin_file / write_data / flush / commit_file / bytes_written - Reopens the file with FILE_FLAG_OVERLAPPED via ReOpenFile and associates it with a private completion port. - Tracks in-flight overlapped operations as Pin<Box<OverlappedOp>> so pointers passed to WriteFile remain stable until completion, and matches each completion to its op by OVERLAPPED address. - Best-effort flush on Drop to avoid leaking pending I/O. - Stub on non-Windows / when the iocp feature is off so cross-platform builds keep compiling. Wires through disk-commit: - DiskCommitConfig gains iocp_policy: fast_io::IocpPolicy (default Auto) - Writer enum gains an Iocp variant (Windows + iocp feature only) - process_file / process_whole_file accept an iocp_batch parameter and select it in make_writer when sparse mode is off and append_offset is zero. - The disk thread constructs a single IocpDiskBatch up-front (only when io_uring is unavailable so the two backends stay mutually exclusive) and threads it into every per-file call. Tests cover policy plumbing, Windows batched submission of N writes, completion ordering independent of submission order, error propagation from a failed reopen, large writes that exceed the buffer, fsync via FlushFileBuffers, drop semantics, and many rotations to verify no leaked overlapped handles. * style: cargo fmt --all * fix(transfer): add iocp feature forwarding to fast_io/iocp The transfer crate uses #[cfg(feature = "iocp")] in disk_commit/{process,writer}.rs without declaring `iocp` as a feature in its own Cargo.toml. Under RUSTFLAGS=-D warnings this triggers `unexpected_cfgs` as a hard clippy error, breaking CI on the feat/iocp-disk-batch branch. Add `iocp = ["fast_io/iocp"]` to the transfer crate features (mirroring the io_uring pattern), and forward `transfer/iocp` from the workspace top-level iocp feature so a single `--features iocp` activates both crates.
oferchen
added a commit
that referenced
this pull request
May 18, 2026
…1898) (#3609) * feat(fast_io): add IocpDiskBatch matching IoUringDiskBatch surface (#1898) Adds a Windows IOCP batched-write API that mirrors the IoUringDiskBatch public surface, so the disk-commit thread can batch overlapped WriteFile submissions through a completion port and drain them with GetQueuedCompletionStatusEx. The new IocpDiskBatch: - begin_file / write_data / flush / commit_file / bytes_written - Reopens the file with FILE_FLAG_OVERLAPPED via ReOpenFile and associates it with a private completion port. - Tracks in-flight overlapped operations as Pin<Box<OverlappedOp>> so pointers passed to WriteFile remain stable until completion, and matches each completion to its op by OVERLAPPED address. - Best-effort flush on Drop to avoid leaking pending I/O. - Stub on non-Windows / when the iocp feature is off so cross-platform builds keep compiling. Wires through disk-commit: - DiskCommitConfig gains iocp_policy: fast_io::IocpPolicy (default Auto) - Writer enum gains an Iocp variant (Windows + iocp feature only) - process_file / process_whole_file accept an iocp_batch parameter and select it in make_writer when sparse mode is off and append_offset is zero. - The disk thread constructs a single IocpDiskBatch up-front (only when io_uring is unavailable so the two backends stay mutually exclusive) and threads it into every per-file call. Tests cover policy plumbing, Windows batched submission of N writes, completion ordering independent of submission order, error propagation from a failed reopen, large writes that exceed the buffer, fsync via FlushFileBuffers, drop semantics, and many rotations to verify no leaked overlapped handles. * style: cargo fmt --all * fix(transfer): add iocp feature forwarding to fast_io/iocp The transfer crate uses #[cfg(feature = "iocp")] in disk_commit/{process,writer}.rs without declaring `iocp` as a feature in its own Cargo.toml. Under RUSTFLAGS=-D warnings this triggers `unexpected_cfgs` as a hard clippy error, breaking CI on the feat/iocp-disk-batch branch. Add `iocp = ["fast_io/iocp"]` to the transfer crate features (mirroring the io_uring pattern), and forward `transfer/iocp` from the workspace top-level iocp feature so a single `--features iocp` activates both crates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IocpDiskBatchincrates/fast_io/src/iocp/disk_batch.rsthat mirrors theIoUringDiskBatchpublic surface (begin_file/write_data/flush/commit_file/bytes_written/bytes_written_with_pending) but submits overlappedWriteFilecalls through a private completion port and drains them withGetQueuedCompletionStatusEx.FILE_FLAG_OVERLAPPEDviaReOpenFile, pins each in-flightOverlappedOpsoWriteFilesees a stableOVERLAPPEDpointer, and matches completions by overlapped address. Best-effort flush onDropso partial submissions cannot leak overlapped I/O.DiskCommitConfiggainsiocp_policy: fast_io::IocpPolicy(defaultAuto),Writergains anIocpvariant (Windows +iocpfeature only), anddisk_thread_mainconstructs a single batch up-front but only when io_uring is not active so the two backends stay mutually exclusive.iocp_stub.rsexposes the same surface returningUnsupported/None/0. Sparse mode and append paths still take the buffered writer.Resolves part of #1898 (does not touch the completion-port pump being reworked in #1897 - this layers on top).
Test plan
cargo nextest run -p fast_io --features iocp -E 'test(disk_batch)'(Windows runner via CI)cargo nextest run -p transfer --all-features -E 'test(disk_commit)'cargo check --workspace --all-targetson Linux + macOS to confirm the stub keeps non-Windows builds greenNew unit tests in
disk_batch.rscover:reopen_overlappedFlushFileBuffersoncommit_file(true)Dropflushing pending data