Skip to content

feat: wire MacosWriter into the disk-commit dispatcher#4018

Merged
oferchen merged 1 commit into
masterfrom
feat/wire-macos-writer-1657
May 14, 2026
Merged

feat: wire MacosWriter into the disk-commit dispatcher#4018
oferchen merged 1 commit into
masterfrom
feat/wire-macos-writer-1657

Conversation

@oferchen
Copy link
Copy Markdown
Owner

Summary

  • Adds a Writer::Macos(MacosWriter) variant in crates/transfer/src/disk_commit/writer.rs, guarded by #[cfg(target_os = "macos")].
  • Extends make_writer in crates/transfer/src/disk_commit/process.rs with a macOS arm that wraps the per-file File in fast_io::MacosWriter::from_file(file, target_size) when !use_sparse && append_offset == 0, mirroring the existing io_uring and IOCP arms.
  • Adds MacosWriter::sync() so the dispatcher's flush_and_sync(do_fsync, ...) can drain pending writev buffers and fsync the underlying file.

Motivation

MacosWriter (crates/fast_io/src/macos_io.rs) has shipped for a while: it pairs F_NOCACHE (above the 1 MiB threshold) with writev(2) flushes (up to 64 iovecs). It is re-exported from crates/fast_io/src/lib.rs, but the disk-commit thread still constructs ReusableBufWriter directly on macOS, so the optimization has zero callers in the workspace.

The audit at docs/audits/macos-fastio-fallback.md (PR #4008, task #1652) called out this gap and recommended exactly the wiring in this PR as the fix for #1657.

Behaviour

  • macOS, non-sparse, append_offset == 0: routes through Writer::Macos(MacosWriter). Files >= 1 MiB get F_NOCACHE (page-cache bypass for tree-wide transfers larger than RAM); writes >= 256 KiB flush via writev covering up to 64 iovecs per syscall.
  • macOS, sparse or append_offset != 0: falls through to Writer::Buffered(ReusableBufWriter). MacosWriter does not implement Seek and would not preserve the file position used by sparse/append mode.
  • Other platforms: unchanged. The macOS arm is gated; Writer::Macos does not exist off-target.

Test plan

  • CI fmt + clippy across all platforms.
  • cargo nextest run -p transfer on macOS picks up make_writer_selects_macos_for_non_sparse_zero_offset and make_writer_falls_back_to_buffered_when_seek_required.
  • Linux/Windows builds remain unchanged - the macOS arm is #[cfg(target_os = "macos")] and the dispatcher signature only adds a size_hint: u64 parameter.

References

Adds a Writer::Macos variant guarded by #[cfg(target_os = "macos")] and
extends make_writer with a matching arm so non-sparse, append_offset=0
writes on macOS go through fast_io::MacosWriter. The new path pairs
F_NOCACHE (for files >= 1 MiB) with writev(2) scatter-gather flushes
(up to 64 iovecs per syscall), eliminating page-cache pollution on
tree-wide transfers larger than RAM and batching more buffers per
syscall than ReusableBufWriter's one-or-two-iovec ceiling.

Sparse mode and append mode continue to use ReusableBufWriter because
both require Seek, which MacosWriter does not implement (it issues
writes from the current position without preserving the seek). The
gate mirrors the existing io_uring and IOCP arms.

Also adds a MacosWriter::sync() helper that drains pending writev
buffers and fsyncs the underlying file, so the dispatcher's
flush_and_sync(do_fsync, ...) path keeps its existing semantics.
@github-actions github-actions Bot added the enhancement New feature or request label May 14, 2026
@oferchen oferchen merged commit be98ffb into master May 14, 2026
39 checks passed
@oferchen oferchen deleted the feat/wire-macos-writer-1657 branch May 14, 2026 14:57
oferchen added a commit that referenced this pull request May 18, 2026
Adds a Writer::Macos variant guarded by #[cfg(target_os = "macos")] and
extends make_writer with a matching arm so non-sparse, append_offset=0
writes on macOS go through fast_io::MacosWriter. The new path pairs
F_NOCACHE (for files >= 1 MiB) with writev(2) scatter-gather flushes
(up to 64 iovecs per syscall), eliminating page-cache pollution on
tree-wide transfers larger than RAM and batching more buffers per
syscall than ReusableBufWriter's one-or-two-iovec ceiling.

Sparse mode and append mode continue to use ReusableBufWriter because
both require Seek, which MacosWriter does not implement (it issues
writes from the current position without preserving the seek). The
gate mirrors the existing io_uring and IOCP arms.

Also adds a MacosWriter::sync() helper that drains pending writev
buffers and fsyncs the underlying file, so the dispatcher's
flush_and_sync(do_fsync, ...) path keeps its existing semantics.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant